2024-03-04T07:14:29.022 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T07:14:29.022 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} --archive /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706 --verbose --name leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/orig.config.yaml 2024-03-04T07:14:29.069 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T07:14:29.159 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706 branch: wip-lusov-qdb description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} email: null first_in_suite: false job_id: '7580706' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: cephfs: max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: dry_run: true install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNx5nWQWa/4dHq4IL0y2koYBMRp0lbmm2KW06oL0OEMIW+lsJZwomGs45GMJZNJsl6WhrLQ7TTNS1zY+EJ29BgQ= smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE77/V2gQpv2FdAfkeHRIYQ4VPJrWTNJeQhWMjdMgn9BVhBXz95IwO4lHKJ1/cuOrVGYMHfs+ho308kHSddU8rc= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbrYJDc/cEMuOlrsDnRJysw6nf7F0Bpy3Zyjj/vUuVB7ASJ5+tIamGzJfZECXZECZXMKiucdSyGRbFB70Emig4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/test_o_trunc.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_07:05:03 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T07:14:29.160 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-04T07:14:29.160 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-04T07:14:29.160 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T07:14:29.162 INFO:teuthology.task.internal:Checking packages... 2024-03-04T07:14:29.183 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-04T07:14:29.183 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T07:14:29.183 INFO:teuthology.packaging:ref: None 2024-03-04T07:14:29.183 INFO:teuthology.packaging:tag: None 2024-03-04T07:14:29.183 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-04T07:14:29.184 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:14:29.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-04T07:14:29.552 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-04T07:14:29.554 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T07:14:29.561 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T07:14:29.562 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T07:14:29.581 INFO:teuthology.task.internal:Saving configuration 2024-03-04T07:14:29.599 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T07:14:29.604 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T07:14:29.631 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 07:06:36.225350', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNx5nWQWa/4dHq4IL0y2koYBMRp0lbmm2KW06oL0OEMIW+lsJZwomGs45GMJZNJsl6WhrLQ7TTNS1zY+EJ29BgQ='} 2024-03-04T07:14:29.654 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 07:06:36.224414', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE77/V2gQpv2FdAfkeHRIYQ4VPJrWTNJeQhWMjdMgn9BVhBXz95IwO4lHKJ1/cuOrVGYMHfs+ho308kHSddU8rc='} 2024-03-04T07:14:29.674 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 07:06:36.222827', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbrYJDc/cEMuOlrsDnRJysw6nf7F0Bpy3Zyjj/vUuVB7ASJ5+tIamGzJfZECXZECZXMKiucdSyGRbFB70Emig4='} 2024-03-04T07:14:29.674 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T07:14:29.680 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-04T07:14:29.681 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-04T07:14:29.681 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-04T07:14:29.681 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T07:14:29.774 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f42ab76af70>, signals=[15]) 2024-03-04T07:14:29.775 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T07:14:29.781 INFO:teuthology.task.internal:Opening connections... 2024-03-04T07:14:29.781 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T07:14:29.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:14:29.872 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2024-03-04T07:14:29.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:14:29.959 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-03-04T07:14:29.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:14:30.044 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T07:14:30.053 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-04T07:14:30.071 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-04T07:14:30.071 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-04T07:14:30.128 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T07:14:30.129 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-04T07:14:30.129 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T07:14:30.129 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T07:14:30.129 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T07:14:30.129 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-04T07:14:30.146 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2024-03-04T07:14:30.164 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2024-03-04T07:14:30.164 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2024-03-04T07:14:30.220 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2024-03-04T07:14:30.220 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="9" 2024-03-04T07:14:30.220 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="9" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el9" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:LOGO="fedora-logo-icon" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T07:14:30.221 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T07:14:30.221 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2024-03-04T07:14:30.246 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-03-04T07:14:30.263 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-03-04T07:14:30.263 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T07:14:30.320 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-03-04T07:14:30.321 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T07:14:30.321 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T07:14:30.321 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T07:14:30.321 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-03-04T07:14:30.346 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T07:14:30.360 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T07:14:30.372 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T07:14:30.373 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-04T07:14:30.376 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2024-03-04T07:14:30.378 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-03-04T07:14:30.395 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T07:14:30.402 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T07:14:30.402 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-04T07:14:30.432 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2024-03-04T07:14:30.436 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-03-04T07:14:30.536 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T07:14:30.610 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T07:14:30.624 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-04T07:14:30.624 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-04T07:14:30.624 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-04T07:14:30.624 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T07:14:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:14:30.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-04T07:14:30.891 DEBUG:teuthology.task.kernel:sha1 for client.0 is 786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T07:14:30.891 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "786939134d7eee4dc7a805245d3f595929af37d7"... 2024-03-04T07:14:30.891 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-04T07:14:30.908 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-04T07:14:30.908 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T07:14:30.909 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-04T07:14:30.909 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum upgrade -y linux-firmware 2024-03-04T07:14:33.252 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-04T07:14:35.061 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-03-04T07:14:37.776 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-04T07:15:06.523 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 710 kB/s | 19 MB 00:27 2024-03-04T07:15:10.416 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-04T07:15:11.958 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-03-04T07:15:15.837 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 20 MB 00:03 2024-03-04T07:15:20.809 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-04T07:15:23.102 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T07:15:23.112 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T07:15:23.112 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T07:15:23.112 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T07:15:23.112 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-04T07:15:23.112 INFO:teuthology.orchestra.run.smithi116.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-04T07:15:23.112 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:15:23.112 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T07:15:23.113 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-04T07:15:23.113 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 1 Package 2024-03-04T07:15:23.113 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:15:23.113 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 375 M 2024-03-04T07:15:23.113 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T07:15:39.908 INFO:teuthology.orchestra.run.smithi116.stdout:linux-firmware-20240219-143.el9.noarch.rpm 22 MB/s | 375 MB 00:16 2024-03-04T07:15:39.909 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T07:15:39.910 INFO:teuthology.orchestra.run.smithi116.stdout:Total 22 MB/s | 375 MB 00:16 2024-03-04T07:15:39.910 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T07:15:40.005 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T07:15:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T07:15:42.646 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T07:15:42.647 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T07:15:48.885 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T07:15:49.108 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-04T07:15:49.184 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T07:15:50.321 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T07:15:50.322 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-04T07:15:50.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-04T07:15:50.732 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:15:50.732 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-04T07:15:50.732 INFO:teuthology.orchestra.run.smithi116.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-04T07:15:50.732 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:15:50.733 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T07:15:51.052 INFO:teuthology.task.kernel:Downloading kernel 786939134d7eee4dc7a805245d3f595929af37d7 on client.0... 2024-03-04T07:15:51.052 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=786939134d7eee4dc7a805245d3f595929af37d7 2024-03-04T07:15:51.195 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/ 2024-03-04T07:15:51.196 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_g786939134d7e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-04T07:15:51.201 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi116.front.sepia.ceph.com to complete... 2024-03-04T07:16:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04 07:16:12 URL:https://3.chacra.ceph.com/r/kernel/testing/786939134d7eee4dc7a805245d3f595929af37d7/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_g786939134d7e-1.x86_64.rpm [336049686/336049686] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-04T07:16:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:FINISHED --2024-03-04 07:16:12-- 2024-03-04T07:16:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:Total wall clock time: 21s 2024-03-04T07:16:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:Downloaded: 1 files, 320M in 21s (15.2 MB/s) 2024-03-04T07:16:12.620 INFO:teuthology.task.kernel:Installing kernel 786939134d7eee4dc7a805245d3f595929af37d7 on client.0... 2024-03-04T07:16:12.620 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-04T07:16:14.540 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying... ######################################## 2024-03-04T07:16:14.591 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing... ######################################## 2024-03-04T07:16:14.668 INFO:teuthology.orchestra.run.smithi116.stdout:Updating / installing... 2024-03-04T07:16:22.178 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-6.8.0_rc6_g786939134d7e-1 ######################################## 2024-03-04T07:16:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T07:17:00.579 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-04T07:17:00.579 DEBUG:teuthology.orchestra.run.smithi116:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-04T07:17:00.655 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e 2024-03-04T07:17:00.655 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/System.map 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/config 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-04T07:17:00.656 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu 2024-03-04T07:17:00.657 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu/mce 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/arch/x86/kvm/kvm.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/ansi_cprng.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/anubis.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/arc4.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_pq.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_tx.ko 2024-03-04T07:17:00.658 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/async_xor.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/async_tx/raid6test.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/authenc.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/authencesn.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blake2b_generic.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blowfish_common.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/blowfish_generic.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/camellia_generic.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast5_generic.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast6_generic.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cast_common.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/ccm.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cmac.ko 2024-03-04T07:17:00.659 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crc32_generic.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crypto_engine.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/crypto_user.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/cts.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/des_generic.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/dh_generic.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/echainiv.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/essiv.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/fcrypt.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/khazad.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/lz4.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/lz4hc.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/md4.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/michael_mic.ko 2024-03-04T07:17:00.660 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/pcbc.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/pcrypt.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/rmd160.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/seed.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/serpent_generic.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/tcrypt.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/tea.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/twofish_common.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/twofish_generic.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/vmac.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/wp512.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xcbc.ko 2024-03-04T07:17:00.661 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xor.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/crypto/xxhash_generic.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/acpi_pad.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/custom_method.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/ec_sys.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/platform_profile.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/sbs.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/sbshc.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/acpi/video.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/acard-ahci.ko 2024-03-04T07:17:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/ahci_platform.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/ata_generic.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/libahci_platform.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_acpi.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ali.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_amd.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_artop.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_atiixp.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_atp867x.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cmd640.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_cypress.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_efar.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt366.ko 2024-03-04T07:17:00.663 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_it8213.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_it821x.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_jmicron.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_marvell.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_mpiix.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_netcell.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ninja32.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ns87410.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_ns87415.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_opti.ko 2024-03-04T07:17:00.664 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_optidma.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_piccolo.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_rdc.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sch.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_serverworks.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sil680.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sis.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_triflex.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pata_via.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/pdc_adma.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_inic162x.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_mv.ko 2024-03-04T07:17:00.665 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_nv.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_promise.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_qstor.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sil.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sil24.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sis.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_svw.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_sx4.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_uli.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_via.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ata/sata_vsc.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/auxdisplay 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base 2024-03-04T07:17:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/bcma 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/bcma/bcma.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/aoe 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/aoe/aoe.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/brd.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/drbd 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/drbd/drbd.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/floppy.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/loop.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/mtip32xx 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-04T07:17:00.667 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/nbd.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/null_blk 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/pktcdvd.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/rbd.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/virtio_blk.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkback 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/xen-blkfront.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/zram 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/block/zram/zram.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hangcheck-timer.ko 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random 2024-03-04T07:17:00.668 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/lp.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/mwave 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/mwave/mwave.ko 2024-03-04T07:17:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/ppdev.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tlclk.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/uv_mmtimer.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/char/virtio_console.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-04T07:17:00.670 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-04T07:17:00.671 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/padlock-aes.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/padlock-sha.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/virtio 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dca 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dca/dca.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/devfreq 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/dw 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-04T07:17:00.672 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/ioat 2024-03-04T07:17:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-04T07:17:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac 2024-03-04T07:17:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/amd64_edac.ko 2024-03-04T07:17:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/e752x_edac.ko 2024-03-04T07:17:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-04T07:17:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i3000_edac.ko 2024-03-04T07:17:00.673 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i3200_edac.ko 2024-03-04T07:17:00.676 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i5100_edac.ko 2024-03-04T07:17:00.676 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i5400_edac.ko 2024-03-04T07:17:00.676 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i7300_edac.ko 2024-03-04T07:17:00.676 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i7core_edac.ko 2024-03-04T07:17:00.676 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/i82975x_edac.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/ie31200_edac.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/sb_edac.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/edac/x38_edac.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware/edd.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio/gpio-ich.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-appleir.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-aureal.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-axff.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-dr.ko 2024-03-04T07:17:00.677 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-elecom.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-elo.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-emsff.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gaff.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gt683r.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-gyration.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-holtekff.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-hyperv.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-icade.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-keytouch.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-kye.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-lcpower.ko 2024-03-04T07:17:00.678 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-led.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-lenovo.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-multitouch.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-ortek.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-penmount.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-petalynx.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-picolcd.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-pl.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-plantronics.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-primax.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-rmi.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-04T07:17:00.679 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-roccat.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-saitek.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-samsung.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sjoy.ko 2024-03-04T07:17:00.680 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sony.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-speedlink.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-steelseries.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-sunplus.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-tivo.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-tmff.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-topseed.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-twinhan.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-uclogic.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-waltop.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-wiimote.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-xinmo.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-zpff.ko 2024-03-04T07:17:00.681 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/hid-zydacron.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/uhid.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hid/wacom.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_balloon.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_utils.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hv/hv_vmbus.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/abituguru.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/abituguru3.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ad7414.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ad7418.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adc128d818.ko 2024-03-04T07:17:00.682 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1025.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1026.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1029.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm1031.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adm9240.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ads7828.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7410.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7411.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7462.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7470.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7475.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/adt7x10.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/amc6821.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/applesmc.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asb100.ko 2024-03-04T07:17:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asc7621.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/atxp1.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/coretemp.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/dme1737.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ds1621.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ds620.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/emc1403.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/emc6w201.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f71805f.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f71882fg.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/f75375s.ko 2024-03-04T07:17:00.684 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/fschmd.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/g760a.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/g762.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/gl518sm.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/gl520sm.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ibmaem.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ibmpex.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ina209.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ina2xx.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/it87.ko 2024-03-04T07:17:00.685 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/k10temp.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/k8temp.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm63.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm73.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm75.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm77.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm78.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm80.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm83.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm85.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm87.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm90.ko 2024-03-04T07:17:00.686 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm92.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm93.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95234.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95241.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/lm95245.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc2945.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4151.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4215.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4222.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4245.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4260.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/ltc4261.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max16065.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max1619.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max1668.ko 2024-03-04T07:17:00.687 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max197.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6639.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6650.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/max6697.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/mcp3021.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6683.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct6775.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/nct7802.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pc87360.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pc87427.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pcf8591.ko 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus 2024-03-04T07:17:00.688 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/powr1220.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch5627.ko 2024-03-04T07:17:00.689 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch5636.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sht15.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sht21.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/shtc1.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/sis5595.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/thmc50.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp102.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp103.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp401.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/tmp421.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-04T07:17:00.690 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/via686a.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/vt1211.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/vt8231.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83627hf.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83781d.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83791d.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83792d.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83793.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83795.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos 2024-03-04T07:17:00.691 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-04T07:17:00.692 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-dev.ko 2024-03-04T07:17:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-mux.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/i2c/i2c-stub.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/cxgb4 2024-03-04T07:17:00.694 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx4 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx5 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mthca 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/ipoib 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/iser 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/isert 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srp 2024-03-04T07:17:00.695 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srpt 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/gameport/gameport.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/joydev.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/keyboard 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/appletouch.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-04T07:17:00.696 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/sermouse.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/rmi4 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/serio/serio_raw.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/input/sparse-keymap.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-blinkm.ko 2024-03-04T07:17:00.697 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-lm3530.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-lp3944.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/leds-ss4200.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/bcache 2024-03-04T07:17:00.698 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/bcache/bcache.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-bio-prison.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-cache-smq.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-cache.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-crypt.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-delay.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-flakey.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-log-userspace.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-multipath.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-queue-length.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-raid.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-round-robin.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-service-time.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-switch.ko 2024-03-04T07:17:00.699 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-thin-pool.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/dm-verity.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/persistent-data 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid0.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid1.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid10.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/md/raid456.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptbase.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptctl.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptfc.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptlan.ko 2024-03-04T07:17:00.700 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptsas.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/message/fusion/mptspi.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/lpc_ich.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/lpc_sch.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/mfd-core.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/sm501.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/viperboard.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/vx855.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/mfd/wl1273-core.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/altera-stapl 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-04T07:17:00.701 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/apds9802als.ko 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/apds990x.ko 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/bh1770glc.ko 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/cb710 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/cb710/cb710.ko 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/at24.ko 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-04T07:17:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/enclosure.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/hpilo.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/isl29003.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/isl29020.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei-me.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/mei/mei.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-gru 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp 2024-03-04T07:17:00.703 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tifm_core.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/tsl2550.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_balloon.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_vmci 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/bonding 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/bonding/bonding.ko 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa 2024-03-04T07:17:00.704 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/b53 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/dummy.ko 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/eql.ko 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-04T07:17:00.705 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/adaptec 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/agere 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/alteon 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-04T07:17:00.706 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/altera 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/alx 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-04T07:17:00.707 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade/bna 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio 2024-03-04T07:17:00.708 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco/enic 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec 2024-03-04T07:17:00.709 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/dnet.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex/benet 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-04T07:17:00.710 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/i40e 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/iavf 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-04T07:17:00.711 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igb 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/jme.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox 2024-03-04T07:17:00.712 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/micrel 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/neterion 2024-03-04T07:17:00.713 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/nvidia 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rdc 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek 2024-03-04T07:17:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rocker 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sfc 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/silan 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-04T07:17:00.715 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/tehuti 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-04T07:17:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ti 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/xircom 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/geneve.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/hyperv 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-04T07:17:00.717 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ieee802154 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ifb.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ipvlan 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/macvlan.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/macvtap.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/mii.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/net_failover.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/netconsole.ko 2024-03-04T07:17:00.718 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/nlmon.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/amd.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/broadcom.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/cicada.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/davicom.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/dp83640.ko 2024-03-04T07:17:00.719 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/et1011c.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/icplus.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/lxt.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/marvell.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/micrel.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/national.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/phylink.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/qsemi.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/realtek.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/smsc.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/phy/vitesse.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-04T07:17:00.720 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pppoe.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pppox.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/ppp/pptp.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip/slhc.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/slip/slip.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/sungem_phy.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/tap.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team.ko 2024-03-04T07:17:00.721 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_random.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/tun.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/veth.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/virtio_net.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vmxnet3 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vxlan 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netback 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/net/xen-netfront.ko 2024-03-04T07:17:00.722 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ntb 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ntb/ntb.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/nvme/host/nvme.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_cs.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_pc.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/parport/parport_serial.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/hotplug 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-04T07:17:00.723 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/pcie 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pci/xen-pcifront.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/i82092.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/pd6729.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl/intel 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-04T07:17:00.724 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/hdaps.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-04T07:17:00.725 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/platform/x86/wmi.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-04T07:17:00.726 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/pps/pps_core.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp/ptp.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-04T07:17:00.727 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-04T07:17:00.728 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-sas.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/BusLogic.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/a100u2w.ko 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aacraid 2024-03-04T07:17:00.729 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/advansys.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/am53c974.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/arcmsr 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/atp870u.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/be2iscsi 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bfa 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2fc 2024-03-04T07:17:00.730 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2i 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ch.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/csiostor 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/dc395x.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-04T07:17:00.731 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/dmx3191d.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esas2r 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/esp_scsi.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fnic 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hpsa.ko 2024-03-04T07:17:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hptiop.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/initio.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ipr.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ips.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/isci 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/isci/isci.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libfc 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libiscsi.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-04T07:17:00.733 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libsas 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/lpfc 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mpt3sas 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvsas 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/mvumi.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pm8001 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-04T07:17:00.734 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/pmcraid.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla1280.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla4xxx 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/raid_class.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_debug.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/ses.ko 2024-03-04T07:17:00.735 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/st.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/stex.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/sym53c8xx_2 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/scsi/wd719x.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ssb 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ssb/ssb.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/iscsi 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/loopback 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_file.ko 2024-03-04T07:17:00.736 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_iblock.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_mod.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_pscsi.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/target_core_user.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/tcm_fc 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-04T07:17:00.737 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty 2024-03-04T07:17:00.738 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/ipwireless 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/n_gsm.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/n_hdlc.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/nozomi.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/8250 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/jsm 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/core 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/host 2024-03-04T07:17:00.739 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_aec.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_cif.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/uio/uio_sercos3.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/usblp.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/class/usbtmc.ko 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/host 2024-03-04T07:17:00.740 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image/mdc800.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/image/microtek.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/adutux.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/emi26.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/emi62.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/ezusb.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/idmouse.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/ldusb.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-04T07:17:00.741 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/sisusbvga 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usb3503.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usblcd.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/uss720.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/misc/yurex.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/phy 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/aircable.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ark3116.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-04T07:17:00.742 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ch341.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cp210x.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/empeg.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/io_ti.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ipaq.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ipw.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-04T07:17:00.743 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/keyspan.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mos7720.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/mos7840.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/navman.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/omninet.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/opticon.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/option.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/oti6858.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/pl2303.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/qcaux.ko 2024-03-04T07:17:00.744 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/qcserial.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/quatech2.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/sierra.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ssu100.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/visor.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage 2024-03-04T07:17:00.745 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/uas.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip 2024-03-04T07:17:00.746 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/vfio.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_net.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight 2024-03-04T07:17:00.747 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/lcd.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/virtio/virtio_ring.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters/ds2482.ko 2024-03-04T07:17:00.748 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/masters/ds2490.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/w1/wire.ko 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog 2024-03-04T07:17:00.749 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/hpwdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/i6300esb.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ibmasr.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/machzwd.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/nv_tco.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-04T07:17:00.750 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/softdog.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/via_wdt.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-evtchn.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-04T07:17:00.751 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-gntdev.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-pciback 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xen-privcmd.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xenfs 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/9p 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/9p/9p.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/affs 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/affs/affs.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/befs 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/befs/befs.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/binfmt_misc.ko 2024-03-04T07:17:00.752 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/btrfs 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/btrfs/btrfs.ko 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cachefiles 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cachefiles/cachefiles.ko 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ceph 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ceph/ceph.ko 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/coda 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/coda/coda.ko 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cramfs 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/cramfs/cramfs.ko 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/dlm 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/dlm/dlm.ko 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ecryptfs 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/f2fs 2024-03-04T07:17:00.753 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/f2fs/f2fs.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/fat.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/msdos.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fat/vfat.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse/cuse.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/fuse/fuse.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/gfs2 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/gfs2/gfs2.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfs 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfs/hfs.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfsplus 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/hfsplus/hfsplus.ko 2024-03-04T07:17:00.754 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/isofs 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/isofs/isofs.ko 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/jfs 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/jfs/jfs.ko 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/lockd 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/lockd/lockd.ko 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/minix 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/minix/minix.ko 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/netfs 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/netfs/netfs.ko 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/blocklayout 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/filelayout 2024-03-04T07:17:00.755 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/flexfilelayout 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfs.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfsv3.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs/nfsv4.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common/grace.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfsd 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nfsd/nfsd.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nilfs2 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nilfs2/nilfs2.ko 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls 2024-03-04T07:17:00.756 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-celtic.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-centeuro.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-croatian.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-cyrillic.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-gaelic.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-greek.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-iceland.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-inuit.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-roman.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-romanian.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/mac-turkish.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1250.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1251.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp1255.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp737.ko 2024-03-04T07:17:00.757 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp775.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp850.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp852.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp855.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp857.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp860.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp861.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp862.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp863.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp864.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp865.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp866.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp869.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp874.ko 2024-03-04T07:17:00.758 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp932.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp936.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp949.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_cp950.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_euc-jp.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-04T07:17:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-r.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_koi8-u.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/nls/nls_utf8.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/cluster 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlm 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlmfs 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2.ko 2024-03-04T07:17:00.760 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/overlayfs 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/overlayfs/overlay.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/pstore 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/pstore/ramoops.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/reiserfs 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/reiserfs/reiserfs.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/romfs 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/romfs/romfs.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/client 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/client/cifs.ko 2024-03-04T07:17:00.761 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/smb/common/cifs_md4.ko 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/squashfs 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/squashfs/squashfs.ko 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/sysv 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/sysv/sysv.ko 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/udf 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/udf/udf.ko 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ufs 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/ufs/ufs.ko 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/xfs 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/fs/xfs/xfs.ko 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel 2024-03-04T07:17:00.762 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/rcu 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/rcu/rcutorture.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/torture.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/trace 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/asn1_encoder.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crc-itu-t.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crc8.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto/libarc4.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/crypto/libdes.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/libcrc32c.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lru_cache.ko 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4 2024-03-04T07:17:00.763 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4/lz4_compress.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/math 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/math/cordic.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/raid6 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/raid6/raid6_pq.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/reed_solomon 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_bm.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_fsm.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/ts_kmp.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/zstd 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/lib/zstd/zstd_compress.ko 2024-03-04T07:17:00.764 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/mm 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/mm/hwpoison-inject.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/6lowpan.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_dest.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_hop.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_routing.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/6lowpan/nhc_udp.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/garp.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/mrp.ko 2024-03-04T07:17:00.765 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/p8022.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/psnap.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/802/stp.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/8021q 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/8021q/8021q.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_fd.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_rdma.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/9p/9pnet_virtio.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/batman-adv 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/batman-adv/batman-adv.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/br_netfilter.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/bridge.ko 2024-03-04T07:17:00.766 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-04T07:17:00.767 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ceph 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ceph/libceph.ko 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core 2024-03-04T07:17:00.768 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core/failover.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/core/pktgen.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_diag.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_ipv4.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dccp/dccp_ipv6.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dns_resolver 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/dsa_core.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_brcm.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_dsa.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_none.ko 2024-03-04T07:17:00.769 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/dsa/tag_trailer.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/6lowpan 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/ieee802154.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ah4.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/esp4.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/fou.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/gre.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/inet_diag.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_gre.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_tunnel.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ip_vti.ko 2024-03-04T07:17:00.770 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ipcomp.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/ipip.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-04T07:17:00.771 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_bic.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_diag.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-04T07:17:00.772 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_htcp.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_hybla.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_illinois.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_lp.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_scalable.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_vegas.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_veno.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_westwood.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tcp_yeah.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/tunnel4.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/udp_diag.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/udp_tunnel.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-04T07:17:00.773 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ah6.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/esp6.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/fou6.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ip6_vti.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/ipcomp6.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-04T07:17:00.774 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-04T07:17:00.775 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/sit.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/tunnel6.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/key 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/key/af_key.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_core.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_eth.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ip.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-04T07:17:00.776 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/llc 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/llc/llc.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mac802154 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mac802154/mac802154.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mpls 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/mpls/mpls_gso.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-04T07:17:00.777 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-04T07:17:00.778 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conncount.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-04T07:17:00.779 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_flow_table.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat.ko 2024-03-04T07:17:00.780 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nf_tables.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_compat.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_ct.ko 2024-03-04T07:17:00.781 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_fib.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_hash.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_limit.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_log.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_masq.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_nat.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_numgen.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_queue.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_quota.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_redir.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_reject.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_socket.ko 2024-03-04T07:17:00.782 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/nft_tunnel.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_CT.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_DSCP.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_HL.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_HMARK.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_LED.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_LOG.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-04T07:17:00.783 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TEE.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_TRACE.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_addrtype.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_bpf.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cgroup.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cluster.ko 2024-03-04T07:17:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_comment.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connbytes.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connlabel.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connlimit.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_connmark.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_conntrack.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_cpu.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_dccp.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_devgroup.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_dscp.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ecn.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_esp.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_helper.ko 2024-03-04T07:17:00.785 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_hl.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_iprange.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_ipvs.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_l2tp.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_length.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_limit.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_mac.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_mark.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_multiport.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_nat.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_nfacct.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_osf.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_owner.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_physdev.ko 2024-03-04T07:17:00.786 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_pkttype.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_policy.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_quota.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_rateest.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_realm.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_recent.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_sctp.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_set.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_socket.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_state.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_statistic.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_string.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_time.ko 2024-03-04T07:17:00.787 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netfilter/xt_u32.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netlink 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/netlink/netlink_diag.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/nsh 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/nsh/nsh.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/openvswitch.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-geneve.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-gre.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/packet 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/packet/af_packet_diag.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds.ko 2024-03-04T07:17:00.788 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds_rdma.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/rds/rds_tcp.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_csum.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_gact.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_mirred.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_nat.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_pedit.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_police.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_simple.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_skbedit.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/act_vlan.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_basic.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_bpf.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_flow.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_fw.ko 2024-03-04T07:17:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_route.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/cls_u32.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_cmp.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_ipset.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_meta.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_nbyte.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_text.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/em_u32.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_choke.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_codel.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_drr.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_fq.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_gred.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_hfsc.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_hhf.ko 2024-03-04T07:17:00.790 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_htb.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_ingress.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_mqprio.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_multiq.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_netem.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_pie.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_plug.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_prio.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_qfq.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_red.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_sfb.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_sfq.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_tbf.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sched/sch_teql.ko 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp 2024-03-04T07:17:00.791 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp/sctp.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sctp/sctp_diag.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/sunrpc.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/xprtrdma 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc/diag.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/tipc/tipc.ko 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/unix 2024-03-04T07:17:00.792 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/unix/unix_diag.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/xfrm 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/encrypted-keys 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/trusted-keys 2024-03-04T07:17:00.793 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt/lib 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/kernel/virt/lib/irqbypass.ko 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.alias 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.alias.bin 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.alias.bin 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.bin 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.builtin.modinfo 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.dep 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.dep.bin 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.devname 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.order 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.softdep 2024-03-04T07:17:00.794 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.symbols 2024-03-04T07:17:00.795 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/modules.symbols.bin 2024-03-04T07:17:00.795 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.8.0-rc6-g786939134d7e/vmlinuz 2024-03-04T07:17:00.799 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-g786939134d7e 2024-03-04T07:17:00.799 DEBUG:teuthology.orchestra.run.smithi116:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-04T07:17:00.821 INFO:teuthology.orchestra.run.smithi116.stdout:postinstall scriptlet (using /bin/sh): 2024-03-04T07:17:00.821 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-04T07:17:00.821 INFO:teuthology.orchestra.run.smithi116.stdout: /usr/bin/kernel-install add 6.8.0-rc6-g786939134d7e /lib/modules/6.8.0-rc6-g786939134d7e/vmlinuz 2024-03-04T07:17:00.821 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2024-03-04T07:17:00.821 INFO:teuthology.orchestra.run.smithi116.stdout:for file in vmlinuz System.map config; do 2024-03-04T07:17:00.821 INFO:teuthology.orchestra.run.smithi116.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-g786939134d7e/${file}" "/boot/${file}-6.8.0-rc6-g786939134d7e"; then 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout: cp "/lib/modules/6.8.0-rc6-g786939134d7e/${file}" "/boot/${file}-6.8.0-rc6-g786939134d7e" 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout: fi 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:done 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:new-kernel-pkg --remove 6.8.0-rc6-g786939134d7e --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-g786939134d7e.img 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-install remove 6.8.0-rc6-g786939134d7e 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-g786939134d7e 2024-03-04T07:17:00.822 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2024-03-04T07:17:00.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-04T07:17:00.898 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-04T07:17:00.898 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-04T07:17:00.898 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T07:17:00.899 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:17:00.900 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T07:17:00.901 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T07:17:00.901 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 6.8.0-rc6-g786939134d7e 2024-03-04T07:17:00.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T07:17:01.538 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-04T07:17:03.044 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T07:17:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-04T07:17:03.081 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-04T07:17:03.096 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.XVAiBKkuGV 2024-03-04T07:17:03.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XVAiBKkuGV 2024-03-04T07:17:03.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.XVAiBKkuGV 2024-03-04T07:17:03.308 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.XVAiBKkuGV is 6KB 2024-03-04T07:17:03.356 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.XVAiBKkuGV 2024-03-04T07:17:03.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T07:17:03.438 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T07:17:03.439 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T07:17:03.439 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T07:17:03.439 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-g786939134d7e.conf 2024-03-04T07:17:03.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-g786939134d7e 2024-03-04T07:17:03.607 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-04T07:17:33.610 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-04T07:17:33.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T07:17:33.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:17:52.064 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-04T07:18:01.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T07:18:01.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:18:04.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-04T07:18:16.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T07:18:16.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:18:47.995 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-04T07:18:48.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T07:18:48.617 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc6-g786939134d7e"... 2024-03-04T07:18:48.617 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-04T07:18:48.656 INFO:teuthology.orchestra.run.smithi116.stdout:6.8.0-rc6-g786939134d7e 2024-03-04T07:18:48.656 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-g786939134d7e vs 6.8.0-rc6-g786939134d7e 2024-03-04T07:18:48.656 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T07:18:48.657 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T07:18:49.658 DEBUG:teuthology.parallel:result is None 2024-03-04T07:18:49.658 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T07:18:49.665 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T07:18:49.665 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T07:18:49.671 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T07:18:49.674 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T07:18:49.713 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T07:18:49.719 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T07:18:49.726 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T07:18:49.726 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T07:18:49.759 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T07:18:49.761 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T07:18:49.991 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T07:18:50.001 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T07:18:50.001 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T07:18:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:18:50.019 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T07:18:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:18:50.036 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T07:18:50.041 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T07:18:50.062 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T07:18:50.099 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T07:18:50.111 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T07:18:50.120 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T07:18:50.150 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T07:18:50.170 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T07:18:50.255 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T07:18:50.260 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T07:18:50.267 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T07:18:50.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T07:18:50.271 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T07:18:50.274 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T07:18:50.341 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T07:18:50.350 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T07:18:50.351 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T07:18:50.359 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T07:18:50.362 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T07:18:50.398 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T07:18:50.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T07:18:50.602 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T07:18:50.670 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T07:18:50.757 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:18:50.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T07:18:50.837 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T07:18:50.882 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T07:18:50.965 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T07:18:51.009 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T07:18:51.052 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:18:51.053 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T07:18:51.120 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T07:18:51.172 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T07:18:51.224 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T07:18:51.308 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T07:18:51.352 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:18:51.353 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T07:18:51.421 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-04T07:18:51.426 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2024-03-04T07:18:51.428 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-03-04T07:18:51.466 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T07:18:51.503 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T07:18:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T07:18:52.005 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T07:18:52.013 INFO:teuthology.task.internal:Starting timer... 2024-03-04T07:18:52.014 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T07:18:52.039 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T07:18:52.048 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-04T07:18:52.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-04T07:18:52.168 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-04T07:18:52.172 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2024-03-04T07:18:52.230 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2024-03-04T07:18:52.232 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-03-04T07:18:52.322 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-03-04T07:18:52.324 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T07:18:52.324 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-04T07:18:52.377 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-04T07:18:52.377 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2024-03-04T07:18:52.404 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2024-03-04T07:18:52.405 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-03-04T07:18:52.436 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-03-04T07:18:52.436 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi133.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-03-04T07:18:52.436 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T07:18:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:18:52.509 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T07:18:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:18:52.541 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T07:18:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:18:52.571 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T07:18:52.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-04T07:18:52.647 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2024-03-04T07:18:52.681 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-03-04T07:18:52.713 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T07:18:52.724 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T07:18:52.743 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T07:18:52.745 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi133.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T07:25:10.050 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi133.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-03-04T07:25:10.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T07:25:10.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:25:10.201 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-04T07:25:10.356 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-04T07:25:10.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-04T07:25:10.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:25:10.474 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-03-04T07:25:10.544 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-04T07:25:10.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T07:25:10.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T07:25:10.650 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-04T07:25:10.725 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-04T07:25:10.727 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T07:25:10.742 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T07:25:10.742 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T07:25:10.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T07:25:10.750 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T07:25:10.751 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T07:25:10.753 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T07:25:10.753 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T07:25:10.785 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T07:25:10.799 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T07:25:10.803 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T07:25:10.817 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T07:25:10.835 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2024-03-04T07:25:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T07:25:10.847 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-03-04T07:25:10.849 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2024-03-04T07:25:10.861 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-03-04T07:25:10.865 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T07:25:10.877 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T07:25:10.882 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T07:25:10.893 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T07:25:10.894 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T07:25:10.925 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-03-04T07:25:10.928 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T07:25:10.929 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-03-04T07:25:10.929 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.929 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.929 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.929 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.938 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-04T07:25:10.942 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T07:25:10.942 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-04T07:25:10.942 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.942 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.942 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.943 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:10.987 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-04T07:25:11.026 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-04T07:25:11.074 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T07:25:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T07:25:11.256 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-04T07:25:11.265 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T07:25:11.266 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-04T07:25:11.266 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:11.266 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:11.266 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:11.266 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:25:11.268 INFO:teuthology.run_tasks:Running task install... 2024-03-04T07:25:11.277 DEBUG:teuthology.task.install:project ceph 2024-03-04T07:25:11.277 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-04T07:25:11.277 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-04T07:25:11.277 INFO:teuthology.task.install:Using flavor: default 2024-03-04T07:25:11.283 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T07:25:11.283 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-04T07:25:11.284 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-04T07:25:11.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:25:11.286 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-04T07:25:11.287 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:25:11.288 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-04T07:25:11.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:25:11.434 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-04T07:25:11.434 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-04T07:25:11.507 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-04T07:25:11.507 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-04T07:25:11.518 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-04T07:25:11.518 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-04T07:25:11.547 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T07:25:11.548 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:25:11.548 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T07:25:11.595 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T07:25:11.596 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T07:25:11.650 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T07:25:11.650 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:25:11.651 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T07:25:11.666 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T07:25:11.680 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T07:25:11.681 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T07:25:11.703 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T07:25:11.704 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:25:11.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T07:25:11.747 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T07:25:11.752 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T07:25:11.804 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T07:25:11.806 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T07:25:11.814 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-03-04T07:25:11.816 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-03-04T07:25:11.833 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T07:25:11.899 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2024-03-04T07:25:11.901 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2024-03-04T07:25:11.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T07:25:12.053 INFO:teuthology.orchestra.run.smithi152.stdout:74 files removed 2024-03-04T07:25:12.074 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T07:25:12.080 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T07:25:12.135 INFO:teuthology.orchestra.run.smithi133.stdout:74 files removed 2024-03-04T07:25:12.163 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T07:25:12.164 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-04T07:25:12.167 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-04T07:25:12.525 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-04T07:25:12.564 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T07:25:12.957 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 165 kB/s | 83 kB 00:00 2024-03-04T07:25:13.019 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 128 kB/s | 83 kB 00:00 2024-03-04T07:25:13.500 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-04T07:25:13.545 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-04T07:25:13.792 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 114 kB/s | 83 kB 00:00 2024-03-04T07:25:14.065 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-04T07:25:14.100 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 3.3 kB/s | 1.9 kB 00:00 2024-03-04T07:25:14.493 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-03-04T07:25:14.607 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T07:25:14.619 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-04T07:25:15.235 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2024-03-04T07:25:15.804 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T07:25:16.049 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-04T07:25:16.103 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-04T07:25:16.831 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-04T07:25:17.185 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-04T07:25:17.946 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 2.4 MB/s | 2.0 MB 00:00 2024-03-04T07:25:19.364 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-04T07:25:19.490 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-03-04T07:25:24.019 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-03-04T07:25:24.170 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.1 MB 00:01 2024-03-04T07:25:25.796 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2024-03-04T07:25:27.000 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2024-03-04T07:25:28.692 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.1 MB 00:00 2024-03-04T07:25:30.195 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 15 kB 00:00 2024-03-04T07:25:31.906 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-04T07:25:32.402 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:02 2024-03-04T07:25:34.434 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T07:25:34.435 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T07:25:34.447 INFO:teuthology.orchestra.run.smithi133.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T07:25:34.448 INFO:teuthology.orchestra.run.smithi133.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T07:25:34.449 INFO:teuthology.orchestra.run.smithi133.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T07:25:34.450 INFO:teuthology.orchestra.run.smithi133.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T07:25:34.451 INFO:teuthology.orchestra.run.smithi133.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T07:25:34.454 INFO:teuthology.orchestra.run.smithi133.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T07:25:34.455 INFO:teuthology.orchestra.run.smithi133.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T07:25:34.536 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-03-04T07:25:34.562 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================================ 2024-03-04T07:25:34.562 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-03-04T07:25:34.562 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================================ 2024-03-04T07:25:34.562 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-03-04T07:25:34.562 INFO:teuthology.orchestra.run.smithi133.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-04T07:25:34.563 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T07:25:34.564 INFO:teuthology.orchestra.run.smithi133.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T07:25:34.565 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T07:25:34.566 INFO:teuthology.orchestra.run.smithi133.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T07:25:34.567 INFO:teuthology.orchestra.run.smithi133.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-04T07:25:34.568 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T07:25:34.569 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T07:25:34.570 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-04T07:25:34.571 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T07:25:34.572 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T07:25:34.573 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T07:25:34.574 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T07:25:34.575 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout:Installing weak dependencies: 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-03-04T07:25:34.576 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================================ 2024-03-04T07:25:34.577 INFO:teuthology.orchestra.run.smithi133.stdout:Install 150 Packages 2024-03-04T07:25:34.577 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 26 Packages 2024-03-04T07:25:34.577 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:25:34.577 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 224 M 2024-03-04T07:25:34.577 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-03-04T07:25:35.273 INFO:teuthology.orchestra.run.smithi133.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-03-04T07:25:35.749 INFO:teuthology.orchestra.run.smithi133.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.8 MB/s | 888 kB 00:00 2024-03-04T07:25:35.832 INFO:teuthology.orchestra.run.smithi133.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 1.7 MB/s | 145 kB 00:00 2024-03-04T07:25:36.308 INFO:teuthology.orchestra.run.smithi133.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 4.9 MB/s | 2.3 MB 00:00 2024-03-04T07:25:36.586 INFO:teuthology.orchestra.run.smithi133.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 5.4 MB/s | 1.5 MB 00:00 2024-03-04T07:25:37.050 INFO:teuthology.orchestra.run.smithi133.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 2.7 MB/s | 5.3 MB 00:01 2024-03-04T07:25:37.250 INFO:teuthology.orchestra.run.smithi133.stdout:(7/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 9.5 MB/s | 20 MB 00:02 2024-03-04T07:25:37.325 INFO:teuthology.orchestra.run.smithi133.stdout:(8/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 6.3 MB/s | 4.7 MB 00:00 2024-03-04T07:25:37.417 INFO:teuthology.orchestra.run.smithi133.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 275 kB/s | 25 kB 00:00 2024-03-04T07:25:37.934 INFO:teuthology.orchestra.run.smithi133.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 14 MB/s | 9.8 MB 00:00 2024-03-04T07:25:38.009 INFO:teuthology.orchestra.run.smithi133.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 2.9 MB/s | 220 kB 00:00 2024-03-04T07:25:38.076 INFO:teuthology.orchestra.run.smithi133.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 406 kB/s | 27 kB 00:00 2024-03-04T07:25:38.143 INFO:teuthology.orchestra.run.smithi133.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 469 kB/s | 31 kB 00:00 2024-03-04T07:25:38.147 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-04T07:25:38.243 INFO:teuthology.orchestra.run.smithi133.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 7.3 MB/s | 747 kB 00:00 2024-03-04T07:25:38.327 INFO:teuthology.orchestra.run.smithi133.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.9 MB/s | 163 kB 00:00 2024-03-04T07:25:38.402 INFO:teuthology.orchestra.run.smithi133.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.6 MB/s | 127 kB 00:00 2024-03-04T07:25:38.494 INFO:teuthology.orchestra.run.smithi133.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 5.4 MB/s | 501 kB 00:00 2024-03-04T07:25:39.103 INFO:teuthology.orchestra.run.smithi133.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 8.4 MB/s | 5.1 MB 00:00 2024-03-04T07:25:39.178 INFO:teuthology.orchestra.run.smithi133.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 604 kB/s | 45 kB 00:00 2024-03-04T07:25:39.262 INFO:teuthology.orchestra.run.smithi133.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.5 MB/s | 131 kB 00:00 2024-03-04T07:25:39.337 INFO:teuthology.orchestra.run.smithi133.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 2.1 MB/s | 162 kB 00:00 2024-03-04T07:25:39.429 INFO:teuthology.orchestra.run.smithi133.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 3.4 MB/s | 321 kB 00:00 2024-03-04T07:25:39.447 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 913 kB/s | 19 MB 00:21 2024-03-04T07:25:39.571 INFO:teuthology.orchestra.run.smithi133.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 2.1 MB/s | 298 kB 00:00 2024-03-04T07:25:39.663 INFO:teuthology.orchestra.run.smithi133.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.1 MB/s | 99 kB 00:00 2024-03-04T07:25:39.746 INFO:teuthology.orchestra.run.smithi133.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-04T07:25:40.155 INFO:teuthology.orchestra.run.smithi133.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 7.6 MB/s | 3.1 MB 00:00 2024-03-04T07:25:40.213 INFO:teuthology.orchestra.run.smithi133.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 2.9 MB/s | 171 kB 00:00 2024-03-04T07:25:40.263 INFO:teuthology.orchestra.run.smithi133.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 630 kB/s | 31 kB 00:00 2024-03-04T07:25:40.314 INFO:teuthology.orchestra.run.smithi133.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 2.7 MB/s | 138 kB 00:00 2024-03-04T07:25:40.606 INFO:teuthology.orchestra.run.smithi133.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 6.1 MB/s | 1.8 MB 00:00 2024-03-04T07:25:40.938 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T07:25:40.939 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T07:25:40.962 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T07:25:40.963 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T07:25:40.965 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T07:25:40.966 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T07:25:40.970 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T07:25:40.973 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T07:25:40.974 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T07:25:41.064 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-04T07:25:41.090 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-04T07:25:41.090 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-04T07:25:41.090 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-04T07:25:41.090 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-04T07:25:41.090 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T07:25:41.090 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-04T07:25:41.090 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-04T07:25:41.091 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T07:25:41.092 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T07:25:41.093 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-04T07:25:41.094 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T07:25:41.095 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T07:25:41.095 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T07:25:41.095 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T07:25:41.095 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T07:25:41.095 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T07:25:41.095 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T07:25:41.095 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T07:25:41.096 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T07:25:41.096 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T07:25:41.096 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-04T07:25:41.096 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T07:25:41.096 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T07:25:41.096 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T07:25:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-04T07:25:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-04T07:25:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-04T07:25:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-04T07:25:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-04T07:25:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-04T07:25:41.097 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-04T07:25:41.098 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-04T07:25:41.098 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T07:25:41.098 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T07:25:41.098 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T07:25:41.098 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T07:25:41.098 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T07:25:41.099 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T07:25:41.099 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T07:25:41.099 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T07:25:41.099 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T07:25:41.099 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-04T07:25:41.099 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T07:25:41.100 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T07:25:41.100 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T07:25:41.100 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T07:25:41.100 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T07:25:41.100 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-04T07:25:41.100 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T07:25:41.101 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-04T07:25:41.101 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T07:25:41.101 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T07:25:41.101 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T07:25:41.101 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T07:25:41.101 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T07:25:41.102 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T07:25:41.102 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T07:25:41.102 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T07:25:41.102 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T07:25:41.102 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T07:25:41.102 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T07:25:41.103 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T07:25:41.104 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T07:25:41.104 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T07:25:41.104 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T07:25:41.104 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T07:25:41.104 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T07:25:41.105 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T07:25:41.106 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T07:25:41.107 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T07:25:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T07:25:41.109 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-04T07:25:41.110 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T07:25:41.110 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:25:41.110 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-04T07:25:41.110 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-04T07:25:41.110 INFO:teuthology.orchestra.run.smithi116.stdout:Install 150 Packages 2024-03-04T07:25:41.110 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 26 Packages 2024-03-04T07:25:41.110 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:25:41.111 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 224 M 2024-03-04T07:25:41.111 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-04T07:25:41.331 INFO:teuthology.orchestra.run.smithi133.stdout:(31/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 3.9 MB/s | 17 MB 00:04 2024-03-04T07:25:41.506 INFO:teuthology.orchestra.run.smithi133.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.4 MB/s | 248 kB 00:00 2024-03-04T07:25:41.607 INFO:teuthology.orchestra.run.smithi133.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 493 kB/s | 49 kB 00:00 2024-03-04T07:25:41.732 INFO:teuthology.orchestra.run.smithi133.stdout:(34/176): ceph-mgr-diskprediction-local-19.0.0- 6.6 MB/s | 7.4 MB 00:01 2024-03-04T07:25:41.758 INFO:teuthology.orchestra.run.smithi133.stdout:(35/176): ceph-prometheus-alerts-19.0.0-1739.g1 102 kB/s | 15 kB 00:00 2024-03-04T07:25:41.891 INFO:teuthology.orchestra.run.smithi133.stdout:(36/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.6 MB/s | 264 kB 00:00 2024-03-04T07:25:41.925 INFO:teuthology.orchestra.run.smithi133.stdout:(37/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-04T07:25:41.958 INFO:teuthology.orchestra.run.smithi133.stdout:(38/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 272 kB/s | 18 kB 00:00 2024-03-04T07:25:42.132 INFO:teuthology.orchestra.run.smithi116.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-04T07:25:42.175 INFO:teuthology.orchestra.run.smithi133.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-03-04T07:25:42.209 INFO:teuthology.orchestra.run.smithi133.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 3.6 MB/s | 920 kB 00:00 2024-03-04T07:25:42.234 INFO:teuthology.orchestra.run.smithi133.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-04T07:25:42.276 INFO:teuthology.orchestra.run.smithi133.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 305 kB/s | 20 kB 00:00 2024-03-04T07:25:42.351 INFO:teuthology.orchestra.run.smithi133.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 307 kB 00:00 2024-03-04T07:25:42.376 INFO:teuthology.orchestra.run.smithi133.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 315 kB/s | 32 kB 00:00 2024-03-04T07:25:42.552 INFO:teuthology.orchestra.run.smithi133.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 98 kB/s | 17 kB 00:00 2024-03-04T07:25:42.694 INFO:teuthology.orchestra.run.smithi133.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.7 MB/s | 3.0 MB 00:00 2024-03-04T07:25:42.719 INFO:teuthology.orchestra.run.smithi133.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 60 kB/s | 10 kB 00:00 2024-03-04T07:25:42.761 INFO:teuthology.orchestra.run.smithi133.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-04T07:25:42.786 INFO:teuthology.orchestra.run.smithi133.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 662 kB/s | 44 kB 00:00 2024-03-04T07:25:42.845 INFO:teuthology.orchestra.run.smithi133.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-04T07:25:42.887 INFO:teuthology.orchestra.run.smithi133.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 578 kB 00:00 2024-03-04T07:25:42.912 INFO:teuthology.orchestra.run.smithi133.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 368 kB/s | 24 kB 00:00 2024-03-04T07:25:42.937 INFO:teuthology.orchestra.run.smithi133.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-04T07:25:42.962 INFO:teuthology.orchestra.run.smithi133.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-04T07:25:42.988 INFO:teuthology.orchestra.run.smithi133.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-04T07:25:43.005 INFO:teuthology.orchestra.run.smithi133.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 827 kB/s | 34 kB 00:00 2024-03-04T07:25:43.032 INFO:teuthology.orchestra.run.smithi116.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 988 kB/s | 888 kB 00:00 2024-03-04T07:25:43.190 INFO:teuthology.orchestra.run.smithi116.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 921 kB/s | 145 kB 00:00 2024-03-04T07:25:43.372 INFO:teuthology.orchestra.run.smithi133.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-03-04T07:25:43.422 INFO:teuthology.orchestra.run.smithi133.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 3.0 MB 00:00 2024-03-04T07:25:43.472 INFO:teuthology.orchestra.run.smithi133.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 16 MB/s | 1.5 MB 00:00 2024-03-04T07:25:43.506 INFO:teuthology.orchestra.run.smithi133.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 9.7 MB/s | 811 kB 00:00 2024-03-04T07:25:43.531 INFO:teuthology.orchestra.run.smithi133.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-04T07:25:43.631 INFO:teuthology.orchestra.run.smithi133.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-03-04T07:25:43.757 INFO:teuthology.orchestra.run.smithi133.stdout:(63/176): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-04T07:25:43.782 INFO:teuthology.orchestra.run.smithi133.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 524 kB/s | 78 kB 00:00 2024-03-04T07:25:43.957 INFO:teuthology.orchestra.run.smithi133.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-04T07:25:43.982 INFO:teuthology.orchestra.run.smithi133.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.1 MB/s | 227 kB 00:00 2024-03-04T07:25:44.008 INFO:teuthology.orchestra.run.smithi133.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 492 kB/s | 24 kB 00:00 2024-03-04T07:25:44.033 INFO:teuthology.orchestra.run.smithi133.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-04T07:25:44.058 INFO:teuthology.orchestra.run.smithi133.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-04T07:25:44.084 INFO:teuthology.orchestra.run.smithi133.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-04T07:25:44.226 INFO:teuthology.orchestra.run.smithi133.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-04T07:25:44.254 INFO:teuthology.orchestra.run.smithi133.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-04T07:25:44.279 INFO:teuthology.orchestra.run.smithi133.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 174 kB 00:00 2024-03-04T07:25:44.305 INFO:teuthology.orchestra.run.smithi133.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-04T07:25:44.322 INFO:teuthology.orchestra.run.smithi133.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 981 kB/s | 41 kB 00:00 2024-03-04T07:25:44.332 INFO:teuthology.orchestra.run.smithi116.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 2.0 MB/s | 2.3 MB 00:01 2024-03-04T07:25:44.347 INFO:teuthology.orchestra.run.smithi133.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-04T07:25:44.372 INFO:teuthology.orchestra.run.smithi133.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-04T07:25:44.398 INFO:teuthology.orchestra.run.smithi133.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-04T07:25:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-04T07:25:44.448 INFO:teuthology.orchestra.run.smithi133.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-04T07:25:44.507 INFO:teuthology.orchestra.run.smithi116.stdout:(5/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.9 MB/s | 5.3 MB 00:02 2024-03-04T07:25:44.549 INFO:teuthology.orchestra.run.smithi133.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 392 kB/s | 39 kB 00:00 2024-03-04T07:25:44.622 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-04T07:25:44.799 INFO:teuthology.orchestra.run.smithi133.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-04T07:25:44.816 INFO:teuthology.orchestra.run.smithi133.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 155 kB/s | 41 kB 00:00 2024-03-04T07:25:44.841 INFO:teuthology.orchestra.run.smithi133.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 215 kB 00:00 2024-03-04T07:25:44.858 INFO:teuthology.orchestra.run.smithi133.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 811 kB/s | 34 kB 00:00 2024-03-04T07:25:44.875 INFO:teuthology.orchestra.run.smithi133.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-04T07:25:44.900 INFO:teuthology.orchestra.run.smithi133.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-04T07:25:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-03-04T07:25:44.917 INFO:teuthology.orchestra.run.smithi133.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-04T07:25:44.943 INFO:teuthology.orchestra.run.smithi133.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-04T07:25:44.960 INFO:teuthology.orchestra.run.smithi133.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-04T07:25:44.976 INFO:teuthology.orchestra.run.smithi133.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-04T07:25:45.002 INFO:teuthology.orchestra.run.smithi133.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-04T07:25:45.027 INFO:teuthology.orchestra.run.smithi133.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-04T07:25:45.053 INFO:teuthology.orchestra.run.smithi133.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-03-04T07:25:45.123 INFO:teuthology.orchestra.run.smithi133.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 194 kB 00:00 2024-03-04T07:25:45.145 INFO:teuthology.orchestra.run.smithi133.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-03-04T07:25:45.170 INFO:teuthology.orchestra.run.smithi133.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 241 kB 00:00 2024-03-04T07:25:45.212 INFO:teuthology.orchestra.run.smithi133.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-04T07:25:45.237 INFO:teuthology.orchestra.run.smithi133.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-04T07:25:45.263 INFO:teuthology.orchestra.run.smithi133.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-04T07:25:45.288 INFO:teuthology.orchestra.run.smithi133.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-04T07:25:45.313 INFO:teuthology.orchestra.run.smithi133.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-04T07:25:45.583 INFO:teuthology.orchestra.run.smithi116.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 4.4 MB/s | 4.7 MB 00:01 2024-03-04T07:25:45.739 INFO:teuthology.orchestra.run.smithi133.stdout:(103/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 5.7 MB/s | 47 MB 00:08 2024-03-04T07:25:45.830 INFO:teuthology.orchestra.run.smithi133.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 43 kB/s | 22 kB 00:00 2024-03-04T07:25:45.914 INFO:teuthology.orchestra.run.smithi133.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 392 kB/s | 245 kB 00:00 2024-03-04T07:25:46.073 INFO:teuthology.orchestra.run.smithi133.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-04T07:25:46.189 INFO:teuthology.orchestra.run.smithi133.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-03-04T07:25:46.215 INFO:teuthology.orchestra.run.smithi133.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-03-04T07:25:46.290 INFO:teuthology.orchestra.run.smithi133.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-04T07:25:46.340 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-04T07:25:46.365 INFO:teuthology.orchestra.run.smithi133.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-04T07:25:46.466 INFO:teuthology.orchestra.run.smithi133.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-04T07:25:46.658 INFO:teuthology.orchestra.run.smithi133.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 788 kB/s | 151 kB 00:00 2024-03-04T07:25:47.491 INFO:teuthology.orchestra.run.smithi133.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.0 MB/s | 838 kB 00:00 2024-03-04T07:25:47.500 INFO:teuthology.orchestra.run.smithi116.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 3.5 MB/s | 20 MB 00:05 2024-03-04T07:25:47.633 INFO:teuthology.orchestra.run.smithi116.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 190 kB/s | 25 kB 00:00 2024-03-04T07:25:48.008 INFO:teuthology.orchestra.run.smithi133.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 1.0 MB/s | 548 kB 00:00 2024-03-04T07:25:48.433 INFO:teuthology.orchestra.run.smithi133.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 102 kB/s | 43 kB 00:00 2024-03-04T07:25:48.516 INFO:teuthology.orchestra.run.smithi133.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 388 kB/s | 32 kB 00:00 2024-03-04T07:25:48.558 INFO:teuthology.orchestra.run.smithi133.stdout:(117/176): abseil-cpp-20211102.0-3.el9.x86_64.r 201 kB/s | 548 kB 00:02 2024-03-04T07:25:48.600 INFO:teuthology.orchestra.run.smithi133.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-03-04T07:25:48.767 INFO:teuthology.orchestra.run.smithi133.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 823 kB/s | 172 kB 00:00 2024-03-04T07:25:48.917 INFO:teuthology.orchestra.run.smithi116.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 2.9 MB/s | 9.8 MB 00:03 2024-03-04T07:25:48.968 INFO:teuthology.orchestra.run.smithi133.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 977 kB/s | 358 kB 00:00 2024-03-04T07:25:49.068 INFO:teuthology.orchestra.run.smithi133.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 353 kB/s | 35 kB 00:00 2024-03-04T07:25:49.150 INFO:teuthology.orchestra.run.smithi116.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 949 kB/s | 220 kB 00:00 2024-03-04T07:25:49.226 INFO:teuthology.orchestra.run.smithi116.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 364 kB/s | 27 kB 00:00 2024-03-04T07:25:49.284 INFO:teuthology.orchestra.run.smithi116.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 544 kB/s | 31 kB 00:00 2024-03-04T07:25:49.318 INFO:teuthology.orchestra.run.smithi133.stdout:(122/176): python3-google-auth-2.28.0-1.el9.noa 810 kB/s | 203 kB 00:00 2024-03-04T07:25:49.670 INFO:teuthology.orchestra.run.smithi116.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.9 MB/s | 747 kB 00:00 2024-03-04T07:25:49.801 INFO:teuthology.orchestra.run.smithi116.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.2 MB/s | 163 kB 00:00 2024-03-04T07:25:49.910 INFO:teuthology.orchestra.run.smithi116.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.2 MB/s | 127 kB 00:00 2024-03-04T07:25:50.160 INFO:teuthology.orchestra.run.smithi116.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 2.0 MB/s | 501 kB 00:00 2024-03-04T07:25:50.611 INFO:teuthology.orchestra.run.smithi133.stdout:(123/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 164 kB/s | 303 kB 00:01 2024-03-04T07:25:50.644 INFO:teuthology.orchestra.run.smithi116.stdout:(18/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 2.9 MB/s | 17 MB 00:05 2024-03-04T07:25:50.703 INFO:teuthology.orchestra.run.smithi116.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 790 kB/s | 45 kB 00:00 2024-03-04T07:25:50.803 INFO:teuthology.orchestra.run.smithi116.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.3 MB/s | 131 kB 00:00 2024-03-04T07:25:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.5 MB/s | 162 kB 00:00 2024-03-04T07:25:51.087 INFO:teuthology.orchestra.run.smithi116.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 1.8 MB/s | 321 kB 00:00 2024-03-04T07:25:51.103 INFO:teuthology.orchestra.run.smithi133.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 1.1 MB/s | 2.0 MB 00:01 2024-03-04T07:25:51.162 INFO:teuthology.orchestra.run.smithi133.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-03-04T07:25:51.220 INFO:teuthology.orchestra.run.smithi133.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-03-04T07:25:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.8 MB/s | 298 kB 00:00 2024-03-04T07:25:51.279 INFO:teuthology.orchestra.run.smithi133.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-03-04T07:25:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.5 MB/s | 99 kB 00:00 2024-03-04T07:25:51.337 INFO:teuthology.orchestra.run.smithi133.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 334 kB/s | 19 kB 00:00 2024-03-04T07:25:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-04T07:25:51.396 INFO:teuthology.orchestra.run.smithi133.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 338 kB/s | 20 kB 00:00 2024-03-04T07:25:51.604 INFO:teuthology.orchestra.run.smithi133.stdout:(130/176): python3-grpcio-tools-1.46.7-10.el9.x 145 kB/s | 144 kB 00:00 2024-03-04T07:25:51.972 INFO:teuthology.orchestra.run.smithi133.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 126 kB/s | 46 kB 00:00 2024-03-04T07:25:52.238 INFO:teuthology.orchestra.run.smithi133.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 1.2 MB/s | 1.0 MB 00:00 2024-03-04T07:25:52.314 INFO:teuthology.orchestra.run.smithi133.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 770 kB/s | 58 kB 00:00 2024-03-04T07:25:52.321 INFO:teuthology.orchestra.run.smithi116.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 2.4 MB/s | 5.1 MB 00:02 2024-03-04T07:25:52.446 INFO:teuthology.orchestra.run.smithi116.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 1.3 MB/s | 171 kB 00:00 2024-03-04T07:25:52.515 INFO:teuthology.orchestra.run.smithi133.stdout:(134/176): python3-more-itertools-8.12.0-2.el9. 145 kB/s | 79 kB 00:00 2024-03-04T07:25:52.555 INFO:teuthology.orchestra.run.smithi116.stdout:(28/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 2.6 MB/s | 3.1 MB 00:01 2024-03-04T07:25:52.580 INFO:teuthology.orchestra.run.smithi116.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1739.g 237 kB/s | 31 kB 00:00 2024-03-04T07:25:52.597 INFO:teuthology.orchestra.run.smithi133.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 961 kB/s | 272 kB 00:00 2024-03-04T07:25:52.630 INFO:teuthology.orchestra.run.smithi116.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.8 MB/s | 138 kB 00:00 2024-03-04T07:25:52.664 INFO:teuthology.orchestra.run.smithi133.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 109 kB/s | 16 kB 00:00 2024-03-04T07:25:52.706 INFO:teuthology.orchestra.run.smithi133.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 828 kB/s | 90 kB 00:00 2024-03-04T07:25:52.773 INFO:teuthology.orchestra.run.smithi133.stdout:(138/176): python3-repoze-lru-0.7-16.el9.noarch 462 kB/s | 31 kB 00:00 2024-03-04T07:25:52.923 INFO:teuthology.orchestra.run.smithi133.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 109 kB/s | 28 kB 00:00 2024-03-04T07:25:52.949 INFO:teuthology.orchestra.run.smithi133.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-03-04T07:25:53.016 INFO:teuthology.orchestra.run.smithi133.stdout:(141/176): python3-tempora-5.0.0-2.el9.noarch.r 538 kB/s | 36 kB 00:00 2024-03-04T07:25:53.108 INFO:teuthology.orchestra.run.smithi133.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 556 kB/s | 51 kB 00:00 2024-03-04T07:25:53.273 INFO:teuthology.orchestra.run.smithi116.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 2.6 MB/s | 1.8 MB 00:00 2024-03-04T07:25:53.317 INFO:teuthology.orchestra.run.smithi133.stdout:(143/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-03-04T07:25:53.381 INFO:teuthology.orchestra.run.smithi116.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 2.3 MB/s | 248 kB 00:00 2024-03-04T07:25:53.383 INFO:teuthology.orchestra.run.smithi133.stdout:(144/176): python3-rsa-4.9-2.el9.noarch.rpm 129 kB/s | 59 kB 00:00 2024-03-04T07:25:53.417 INFO:teuthology.orchestra.run.smithi133.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-03-04T07:25:53.448 INFO:teuthology.orchestra.run.smithi116.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 753 kB/s | 49 kB 00:00 2024-03-04T07:25:53.467 INFO:teuthology.orchestra.run.smithi133.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 400 kB/s | 20 kB 00:00 2024-03-04T07:25:53.535 INFO:teuthology.orchestra.run.smithi116.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 178 kB/s | 15 kB 00:00 2024-03-04T07:25:53.642 INFO:teuthology.orchestra.run.smithi133.stdout:(147/176): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-04T07:25:53.690 INFO:teuthology.orchestra.run.smithi116.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.7 MB/s | 264 kB 00:00 2024-03-04T07:25:54.049 INFO:teuthology.orchestra.run.smithi116.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 2.0 MB/s | 748 kB 00:00 2024-03-04T07:25:54.116 INFO:teuthology.orchestra.run.smithi116.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 277 kB/s | 18 kB 00:00 2024-03-04T07:25:54.308 INFO:teuthology.orchestra.run.smithi116.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-04T07:25:54.375 INFO:teuthology.orchestra.run.smithi116.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-04T07:25:54.409 INFO:teuthology.orchestra.run.smithi116.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.2 MB/s | 106 kB 00:00 2024-03-04T07:25:54.442 INFO:teuthology.orchestra.run.smithi116.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 625 kB/s | 20 kB 00:00 2024-03-04T07:25:54.484 INFO:teuthology.orchestra.run.smithi116.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2024-03-04T07:25:54.540 INFO:teuthology.orchestra.run.smithi116.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 575 kB/s | 32 kB 00:00 2024-03-04T07:25:54.701 INFO:teuthology.orchestra.run.smithi116.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-03-04T07:25:54.743 INFO:teuthology.orchestra.run.smithi116.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 420 kB/s | 17 kB 00:00 2024-03-04T07:25:54.767 INFO:teuthology.orchestra.run.smithi133.stdout:(148/176): thrift-0.15.0-2.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-03-04T07:25:55.226 INFO:teuthology.orchestra.run.smithi133.stdout:(149/176): librados2-19.0.0-1739.g1bc250c7.el9. 7.4 MB/s | 3.4 MB 00:00 2024-03-04T07:25:55.285 INFO:teuthology.orchestra.run.smithi116.stdout:(46/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 6.2 MB/s | 47 MB 00:07 2024-03-04T07:25:55.302 INFO:teuthology.orchestra.run.smithi116.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 18 kB/s | 10 kB 00:00 2024-03-04T07:25:55.327 INFO:teuthology.orchestra.run.smithi116.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-04T07:25:55.353 INFO:teuthology.orchestra.run.smithi116.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 918 kB/s | 44 kB 00:00 2024-03-04T07:25:55.394 INFO:teuthology.orchestra.run.smithi116.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-04T07:25:55.428 INFO:teuthology.orchestra.run.smithi116.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 8.2 MB/s | 578 kB 00:00 2024-03-04T07:25:55.453 INFO:teuthology.orchestra.run.smithi116.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 456 kB/s | 24 kB 00:00 2024-03-04T07:25:55.479 INFO:teuthology.orchestra.run.smithi116.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 46 kB 00:00 2024-03-04T07:25:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:(150/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 12 MB/s | 3.1 MB 00:00 2024-03-04T07:25:55.512 INFO:teuthology.orchestra.run.smithi116.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-04T07:25:55.527 INFO:teuthology.orchestra.run.smithi133.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-04T07:25:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.3 MB/s | 187 kB 00:00 2024-03-04T07:25:55.595 INFO:teuthology.orchestra.run.smithi116.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 426 kB/s | 34 kB 00:00 2024-03-04T07:25:55.711 INFO:teuthology.orchestra.run.smithi133.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-04T07:25:55.747 INFO:teuthology.orchestra.run.smithi116.stdout:(57/176): ceph-mgr-diskprediction-local-19.0.0- 2.4 MB/s | 7.4 MB 00:03 2024-03-04T07:25:55.872 INFO:teuthology.orchestra.run.smithi116.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-03-04T07:25:55.972 INFO:teuthology.orchestra.run.smithi116.stdout:(59/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 3.0 MB 00:00 2024-03-04T07:25:55.986 INFO:teuthology.orchestra.run.smithi133.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 6.6 MB/s | 1.8 MB 00:00 2024-03-04T07:25:56.011 INFO:teuthology.orchestra.run.smithi133.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.6 MB/s | 41 kB 00:00 2024-03-04T07:25:56.031 INFO:teuthology.orchestra.run.smithi116.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 5.6 MB/s | 1.5 MB 00:00 2024-03-04T07:25:56.064 INFO:teuthology.orchestra.run.smithi116.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 4.8 MB/s | 811 kB 00:00 2024-03-04T07:25:56.070 INFO:teuthology.orchestra.run.smithi133.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.1 MB/s | 247 kB 00:00 2024-03-04T07:25:56.095 INFO:teuthology.orchestra.run.smithi133.stdout:(156/176): python3-werkzeug-2.0.3-3.el9.1.noarc 158 kB/s | 427 kB 00:02 2024-03-04T07:25:56.106 INFO:teuthology.orchestra.run.smithi116.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.0 MB/s | 134 kB 00:00 2024-03-04T07:25:56.120 INFO:teuthology.orchestra.run.smithi133.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 69 kB 00:00 2024-03-04T07:25:56.145 INFO:teuthology.orchestra.run.smithi133.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 705 kB/s | 35 kB 00:00 2024-03-04T07:25:56.165 INFO:teuthology.orchestra.run.smithi116.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-03-04T07:25:56.171 INFO:teuthology.orchestra.run.smithi133.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 191 kB 00:00 2024-03-04T07:25:56.196 INFO:teuthology.orchestra.run.smithi133.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-03-04T07:25:56.263 INFO:teuthology.orchestra.run.smithi133.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 8.6 MB/s | 802 kB 00:00 2024-03-04T07:25:56.288 INFO:teuthology.orchestra.run.smithi133.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 4.6 MB/s | 433 kB 00:00 2024-03-04T07:25:56.313 INFO:teuthology.orchestra.run.smithi133.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-03-04T07:25:56.323 INFO:teuthology.orchestra.run.smithi116.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-04T07:25:56.339 INFO:teuthology.orchestra.run.smithi133.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-04T07:25:56.348 INFO:teuthology.orchestra.run.smithi116.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 331 kB/s | 78 kB 00:00 2024-03-04T07:25:56.364 INFO:teuthology.orchestra.run.smithi133.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-03-04T07:25:56.389 INFO:teuthology.orchestra.run.smithi133.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-04T07:25:56.448 INFO:teuthology.orchestra.run.smithi133.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 755 kB/s | 63 kB 00:00 2024-03-04T07:25:56.473 INFO:teuthology.orchestra.run.smithi133.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 330 kB/s | 28 kB 00:00 2024-03-04T07:25:56.482 INFO:teuthology.orchestra.run.smithi116.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-04T07:25:56.516 INFO:teuthology.orchestra.run.smithi116.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 227 kB 00:00 2024-03-04T07:25:56.541 INFO:teuthology.orchestra.run.smithi116.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 137 kB/s | 24 kB 00:00 2024-03-04T07:25:56.548 INFO:teuthology.orchestra.run.smithi133.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-03-04T07:25:56.566 INFO:teuthology.orchestra.run.smithi116.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 219 kB/s | 18 kB 00:00 2024-03-04T07:25:56.616 INFO:teuthology.orchestra.run.smithi116.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-04T07:25:56.642 INFO:teuthology.orchestra.run.smithi116.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 319 kB/s | 32 kB 00:00 2024-03-04T07:25:56.666 INFO:teuthology.orchestra.run.smithi133.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-04T07:25:56.766 INFO:teuthology.orchestra.run.smithi133.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-04T07:25:56.767 INFO:teuthology.orchestra.run.smithi116.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-04T07:25:56.799 INFO:teuthology.orchestra.run.smithi133.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 3.3 MB/s | 438 kB 00:00 2024-03-04T07:25:56.801 INFO:teuthology.orchestra.run.smithi116.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-04T07:25:56.825 INFO:teuthology.orchestra.run.smithi133.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 521 kB/s | 30 kB 00:00 2024-03-04T07:25:56.825 INFO:teuthology.orchestra.run.smithi116.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-04T07:25:56.850 INFO:teuthology.orchestra.run.smithi133.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-04T07:25:56.851 INFO:teuthology.orchestra.run.smithi116.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 854 kB/s | 69 kB 00:00 2024-03-04T07:25:56.876 INFO:teuthology.orchestra.run.smithi116.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 41 kB 00:00 2024-03-04T07:25:56.901 INFO:teuthology.orchestra.run.smithi116.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 238 kB 00:00 2024-03-04T07:25:56.917 INFO:teuthology.orchestra.run.smithi133.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-03-04T07:25:56.926 INFO:teuthology.orchestra.run.smithi116.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 132 kB 00:00 2024-03-04T07:25:56.952 INFO:teuthology.orchestra.run.smithi116.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 982 kB/s | 47 kB 00:00 2024-03-04T07:25:56.986 INFO:teuthology.orchestra.run.smithi116.stdout:(80/176): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-04T07:25:57.011 INFO:teuthology.orchestra.run.smithi116.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 517 kB/s | 43 kB 00:00 2024-03-04T07:25:57.494 INFO:teuthology.orchestra.run.smithi116.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-04T07:25:57.519 INFO:teuthology.orchestra.run.smithi116.stdout:(83/176): python3-toml-0.10.2-6.el9.noarch.rpm 78 kB/s | 39 kB 00:00 2024-03-04T07:25:57.545 INFO:teuthology.orchestra.run.smithi116.stdout:(84/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 79 kB/s | 41 kB 00:00 2024-03-04T07:25:57.570 INFO:teuthology.orchestra.run.smithi116.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 695 kB/s | 34 kB 00:00 2024-03-04T07:25:57.620 INFO:teuthology.orchestra.run.smithi116.stdout:(86/176): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 215 kB 00:00 2024-03-04T07:25:57.645 INFO:teuthology.orchestra.run.smithi116.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-03-04T07:25:57.679 INFO:teuthology.orchestra.run.smithi116.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 300 kB 00:00 2024-03-04T07:25:57.704 INFO:teuthology.orchestra.run.smithi116.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2024-03-04T07:25:57.730 INFO:teuthology.orchestra.run.smithi116.stdout:(90/176): uuid-1.6.2-55.el9.x86_64.rpm 701 kB/s | 56 kB 00:00 2024-03-04T07:25:57.755 INFO:teuthology.orchestra.run.smithi116.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 852 kB/s | 63 kB 00:00 2024-03-04T07:25:57.813 INFO:teuthology.orchestra.run.smithi116.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-03-04T07:25:57.939 INFO:teuthology.orchestra.run.smithi116.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 431 kB/s | 79 kB 00:00 2024-03-04T07:25:57.964 INFO:teuthology.orchestra.run.smithi116.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 898 kB/s | 186 kB 00:00 2024-03-04T07:25:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 802 kB 00:00 2024-03-04T07:25:58.047 INFO:teuthology.orchestra.run.smithi116.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-04T07:25:58.081 INFO:teuthology.orchestra.run.smithi116.stdout:(97/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 194 kB 00:00 2024-03-04T07:25:58.123 INFO:teuthology.orchestra.run.smithi116.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-04T07:25:58.174 INFO:teuthology.orchestra.run.smithi116.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 9.4 MB/s | 1.1 MB 00:00 2024-03-04T07:25:58.199 INFO:teuthology.orchestra.run.smithi116.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-04T07:25:58.232 INFO:teuthology.orchestra.run.smithi116.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.2 MB/s | 124 kB 00:00 2024-03-04T07:25:58.258 INFO:teuthology.orchestra.run.smithi116.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.4 MB/s | 113 kB 00:00 2024-03-04T07:25:58.291 INFO:teuthology.orchestra.run.smithi116.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 3.2 MB/s | 187 kB 00:00 2024-03-04T07:25:58.441 INFO:teuthology.orchestra.run.smithi116.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 123 kB/s | 22 kB 00:00 2024-03-04T07:25:58.542 INFO:teuthology.orchestra.run.smithi116.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-03-04T07:25:58.676 INFO:teuthology.orchestra.run.smithi116.stdout:(106/176): python3-devel-3.9.17-2.el9.x86_64.rp 557 kB/s | 245 kB 00:00 2024-03-04T07:25:59.093 INFO:teuthology.orchestra.run.smithi116.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 47 kB/s | 19 kB 00:00 2024-03-04T07:25:59.268 INFO:teuthology.orchestra.run.smithi116.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 666 kB/s | 548 kB 00:00 2024-03-04T07:25:59.318 INFO:teuthology.orchestra.run.smithi116.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 436 kB/s | 309 kB 00:00 2024-03-04T07:25:59.376 INFO:teuthology.orchestra.run.smithi116.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-03-04T07:25:59.427 INFO:teuthology.orchestra.run.smithi116.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-04T07:25:59.494 INFO:teuthology.orchestra.run.smithi116.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 580 kB/s | 67 kB 00:00 2024-03-04T07:25:59.561 INFO:teuthology.orchestra.run.smithi116.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-04T07:25:59.719 INFO:teuthology.orchestra.run.smithi116.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-03-04T07:25:59.778 INFO:teuthology.orchestra.run.smithi116.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-03-04T07:25:59.820 INFO:teuthology.orchestra.run.smithi116.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 438 kB/s | 43 kB 00:00 2024-03-04T07:25:59.912 INFO:teuthology.orchestra.run.smithi116.stdout:(117/176): libarrow-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 4.4 MB 00:00 2024-03-04T07:25:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2024-03-04T07:25:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:(119/176): python3-cachetools-4.2.4-1.el9.noarc 184 kB/s | 32 kB 00:00 2024-03-04T07:26:00.029 INFO:teuthology.orchestra.run.smithi116.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-04T07:26:00.063 INFO:teuthology.orchestra.run.smithi116.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-04T07:26:00.096 INFO:teuthology.orchestra.run.smithi116.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-04T07:26:00.155 INFO:teuthology.orchestra.run.smithi116.stdout:(123/176): python3-colorama-0.4.4-12.el9.noarch 282 kB/s | 35 kB 00:00 2024-03-04T07:26:00.189 INFO:teuthology.orchestra.run.smithi116.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-04T07:26:00.280 INFO:teuthology.orchestra.run.smithi116.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-04T07:26:00.306 INFO:teuthology.orchestra.run.smithi116.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-03-04T07:26:00.389 INFO:teuthology.orchestra.run.smithi116.stdout:(127/176): python3-grpcio-1.46.7-10.el9.x86_64. 7.0 MB/s | 2.0 MB 00:00 2024-03-04T07:26:00.415 INFO:teuthology.orchestra.run.smithi116.stdout:(128/176): python3-jaraco-classes-3.2.1-5.el9.n 133 kB/s | 18 kB 00:00 2024-03-04T07:26:00.440 INFO:teuthology.orchestra.run.smithi116.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 174 kB/s | 23 kB 00:00 2024-03-04T07:26:00.507 INFO:teuthology.orchestra.run.smithi116.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 167 kB/s | 19 kB 00:00 2024-03-04T07:26:00.549 INFO:teuthology.orchestra.run.smithi116.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 183 kB/s | 20 kB 00:00 2024-03-04T07:26:00.607 INFO:teuthology.orchestra.run.smithi116.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-03-04T07:26:00.666 INFO:teuthology.orchestra.run.smithi116.stdout:(133/176): python3-logutils-0.3.5-21.el9.noarch 292 kB/s | 46 kB 00:00 2024-03-04T07:26:00.683 INFO:teuthology.orchestra.run.smithi116.stdout:(134/176): python3-more-itertools-8.12.0-2.el9. 591 kB/s | 79 kB 00:00 2024-03-04T07:26:00.716 INFO:teuthology.orchestra.run.smithi116.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 534 kB/s | 58 kB 00:00 2024-03-04T07:26:00.784 INFO:teuthology.orchestra.run.smithi116.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-04T07:26:00.809 INFO:teuthology.orchestra.run.smithi116.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-03-04T07:26:00.834 INFO:teuthology.orchestra.run.smithi116.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 770 kB/s | 90 kB 00:00 2024-03-04T07:26:00.893 INFO:teuthology.orchestra.run.smithi116.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 261 kB/s | 28 kB 00:00 2024-03-04T07:26:00.918 INFO:teuthology.orchestra.run.smithi116.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-03-04T07:26:00.951 INFO:teuthology.orchestra.run.smithi116.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-04T07:26:01.002 INFO:teuthology.orchestra.run.smithi116.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 546 kB/s | 59 kB 00:00 2024-03-04T07:26:01.044 INFO:teuthology.orchestra.run.smithi116.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 286 kB/s | 36 kB 00:00 2024-03-04T07:26:01.077 INFO:teuthology.orchestra.run.smithi116.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-03-04T07:26:01.169 INFO:teuthology.orchestra.run.smithi116.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-04T07:26:01.194 INFO:teuthology.orchestra.run.smithi116.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 598 kB/s | 90 kB 00:00 2024-03-04T07:26:01.228 INFO:teuthology.orchestra.run.smithi116.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-03-04T07:26:01.295 INFO:teuthology.orchestra.run.smithi116.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2024-03-04T07:26:01.337 INFO:teuthology.orchestra.run.smithi116.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-04T07:26:01.420 INFO:teuthology.orchestra.run.smithi116.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2024-03-04T07:26:01.471 INFO:teuthology.orchestra.run.smithi116.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 68 kB 00:00 2024-03-04T07:26:01.612 INFO:teuthology.orchestra.run.smithi116.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-03-04T07:26:01.721 INFO:teuthology.orchestra.run.smithi116.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-04T07:26:01.747 INFO:teuthology.orchestra.run.smithi116.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.7 MB/s | 41 kB 00:00 2024-03-04T07:26:01.788 INFO:teuthology.orchestra.run.smithi116.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-03-04T07:26:01.830 INFO:teuthology.orchestra.run.smithi116.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 69 kB 00:00 2024-03-04T07:26:01.856 INFO:teuthology.orchestra.run.smithi116.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-03-04T07:26:01.898 INFO:teuthology.orchestra.run.smithi116.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-03-04T07:26:01.939 INFO:teuthology.orchestra.run.smithi116.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.1 MB/s | 254 kB 00:00 2024-03-04T07:26:01.998 INFO:teuthology.orchestra.run.smithi116.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 14 MB/s | 802 kB 00:00 2024-03-04T07:26:02.057 INFO:teuthology.orchestra.run.smithi116.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 7.4 MB/s | 433 kB 00:00 2024-03-04T07:26:02.099 INFO:teuthology.orchestra.run.smithi116.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 538 kB/s | 22 kB 00:00 2024-03-04T07:26:02.154 INFO:teuthology.orchestra.run.smithi116.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 1.9 MB/s | 107 kB 00:00 2024-03-04T07:26:02.204 INFO:teuthology.orchestra.run.smithi116.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 154 kB 00:00 2024-03-04T07:26:02.238 INFO:teuthology.orchestra.run.smithi116.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 135 kB 00:00 2024-03-04T07:26:02.272 INFO:teuthology.orchestra.run.smithi116.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.9 MB/s | 63 kB 00:00 2024-03-04T07:26:02.305 INFO:teuthology.orchestra.run.smithi116.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 848 kB/s | 28 kB 00:00 2024-03-04T07:26:02.372 INFO:teuthology.orchestra.run.smithi116.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-03-04T07:26:02.489 INFO:teuthology.orchestra.run.smithi116.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 19 MB/s | 2.2 MB 00:00 2024-03-04T07:26:02.598 INFO:teuthology.orchestra.run.smithi116.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-04T07:26:02.665 INFO:teuthology.orchestra.run.smithi116.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 6.4 MB/s | 438 kB 00:00 2024-03-04T07:26:02.698 INFO:teuthology.orchestra.run.smithi116.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 933 kB/s | 30 kB 00:00 2024-03-04T07:26:02.741 INFO:teuthology.orchestra.run.smithi116.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-04T07:26:02.807 INFO:teuthology.orchestra.run.smithi116.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-04T07:26:03.874 INFO:teuthology.orchestra.run.smithi116.stdout:(175/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.2 MB/s | 3.1 MB 00:02 2024-03-04T07:26:04.049 INFO:teuthology.orchestra.run.smithi116.stdout:(176/176): librados2-19.0.0-1739.g1bc250c7.el9. 1.2 MB/s | 3.4 MB 00:02 2024-03-04T07:26:04.065 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T07:26:04.065 INFO:teuthology.orchestra.run.smithi116.stdout:Total 9.7 MB/s | 224 MB 00:22 2024-03-04T07:26:04.196 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-04T07:26:04.568 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-04T07:26:04.569 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-04T07:26:06.894 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-04T07:26:06.894 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-04T07:26:08.267 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 962 kB/s | 20 MB 00:21 2024-03-04T07:26:09.883 INFO:teuthology.orchestra.run.smithi133.stdout:(176/176): libarrow-9.0.0-10.el9.x86_64.rpm 191 kB/s | 4.4 MB 00:23 2024-03-04T07:26:09.888 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-03-04T07:26:09.888 INFO:teuthology.orchestra.run.smithi133.stdout:Total 6.3 MB/s | 224 MB 00:35 2024-03-04T07:26:09.935 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-03-04T07:26:10.220 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-03-04T07:26:10.220 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-03-04T07:26:11.990 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-03-04T07:26:11.990 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-03-04T07:26:13.264 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-04T07:26:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-04T07:26:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T07:26:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T07:26:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T07:26:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T07:26:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T07:26:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T07:26:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T07:26:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T07:26:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T07:26:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T07:26:15.786 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T07:26:15.787 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T07:26:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T07:26:15.802 INFO:teuthology.orchestra.run.smithi152.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T07:26:15.802 INFO:teuthology.orchestra.run.smithi152.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T07:26:15.804 INFO:teuthology.orchestra.run.smithi152.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T07:26:15.805 INFO:teuthology.orchestra.run.smithi152.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T07:26:15.807 INFO:teuthology.orchestra.run.smithi152.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T07:26:15.810 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T07:26:15.811 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T07:26:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:26:15.894 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-04T07:26:15.909 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-04T07:26:15.909 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-04T07:26:15.909 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-04T07:26:15.910 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-04T07:26:15.911 INFO:teuthology.orchestra.run.smithi152.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-04T07:26:15.912 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T07:26:15.913 INFO:teuthology.orchestra.run.smithi152.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T07:26:15.914 INFO:teuthology.orchestra.run.smithi152.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-04T07:26:15.915 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-04T07:26:15.916 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-04T07:26:15.916 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-04T07:26:15.916 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-04T07:26:15.916 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-04T07:26:15.916 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-04T07:26:15.916 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T07:26:15.916 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T07:26:15.917 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T07:26:15.918 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T07:26:15.919 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T07:26:15.920 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T07:26:15.921 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T07:26:15.922 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T07:26:15.923 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T07:26:15.924 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout:============================================================================================ 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout:Install 150 Packages 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 26 Packages 2024-03-04T07:26:15.925 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:26:15.926 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 224 M 2024-03-04T07:26:15.926 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-04T07:26:16.024 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:26:16.050 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:26:16.178 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:26:16.220 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T07:26:16.467 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T07:26:16.522 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T07:26:16.547 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:26:16.648 INFO:teuthology.orchestra.run.smithi152.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-03-04T07:26:16.693 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:26:16.723 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T07:26:16.813 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T07:26:16.875 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T07:26:16.900 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T07:26:16.964 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T07:26:17.219 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T07:26:17.265 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T07:26:17.317 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T07:26:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T07:26:17.824 INFO:teuthology.orchestra.run.smithi152.stdout:(2/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 3.5 MB/s | 5.3 MB 00:01 2024-03-04T07:26:17.857 INFO:teuthology.orchestra.run.smithi152.stdout:(3/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 735 kB/s | 888 kB 00:01 2024-03-04T07:26:17.865 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T07:26:17.916 INFO:teuthology.orchestra.run.smithi152.stdout:(4/176): ceph-immutable-object-cache-19.0.0-173 1.5 MB/s | 145 kB 00:00 2024-03-04T07:26:17.979 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T07:26:18.166 INFO:teuthology.orchestra.run.smithi152.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 6.0 MB/s | 1.5 MB 00:00 2024-03-04T07:26:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:18.260 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:18.743 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T07:26:18.816 INFO:teuthology.orchestra.run.smithi152.stdout:(6/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 7.2 MB/s | 4.7 MB 00:00 2024-03-04T07:26:18.872 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T07:26:18.877 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-03-04T07:26:18.909 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T07:26:19.003 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T07:26:19.033 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T07:26:19.037 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T07:26:19.076 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T07:26:19.096 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T07:26:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T07:26:19.130 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T07:26:19.138 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T07:26:19.176 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T07:26:19.183 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T07:26:19.230 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T07:26:19.232 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T07:26:19.264 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T07:26:19.289 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T07:26:19.322 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T07:26:19.333 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T07:26:19.382 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T07:26:19.446 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T07:26:19.511 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T07:26:19.544 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T07:26:19.660 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T07:26:19.661 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T07:26:19.697 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-04T07:26:19.715 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:26:19.751 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-04T07:26:19.788 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-04T07:26:19.837 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:26:19.866 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:26:19.907 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T07:26:19.941 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:26:19.964 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T07:26:19.971 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T07:26:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T07:26:20.119 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T07:26:20.167 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T07:26:20.191 INFO:teuthology.orchestra.run.smithi152.stdout:(7/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.0 MB/s | 2.3 MB 00:02 2024-03-04T07:26:20.195 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:26:20.251 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T07:26:20.299 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:26:20.318 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T07:26:20.329 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T07:26:20.393 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T07:26:20.433 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T07:26:20.456 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T07:26:20.471 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T07:26:20.648 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T07:26:20.678 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T07:26:20.710 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T07:26:20.809 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T07:26:20.897 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T07:26:20.956 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T07:26:21.016 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T07:26:21.018 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:21.082 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:21.123 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T07:26:21.263 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T07:26:21.277 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:21.308 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T07:26:21.334 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T07:26:21.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T07:26:21.397 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T07:26:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-04T07:26:21.488 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T07:26:21.491 INFO:teuthology.orchestra.run.smithi152.stdout:(8/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 6.3 MB/s | 17 MB 00:02 2024-03-04T07:26:21.511 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T07:26:21.520 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T07:26:21.587 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T07:26:21.592 INFO:teuthology.orchestra.run.smithi152.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 252 kB/s | 25 kB 00:00 2024-03-04T07:26:21.633 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T07:26:21.665 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T07:26:21.692 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T07:26:21.700 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T07:26:21.740 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T07:26:21.752 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T07:26:21.803 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T07:26:21.839 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T07:26:21.882 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T07:26:21.915 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T07:26:21.937 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T07:26:22.005 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T07:26:22.050 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-04T07:26:22.085 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-04T07:26:22.114 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-04T07:26:22.175 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T07:26:22.181 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T07:26:22.228 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T07:26:22.300 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T07:26:22.429 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T07:26:22.432 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T07:26:22.480 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T07:26:22.531 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T07:26:22.602 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T07:26:22.806 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T07:26:22.860 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T07:26:22.925 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T07:26:22.991 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T07:26:23.011 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T07:26:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T07:26:23.075 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T07:26:23.099 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T07:26:23.150 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T07:26:23.167 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T07:26:23.187 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T07:26:23.194 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T07:26:23.207 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T07:26:23.234 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T07:26:23.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T07:26:23.271 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-04T07:26:23.303 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T07:26:23.326 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T07:26:23.390 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T07:26:23.437 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T07:26:23.515 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T07:26:23.580 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T07:26:23.618 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T07:26:23.661 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T07:26:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T07:26:23.807 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T07:26:23.830 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T07:26:23.850 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T07:26:23.894 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T07:26:23.906 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T07:26:23.932 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T07:26:23.950 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T07:26:23.965 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T07:26:24.088 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T07:26:24.124 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T07:26:24.136 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T07:26:24.190 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T07:26:24.255 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T07:26:24.287 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T07:26:24.307 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T07:26:24.323 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T07:26:24.378 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T07:26:24.400 INFO:teuthology.orchestra.run.smithi152.stdout:(10/176): ceph-common-19.0.0-1739.g1bc250c7.el9 2.5 MB/s | 20 MB 00:08 2024-03-04T07:26:24.414 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T07:26:24.448 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T07:26:24.465 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T07:26:24.506 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T07:26:24.516 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T07:26:24.583 INFO:teuthology.orchestra.run.smithi152.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.2 MB/s | 220 kB 00:00 2024-03-04T07:26:24.596 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T07:26:24.648 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T07:26:24.673 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T07:26:24.684 INFO:teuthology.orchestra.run.smithi152.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 270 kB/s | 27 kB 00:00 2024-03-04T07:26:24.695 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T07:26:24.726 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T07:26:24.767 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T07:26:24.767 INFO:teuthology.orchestra.run.smithi152.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 376 kB/s | 31 kB 00:00 2024-03-04T07:26:24.860 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T07:26:24.892 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T07:26:24.916 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T07:26:24.932 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T07:26:24.958 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T07:26:24.971 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T07:26:24.983 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T07:26:25.026 INFO:teuthology.orchestra.run.smithi152.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 2.8 MB/s | 747 kB 00:00 2024-03-04T07:26:25.045 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T07:26:25.118 INFO:teuthology.orchestra.run.smithi152.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.7 MB/s | 163 kB 00:00 2024-03-04T07:26:25.140 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T07:26:25.173 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T07:26:25.185 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T07:26:25.210 INFO:teuthology.orchestra.run.smithi152.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.3 MB/s | 127 kB 00:00 2024-03-04T07:26:25.226 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T07:26:25.267 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T07:26:25.272 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T07:26:25.315 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T07:26:25.315 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T07:26:25.342 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T07:26:25.363 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T07:26:25.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T07:26:25.421 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T07:26:25.426 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T07:26:25.444 INFO:teuthology.orchestra.run.smithi152.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 2.1 MB/s | 501 kB 00:00 2024-03-04T07:26:25.487 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T07:26:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T07:26:25.593 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T07:26:25.616 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T07:26:25.683 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T07:26:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T07:26:25.773 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T07:26:25.812 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T07:26:25.815 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T07:26:25.850 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T07:26:25.870 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T07:26:25.883 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T07:26:25.910 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T07:26:25.915 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T07:26:25.950 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T07:26:25.981 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T07:26:26.009 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T07:26:26.042 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T07:26:26.063 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T07:26:26.099 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T07:26:26.130 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T07:26:26.161 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T07:26:26.164 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T07:26:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T07:26:26.235 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T07:26:26.251 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T07:26:26.271 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T07:26:26.304 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T07:26:26.306 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T07:26:26.345 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T07:26:26.372 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T07:26:26.445 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T07:26:26.458 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T07:26:26.516 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T07:26:26.559 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T07:26:26.755 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T07:26:26.893 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T07:26:27.015 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T07:26:27.136 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T07:26:27.185 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T07:26:27.493 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T07:26:27.540 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T07:26:27.606 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T07:26:27.619 INFO:teuthology.orchestra.run.smithi152.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 2.4 MB/s | 5.1 MB 00:02 2024-03-04T07:26:27.674 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T07:26:27.719 INFO:teuthology.orchestra.run.smithi152.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 452 kB/s | 45 kB 00:00 2024-03-04T07:26:27.797 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T07:26:27.841 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T07:26:27.894 INFO:teuthology.orchestra.run.smithi152.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 750 kB/s | 131 kB 00:00 2024-03-04T07:26:27.907 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T07:26:27.958 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T07:26:27.991 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T07:26:28.039 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T07:26:28.045 INFO:teuthology.orchestra.run.smithi152.stdout:(21/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 1.2 MB/s | 9.8 MB 00:07 2024-03-04T07:26:28.070 INFO:teuthology.orchestra.run.smithi152.stdout:(22/176): python3-cephfs-19.0.0-1739.g1bc250c7. 921 kB/s | 162 kB 00:00 2024-03-04T07:26:28.082 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T07:26:28.125 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T07:26:28.130 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T07:26:28.158 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T07:26:28.226 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T07:26:28.318 INFO:teuthology.orchestra.run.smithi152.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.2 MB/s | 298 kB 00:00 2024-03-04T07:26:28.336 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T07:26:28.345 INFO:teuthology.orchestra.run.smithi152.stdout:(24/176): python3-rados-19.0.0-1739.g1bc250c7.e 1.0 MB/s | 321 kB 00:00 2024-03-04T07:26:28.364 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T07:26:28.413 INFO:teuthology.orchestra.run.smithi152.stdout:(25/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.0 MB/s | 99 kB 00:00 2024-03-04T07:26:28.442 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T07:26:28.455 INFO:teuthology.orchestra.run.smithi152.stdout:(26/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 784 kB/s | 85 kB 00:00 2024-03-04T07:26:28.638 INFO:teuthology.orchestra.run.smithi152.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 931 kB/s | 171 kB 00:00 2024-03-04T07:26:28.738 INFO:teuthology.orchestra.run.smithi152.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 314 kB/s | 31 kB 00:00 2024-03-04T07:26:28.930 INFO:teuthology.orchestra.run.smithi152.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 720 kB/s | 138 kB 00:00 2024-03-04T07:26:29.005 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T07:26:29.073 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T07:26:29.142 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T07:26:29.187 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T07:26:29.672 INFO:teuthology.orchestra.run.smithi152.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 2.5 MB/s | 3.1 MB 00:01 2024-03-04T07:26:30.240 INFO:teuthology.orchestra.run.smithi152.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.4 MB/s | 1.8 MB 00:01 2024-03-04T07:26:30.398 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T07:26:30.484 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T07:26:30.489 INFO:teuthology.orchestra.run.smithi152.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 994 kB/s | 248 kB 00:00 2024-03-04T07:26:30.511 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T07:26:30.615 INFO:teuthology.orchestra.run.smithi152.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 394 kB/s | 49 kB 00:00 2024-03-04T07:26:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T07:26:30.723 INFO:teuthology.orchestra.run.smithi152.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 141 kB/s | 15 kB 00:00 2024-03-04T07:26:30.744 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T07:26:30.779 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T07:26:30.824 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T07:26:30.849 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T07:26:30.939 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T07:26:30.970 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-04T07:26:30.990 INFO:teuthology.orchestra.run.smithi152.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 990 kB/s | 264 kB 00:00 2024-03-04T07:26:31.007 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T07:26:31.136 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T07:26:31.173 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T07:26:31.292 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T07:26:31.321 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-04T07:26:31.375 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T07:26:31.460 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T07:26:31.474 INFO:teuthology.orchestra.run.smithi152.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.5 MB/s | 748 kB 00:00 2024-03-04T07:26:31.485 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T07:26:31.556 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T07:26:31.566 INFO:teuthology.orchestra.run.smithi152.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 199 kB/s | 18 kB 00:00 2024-03-04T07:26:31.578 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-04T07:26:31.791 INFO:teuthology.orchestra.run.smithi152.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 662 kB 00:00 2024-03-04T07:26:31.879 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T07:26:31.908 INFO:teuthology.orchestra.run.smithi152.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 7.7 MB/s | 920 kB 00:00 2024-03-04T07:26:31.909 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T07:26:31.967 INFO:teuthology.orchestra.run.smithi152.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-04T07:26:31.991 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T07:26:32.017 INFO:teuthology.orchestra.run.smithi152.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 407 kB/s | 20 kB 00:00 2024-03-04T07:26:32.019 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-04T07:26:32.084 INFO:teuthology.orchestra.run.smithi152.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-04T07:26:32.193 INFO:teuthology.orchestra.run.smithi152.stdout:(43/176): ceph-mgr-diskprediction-local-19.0.0- 2.9 MB/s | 7.4 MB 00:02 2024-03-04T07:26:32.218 INFO:teuthology.orchestra.run.smithi152.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2024-03-04T07:26:32.310 INFO:teuthology.orchestra.run.smithi152.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 187 kB/s | 17 kB 00:00 2024-03-04T07:26:32.394 INFO:teuthology.orchestra.run.smithi152.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-03-04T07:26:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 94 kB/s | 10 kB 00:00 2024-03-04T07:26:32.444 INFO:teuthology.orchestra.run.smithi152.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-04T07:26:32.470 INFO:teuthology.orchestra.run.smithi152.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-04T07:26:32.512 INFO:teuthology.orchestra.run.smithi152.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-03-04T07:26:32.553 INFO:teuthology.orchestra.run.smithi152.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-04T07:26:32.579 INFO:teuthology.orchestra.run.smithi152.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 368 kB/s | 24 kB 00:00 2024-03-04T07:26:32.604 INFO:teuthology.orchestra.run.smithi152.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-03-04T07:26:32.629 INFO:teuthology.orchestra.run.smithi152.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-04T07:26:32.663 INFO:teuthology.orchestra.run.smithi152.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-04T07:26:32.688 INFO:teuthology.orchestra.run.smithi152.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 582 kB/s | 34 kB 00:00 2024-03-04T07:26:32.897 INFO:teuthology.orchestra.run.smithi152.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-04T07:26:32.956 INFO:teuthology.orchestra.run.smithi152.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-04T07:26:33.006 INFO:teuthology.orchestra.run.smithi152.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2024-03-04T07:26:33.048 INFO:teuthology.orchestra.run.smithi152.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 8.8 MB/s | 811 kB 00:00 2024-03-04T07:26:33.073 INFO:teuthology.orchestra.run.smithi152.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-04T07:26:33.307 INFO:teuthology.orchestra.run.smithi152.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-04T07:26:33.349 INFO:teuthology.orchestra.run.smithi152.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-03-04T07:26:33.374 INFO:teuthology.orchestra.run.smithi152.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-04T07:26:33.608 INFO:teuthology.orchestra.run.smithi152.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-04T07:26:33.633 INFO:teuthology.orchestra.run.smithi152.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 878 kB/s | 227 kB 00:00 2024-03-04T07:26:33.659 INFO:teuthology.orchestra.run.smithi152.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 491 kB/s | 24 kB 00:00 2024-03-04T07:26:33.684 INFO:teuthology.orchestra.run.smithi152.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-04T07:26:33.710 INFO:teuthology.orchestra.run.smithi152.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-04T07:26:33.735 INFO:teuthology.orchestra.run.smithi152.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 628 kB/s | 32 kB 00:00 2024-03-04T07:26:33.747 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:26:33.783 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:26:34.310 INFO:teuthology.orchestra.run.smithi152.stdout:(71/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 3.7 MB/s | 47 MB 00:12 2024-03-04T07:26:34.319 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:26:34.361 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T07:26:34.377 INFO:teuthology.orchestra.run.smithi152.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.2 MB/s | 4.8 MB 00:00 2024-03-04T07:26:34.402 INFO:teuthology.orchestra.run.smithi152.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 326 kB/s | 217 kB 00:00 2024-03-04T07:26:34.427 INFO:teuthology.orchestra.run.smithi152.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 174 kB 00:00 2024-03-04T07:26:34.433 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T07:26:34.433 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T07:26:34.433 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T07:26:34.433 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:34.461 INFO:teuthology.orchestra.run.smithi152.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-04T07:26:34.462 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T07:26:34.486 INFO:teuthology.orchestra.run.smithi152.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 495 kB/s | 41 kB 00:00 2024-03-04T07:26:34.512 INFO:teuthology.orchestra.run.smithi152.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-04T07:26:34.543 INFO:teuthology.orchestra.run.smithi152.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-04T07:26:34.568 INFO:teuthology.orchestra.run.smithi152.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-04T07:26:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:26:34.593 INFO:teuthology.orchestra.run.smithi152.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 583 kB/s | 47 kB 00:00 2024-03-04T07:26:34.618 INFO:teuthology.orchestra.run.smithi152.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-03-04T07:26:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:26:34.744 INFO:teuthology.orchestra.run.smithi152.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 258 kB/s | 39 kB 00:00 2024-03-04T07:26:34.994 INFO:teuthology.orchestra.run.smithi152.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 109 kB/s | 41 kB 00:00 2024-03-04T07:26:35.061 INFO:teuthology.orchestra.run.smithi152.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 685 kB/s | 215 kB 00:00 2024-03-04T07:26:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:26:35.278 INFO:teuthology.orchestra.run.smithi152.stdout:(85/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-03-04T07:26:35.303 INFO:teuthology.orchestra.run.smithi152.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 110 kB/s | 34 kB 00:00 2024-03-04T07:26:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T07:26:35.329 INFO:teuthology.orchestra.run.smithi152.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 164 kB/s | 43 kB 00:00 2024-03-04T07:26:35.354 INFO:teuthology.orchestra.run.smithi152.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-04T07:26:35.379 INFO:teuthology.orchestra.run.smithi152.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-04T07:26:35.413 INFO:teuthology.orchestra.run.smithi152.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 87 kB 00:00 2024-03-04T07:26:35.438 INFO:teuthology.orchestra.run.smithi152.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 749 kB/s | 63 kB 00:00 2024-03-04T07:26:35.464 INFO:teuthology.orchestra.run.smithi152.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 525 kB/s | 44 kB 00:00 2024-03-04T07:26:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T07:26:35.489 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T07:26:35.489 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T07:26:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:35.506 INFO:teuthology.orchestra.run.smithi152.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2024-03-04T07:26:35.531 INFO:teuthology.orchestra.run.smithi152.stdout:(94/176): fuse-2.9.9-15.el9.x86_64.rpm 855 kB/s | 79 kB 00:00 2024-03-04T07:26:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T07:26:35.631 INFO:teuthology.orchestra.run.smithi152.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 802 kB 00:00 2024-03-04T07:26:35.656 INFO:teuthology.orchestra.run.smithi152.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 194 kB 00:00 2024-03-04T07:26:35.682 INFO:teuthology.orchestra.run.smithi152.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-04T07:26:35.707 INFO:teuthology.orchestra.run.smithi152.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-04T07:26:35.799 INFO:teuthology.orchestra.run.smithi152.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.1 MB 00:00 2024-03-04T07:26:35.824 INFO:teuthology.orchestra.run.smithi152.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 727 kB/s | 103 kB 00:00 2024-03-04T07:26:35.849 INFO:teuthology.orchestra.run.smithi152.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 872 kB/s | 124 kB 00:00 2024-03-04T07:26:35.875 INFO:teuthology.orchestra.run.smithi152.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.5 MB/s | 113 kB 00:00 2024-03-04T07:26:35.925 INFO:teuthology.orchestra.run.smithi152.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.4 MB/s | 187 kB 00:00 2024-03-04T07:26:36.058 INFO:teuthology.orchestra.run.smithi152.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.1 MB/s | 245 kB 00:00 2024-03-04T07:26:36.117 INFO:teuthology.orchestra.run.smithi152.stdout:(105/176): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-03-04T07:26:36.409 INFO:teuthology.orchestra.run.smithi152.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 1.7 MB/s | 863 kB 00:00 2024-03-04T07:26:36.568 INFO:teuthology.orchestra.run.smithi152.stdout:(107/176): gperftools-libs-2.9.1-2.el9.x86_64.r 686 kB/s | 309 kB 00:00 2024-03-04T07:26:36.593 INFO:teuthology.orchestra.run.smithi152.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-04T07:26:36.668 INFO:teuthology.orchestra.run.smithi152.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-04T07:26:36.693 INFO:teuthology.orchestra.run.smithi152.stdout:(110/176): grpc-data-1.46.7-10.el9.noarch.rpm 68 kB/s | 19 kB 00:00 2024-03-04T07:26:36.744 INFO:teuthology.orchestra.run.smithi152.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-04T07:26:36.827 INFO:teuthology.orchestra.run.smithi152.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-04T07:26:36.852 INFO:teuthology.orchestra.run.smithi152.stdout:(113/176): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-03-04T07:26:36.945 INFO:teuthology.orchestra.run.smithi152.stdout:(114/176): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-04T07:26:37.000 INFO:teuthology.orchestra.run.smithi152.stdout:(115/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-04T07:26:37.028 INFO:teuthology.orchestra.run.smithi152.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 520 kB/s | 43 kB 00:00 2024-03-04T07:26:37.071 INFO:teuthology.orchestra.run.smithi152.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 460 kB/s | 32 kB 00:00 2024-03-04T07:26:37.104 INFO:teuthology.orchestra.run.smithi152.stdout:(118/176): python3-asyncssh-2.13.2-4.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-03-04T07:26:37.130 INFO:teuthology.orchestra.run.smithi152.stdout:(119/176): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-04T07:26:37.155 INFO:teuthology.orchestra.run.smithi152.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-04T07:26:37.180 INFO:teuthology.orchestra.run.smithi152.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-04T07:26:37.231 INFO:teuthology.orchestra.run.smithi152.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-03-04T07:26:37.256 INFO:teuthology.orchestra.run.smithi152.stdout:(123/176): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-03-04T07:26:37.281 INFO:teuthology.orchestra.run.smithi152.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-04T07:26:37.331 INFO:teuthology.orchestra.run.smithi152.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-04T07:26:37.357 INFO:teuthology.orchestra.run.smithi152.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-04T07:26:37.407 INFO:teuthology.orchestra.run.smithi152.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-04T07:26:37.432 INFO:teuthology.orchestra.run.smithi152.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-03-04T07:26:37.483 INFO:teuthology.orchestra.run.smithi152.stdout:(129/176): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-03-04T07:26:37.508 INFO:teuthology.orchestra.run.smithi152.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-03-04T07:26:37.533 INFO:teuthology.orchestra.run.smithi152.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-04T07:26:37.583 INFO:teuthology.orchestra.run.smithi152.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-04T07:26:37.609 INFO:teuthology.orchestra.run.smithi152.stdout:(133/176): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-03-04T07:26:37.634 INFO:teuthology.orchestra.run.smithi152.stdout:(134/176): python3-more-itertools-8.12.0-2.el9. 785 kB/s | 79 kB 00:00 2024-03-04T07:26:37.654 INFO:teuthology.orchestra.run.smithi152.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 821 kB/s | 58 kB 00:00 2024-03-04T07:26:37.696 INFO:teuthology.orchestra.run.smithi152.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.1 MB/s | 272 kB 00:00 2024-03-04T07:26:37.713 INFO:teuthology.orchestra.run.smithi152.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 208 kB/s | 16 kB 00:00 2024-03-04T07:26:37.738 INFO:teuthology.orchestra.run.smithi152.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-04T07:26:37.763 INFO:teuthology.orchestra.run.smithi152.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-03-04T07:26:37.780 INFO:teuthology.orchestra.run.smithi152.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-03-04T07:26:37.814 INFO:teuthology.orchestra.run.smithi152.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-04T07:26:37.848 INFO:teuthology.orchestra.run.smithi152.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-03-04T07:26:37.864 INFO:teuthology.orchestra.run.smithi152.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 426 kB/s | 36 kB 00:00 2024-03-04T07:26:37.881 INFO:teuthology.orchestra.run.smithi152.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 761 kB/s | 51 kB 00:00 2024-03-04T07:26:37.923 INFO:teuthology.orchestra.run.smithi152.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-04T07:26:37.949 INFO:teuthology.orchestra.run.smithi152.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-04T07:26:37.974 INFO:teuthology.orchestra.run.smithi152.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-04T07:26:37.999 INFO:teuthology.orchestra.run.smithi152.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-04T07:26:38.024 INFO:teuthology.orchestra.run.smithi152.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-04T07:26:38.116 INFO:teuthology.orchestra.run.smithi152.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-04T07:26:38.158 INFO:teuthology.orchestra.run.smithi152.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-04T07:26:38.300 INFO:teuthology.orchestra.run.smithi152.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-03-04T07:26:38.417 INFO:teuthology.orchestra.run.smithi152.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 16 MB/s | 1.8 MB 00:00 2024-03-04T07:26:38.476 INFO:teuthology.orchestra.run.smithi152.stdout:(154/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 6.9 MB/s | 3.1 MB 00:00 2024-03-04T07:26:38.501 INFO:teuthology.orchestra.run.smithi152.stdout:(155/176): NetworkManager-team-1.46.0-1.el9.x86 495 kB/s | 41 kB 00:00 2024-03-04T07:26:38.535 INFO:teuthology.orchestra.run.smithi152.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.1 MB/s | 247 kB 00:00 2024-03-04T07:26:38.559 INFO:teuthology.orchestra.run.smithi152.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2024-03-04T07:26:38.593 INFO:teuthology.orchestra.run.smithi152.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 607 kB/s | 35 kB 00:00 2024-03-04T07:26:38.627 INFO:teuthology.orchestra.run.smithi152.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 191 kB 00:00 2024-03-04T07:26:38.660 INFO:teuthology.orchestra.run.smithi152.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 254 kB 00:00 2024-03-04T07:26:38.711 INFO:teuthology.orchestra.run.smithi152.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-04T07:26:38.744 INFO:teuthology.orchestra.run.smithi152.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 5.1 MB/s | 433 kB 00:00 2024-03-04T07:26:38.761 INFO:teuthology.orchestra.run.smithi152.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 442 kB/s | 22 kB 00:00 2024-03-04T07:26:38.795 INFO:teuthology.orchestra.run.smithi152.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-04T07:26:38.820 INFO:teuthology.orchestra.run.smithi152.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 154 kB 00:00 2024-03-04T07:26:38.845 INFO:teuthology.orchestra.run.smithi152.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-04T07:26:38.863 INFO:teuthology.orchestra.run.smithi152.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-04T07:26:38.888 INFO:teuthology.orchestra.run.smithi152.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 659 kB/s | 28 kB 00:00 2024-03-04T07:26:38.938 INFO:teuthology.orchestra.run.smithi152.stdout:(169/176): librados2-19.0.0-1739.g1bc250c7.el9. 3.6 MB/s | 3.4 MB 00:00 2024-03-04T07:26:38.971 INFO:teuthology.orchestra.run.smithi152.stdout:(170/176): openssl-3.0.7-27.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-03-04T07:26:39.088 INFO:teuthology.orchestra.run.smithi152.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-04T07:26:39.130 INFO:teuthology.orchestra.run.smithi152.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-04T07:26:39.156 INFO:teuthology.orchestra.run.smithi152.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 2.4 MB/s | 438 kB 00:00 2024-03-04T07:26:39.172 INFO:teuthology.orchestra.run.smithi152.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 371 kB/s | 30 kB 00:00 2024-03-04T07:26:39.206 INFO:teuthology.orchestra.run.smithi152.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-04T07:26:39.256 INFO:teuthology.orchestra.run.smithi152.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-04T07:26:39.263 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-04T07:26:39.264 INFO:teuthology.orchestra.run.smithi152.stdout:Total 9.6 MB/s | 224 MB 00:23 2024-03-04T07:26:39.313 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-04T07:26:39.592 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-04T07:26:39.592 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-04T07:26:41.276 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-04T07:26:41.277 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-04T07:26:46.197 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T07:26:46.197 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2024-03-04T07:26:46.198 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2024-03-04T07:26:46.198 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2024-03-04T07:26:46.198 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2024-03-04T07:26:46.198 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2024-03-04T07:26:46.198 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /root 2024-03-04T07:26:46.198 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2024-03-04T07:26:46.198 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:46.366 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T07:26:46.381 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T07:26:46.381 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:46.381 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T07:26:46.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:47.868 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T07:26:47.894 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T07:26:47.894 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:47.894 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T07:26:47.894 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T07:26:47.894 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T07:26:47.894 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:48.111 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T07:26:48.116 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T07:26:48.116 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-04T07:26:48.116 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-04T07:26:48.116 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-04T07:26:48.116 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-04T07:26:48.116 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-04T07:26:48.117 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-04T07:26:48.117 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-04T07:26:48.117 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:48.133 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T07:26:48.133 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:48.134 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T07:26:48.134 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T07:26:48.134 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T07:26:48.134 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T07:26:48.390 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T07:26:48.390 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:48.391 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T07:26:48.391 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:48.415 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-04T07:26:48.551 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T07:26:48.578 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T07:26:48.594 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T07:26:48.601 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T07:26:48.601 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:48.601 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T07:26:48.601 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T07:26:48.601 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T07:26:48.601 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:48.633 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T07:26:48.654 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T07:26:48.688 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T07:26:48.703 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T07:26:48.742 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T07:26:48.945 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T07:26:48.979 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T07:26:49.005 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T07:26:49.039 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T07:26:49.105 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T07:26:49.128 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T07:26:49.159 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-04T07:26:49.219 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-04T07:26:49.243 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T07:26:49.360 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T07:26:49.413 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:26:49.470 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T07:26:49.502 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T07:26:49.535 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:26:49.564 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:26:49.655 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:26:49.685 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T07:26:49.834 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T07:26:49.890 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T07:26:49.918 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:26:49.986 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T07:26:50.023 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:26:50.052 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T07:26:50.068 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T07:26:50.068 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:50.068 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T07:26:50.068 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T07:26:50.069 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T07:26:50.069 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:50.116 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-04T07:26:50.156 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T07:26:50.179 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T07:26:50.194 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T07:26:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T07:26:50.379 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T07:26:50.379 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:50.379 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T07:26:50.379 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T07:26:50.380 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T07:26:50.380 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:50.386 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T07:26:50.418 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T07:26:50.450 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T07:26:50.460 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T07:26:50.485 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T07:26:50.556 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T07:26:50.584 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T07:26:50.660 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T07:26:50.704 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T07:26:50.739 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-04T07:26:50.762 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T07:26:50.773 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:50.785 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T07:26:50.785 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:50.785 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T07:26:50.785 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T07:26:50.785 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T07:26:50.785 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:50.821 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:50.829 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T07:26:50.840 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T07:26:50.854 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T07:26:50.877 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-04T07:26:50.918 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T07:26:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T07:26:50.919 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T07:26:50.919 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T07:26:50.919 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:51.032 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T07:26:51.039 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T07:26:51.142 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T07:26:51.230 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T07:26:51.260 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T07:26:51.328 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T07:26:51.382 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T07:26:51.413 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T07:26:51.448 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T07:26:51.488 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T07:26:51.552 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T07:26:51.595 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T07:26:51.639 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T07:26:51.671 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T07:26:51.687 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T07:26:51.699 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T07:26:51.723 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T07:26:51.730 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T07:26:51.757 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T07:26:51.768 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T07:26:51.774 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-04T07:26:51.778 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T07:26:51.778 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:51.778 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T07:26:51.779 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T07:26:51.779 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T07:26:51.779 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:51.804 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-04T07:26:51.824 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T07:26:51.846 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T07:26:51.846 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:51.846 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T07:26:51.847 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:51.847 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-04T07:26:51.874 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-04T07:26:51.876 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-04T07:26:51.891 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T07:26:51.902 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T07:26:51.913 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T07:26:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T07:26:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T07:26:51.914 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T07:26:51.914 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:51.945 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T07:26:51.991 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T07:26:52.064 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T07:26:52.206 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T07:26:52.217 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:26:52.225 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T07:26:52.239 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:26:52.239 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:52.239 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T07:26:52.239 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T07:26:52.239 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T07:26:52.239 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:26:52.259 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T07:26:52.261 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T07:26:52.782 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T07:26:52.854 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T07:26:52.923 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T07:26:52.966 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T07:26:52.974 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T07:26:53.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T07:26:53.051 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-04T07:26:53.082 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T07:26:53.188 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T07:26:53.241 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T07:26:53.381 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T07:26:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T07:26:53.488 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T07:26:53.546 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T07:26:53.628 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T07:26:53.701 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T07:26:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-04T07:26:53.745 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T07:26:53.755 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T07:26:53.820 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T07:26:53.820 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:53.820 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T07:26:53.820 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T07:26:53.820 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T07:26:53.820 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T07:26:53.891 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T07:26:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T07:26:53.965 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T07:26:54.009 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-04T07:26:54.020 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T07:26:54.092 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T07:26:54.123 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T07:26:54.160 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T07:26:54.206 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T07:26:54.243 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T07:26:54.286 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T07:26:54.327 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T07:26:54.469 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T07:26:54.516 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T07:26:54.547 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T07:26:54.585 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T07:26:54.679 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T07:26:54.710 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T07:26:54.750 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T07:26:54.776 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T07:26:54.801 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T07:26:54.874 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T07:26:54.967 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T07:26:55.000 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T07:26:55.001 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T07:26:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T07:26:55.074 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:55.074 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T07:26:55.074 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T07:26:55.074 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T07:26:55.074 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:55.107 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T07:26:55.150 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T07:26:55.159 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T07:26:55.198 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T07:26:55.231 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T07:26:55.231 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:55.232 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T07:26:55.232 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:55.262 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T07:26:55.310 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T07:26:55.387 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T07:26:55.387 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:55.387 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T07:26:55.387 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T07:26:55.387 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T07:26:55.387 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:55.445 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T07:26:55.487 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T07:26:55.615 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T07:26:55.652 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T07:26:55.693 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T07:26:55.724 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T07:26:55.760 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T07:26:55.768 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:26:55.791 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T07:26:55.818 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T07:26:55.853 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T07:26:55.861 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:26:55.861 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:26:55.862 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T07:26:55.862 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T07:26:55.862 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T07:26:55.862 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:26:55.892 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T07:26:55.928 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T07:26:55.962 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T07:26:56.037 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T07:26:56.072 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T07:26:56.105 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T07:26:56.138 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T07:26:56.173 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T07:26:56.261 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T07:26:56.344 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T07:26:56.387 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T07:26:56.418 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T07:26:56.465 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T07:26:56.562 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T07:26:56.620 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T07:26:56.637 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-04T07:26:56.679 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-04T07:26:56.709 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-04T07:26:56.736 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T07:26:56.935 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-04T07:26:57.052 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T07:26:57.092 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-04T07:26:57.132 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T07:26:57.176 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T07:26:57.242 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T07:26:57.331 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T07:26:57.346 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T07:26:57.380 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T07:26:57.391 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T07:26:57.410 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-04T07:26:57.425 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T07:26:57.439 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T07:26:57.473 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T07:26:57.507 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T07:26:57.553 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T07:26:57.560 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T07:26:57.597 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T07:26:57.601 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T07:26:57.778 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T07:26:57.807 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T07:26:57.884 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T07:26:58.143 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:26:58.178 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:26:58.197 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:26:58.521 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T07:26:58.636 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T07:26:59.001 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-04T07:26:59.073 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T07:26:59.112 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T07:26:59.193 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T07:26:59.234 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T07:26:59.259 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T07:26:59.288 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T07:26:59.311 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T07:26:59.311 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T07:26:59.396 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T07:26:59.425 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T07:26:59.456 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T07:26:59.456 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T07:26:59.549 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T07:26:59.575 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T07:26:59.609 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T07:26:59.634 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T07:26:59.661 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T07:26:59.718 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T07:26:59.751 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T07:26:59.791 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T07:26:59.823 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T07:26:59.848 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T07:26:59.848 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T07:26:59.874 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T07:26:59.900 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T07:26:59.924 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T07:26:59.930 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:26:59.930 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:26:59.934 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T07:26:59.961 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T07:27:00.110 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:27:00.110 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T07:27:00.201 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T07:27:00.234 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T07:27:00.260 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T07:27:00.272 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T07:27:00.285 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T07:27:00.299 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T07:27:00.311 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T07:27:00.373 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-04T07:27:00.375 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T07:27:00.404 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-04T07:27:00.440 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T07:27:00.674 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-04T07:27:00.727 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-04T07:27:00.764 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-04T07:27:00.801 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T07:27:00.904 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T07:27:00.936 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T07:27:01.006 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-04T07:27:01.030 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-04T07:27:01.071 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-04T07:27:01.192 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T07:27:01.248 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-04T07:27:01.302 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T07:27:01.366 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T07:27:01.475 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T07:27:01.518 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T07:27:01.563 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T07:27:01.593 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-04T07:27:01.644 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T07:27:01.790 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T07:27:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T07:27:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T07:27:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:27:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:27:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:27:03.183 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:27:03.225 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:27:03.764 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-04T07:27:03.804 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T07:27:03.884 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-04T07:27:03.885 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T07:27:03.885 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T07:27:03.885 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:03.912 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T07:27:04.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-04T07:27:04.245 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T07:27:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T07:27:04.463 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T07:27:04.506 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T07:27:04.539 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T07:27:04.574 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T07:27:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T07:27:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T07:27:04.773 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T07:27:04.823 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T07:27:04.869 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T07:27:04.869 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T07:27:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T07:27:05.085 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T07:27:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T07:27:05.173 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T07:27:05.205 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:27:05.206 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-04T07:27:05.207 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-04T07:27:05.208 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T07:27:05.209 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T07:27:05.210 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T07:27:05.211 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T07:27:05.212 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T07:27:05.213 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T07:27:05.214 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T07:27:05.215 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T07:27:05.216 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T07:27:05.217 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T07:27:05.218 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T07:27:05.219 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T07:27:05.220 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T07:27:05.221 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T07:27:05.221 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T07:27:05.221 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T07:27:05.254 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T07:27:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T07:27:05.368 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T07:27:05.437 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T07:27:05.478 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T07:27:05.512 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T07:27:05.545 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T07:27:05.579 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T07:27:05.612 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T07:27:05.648 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:27:05.648 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:27:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:27:05.987 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T07:27:06.126 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T07:27:06.178 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T07:27:06.226 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T07:27:06.265 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T07:27:06.301 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T07:27:06.428 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T07:27:06.683 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T07:27:06.684 INFO:teuthology.orchestra.run.smithi116.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-04T07:27:06.684 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:27:07.987 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T07:27:07.987 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:27:07.987 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2024-03-04T07:27:07.987 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T07:27:07.988 INFO:teuthology.orchestra.run.smithi133.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.989 INFO:teuthology.orchestra.run.smithi133.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.990 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.991 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T07:27:07.992 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: jq-1.6-16.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.993 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T07:27:07.994 INFO:teuthology.orchestra.run.smithi133.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T07:27:07.995 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T07:27:07.996 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T07:27:07.997 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T07:27:07.998 INFO:teuthology.orchestra.run.smithi133.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:07.999 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T07:27:08.000 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T07:27:08.001 INFO:teuthology.orchestra.run.smithi133.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T07:27:08.002 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T07:27:08.002 INFO:teuthology.orchestra.run.smithi133.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T07:27:08.002 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-04T07:27:08.002 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-03-04T07:27:08.566 DEBUG:teuthology.parallel:result is None 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-04T07:27:11.349 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-04T07:27:11.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-04T07:27:11.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:27:11.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:27:11.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T07:27:11.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-04T07:27:11.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-04T07:27:11.350 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-04T07:27:11.351 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-04T07:27:11.352 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-04T07:27:11.352 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-04T07:27:11.352 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-04T07:27:11.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T07:27:11.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T07:27:11.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T07:27:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T07:27:11.375 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T07:27:11.376 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T07:27:11.377 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T07:27:11.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T07:27:11.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T07:27:11.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T07:27:11.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T07:27:11.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T07:27:11.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T07:27:11.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T07:27:11.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T07:27:11.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T07:27:11.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T07:27:11.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T07:27:11.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T07:27:11.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T07:27:13.843 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T07:27:13.843 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:27:13.843 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T07:27:13.844 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T07:27:13.845 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.846 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.847 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T07:27:13.848 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.849 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T07:27:13.850 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T07:27:13.851 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T07:27:13.852 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T07:27:13.853 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T07:27:13.854 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T07:27:13.854 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T07:27:13.854 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T07:27:13.854 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T07:27:13.854 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T07:27:13.855 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:13.856 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:27:13.857 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-04T07:27:14.452 DEBUG:teuthology.parallel:result is None 2024-03-04T07:27:16.142 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-04T07:27:16.142 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-03-04T07:27:16.143 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-03-04T07:27:16.143 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-03-04T07:27:16.143 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-03-04T07:27:16.143 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-03-04T07:27:16.143 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-03-04T07:27:16.143 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-03-04T07:27:16.143 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:16.308 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T07:27:16.323 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-04T07:27:16.324 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:16.324 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T07:27:16.324 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:17.811 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T07:27:17.836 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-04T07:27:17.836 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:17.836 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T07:27:17.837 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T07:27:17.837 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T07:27:17.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:18.062 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T07:27:18.084 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-04T07:27:18.084 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:18.084 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T07:27:18.084 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T07:27:18.084 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T07:27:18.084 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:18.529 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T07:27:18.552 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-04T07:27:18.552 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:18.552 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T07:27:18.552 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T07:27:18.552 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T07:27:18.552 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:18.639 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T07:27:19.115 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T07:27:19.145 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-04T07:27:19.177 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-04T07:27:19.237 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-04T07:27:19.260 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T07:27:19.489 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-04T07:27:19.519 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T07:27:20.485 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-04T07:27:20.511 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T07:27:20.612 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-04T07:27:20.775 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-04T07:27:20.805 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T07:27:20.828 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-04T07:27:20.828 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:20.828 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T07:27:20.829 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T07:27:20.829 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T07:27:20.829 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:20.872 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T07:27:20.883 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-04T07:27:20.928 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-04T07:27:21.787 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T07:27:21.824 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T07:27:21.847 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-04T07:27:21.847 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:21.847 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T07:27:21.847 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T07:27:21.847 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T07:27:21.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:21.891 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T07:27:21.913 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-04T07:27:21.913 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:21.913 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T07:27:21.913 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:21.967 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T07:27:21.989 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-04T07:27:21.989 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:21.989 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T07:27:21.989 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T07:27:21.990 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T07:27:21.990 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:22.292 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:27:22.315 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:27:22.315 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T07:27:22.315 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T07:27:22.315 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T07:27:22.315 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T07:27:22.315 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:26.730 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-04T07:27:26.771 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-04T07:27:26.809 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-04T07:27:26.837 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T07:27:27.016 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-04T07:27:27.136 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T07:27:27.201 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-04T07:27:27.240 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T07:27:27.284 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T07:27:27.353 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T07:27:27.394 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T07:27:27.438 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T07:27:27.460 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-04T07:27:27.490 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T07:27:27.558 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T07:27:27.612 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T07:27:27.655 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T07:27:28.233 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:27:28.269 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:27:28.289 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T07:27:29.113 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-04T07:27:29.188 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T07:27:29.229 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T07:27:29.309 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T07:27:29.384 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T07:27:29.408 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T07:27:29.442 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T07:27:29.468 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T07:27:29.469 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T07:27:29.553 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T07:27:29.581 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T07:27:29.611 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T07:27:29.611 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T07:27:29.704 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T07:27:29.730 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T07:27:29.773 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T07:27:29.797 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T07:27:29.825 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T07:27:29.857 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T07:27:29.890 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T07:27:29.929 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T07:27:29.961 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T07:27:29.986 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T07:27:30.020 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T07:27:30.054 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T07:27:30.079 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T07:27:30.085 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:27:30.085 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:27:30.265 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T07:27:30.266 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T07:27:30.348 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T07:27:30.380 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T07:27:30.407 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T07:27:30.432 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T07:27:30.458 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T07:27:30.521 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-04T07:27:35.294 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-04T07:27:35.295 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-04T07:27:35.296 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-04T07:27:35.311 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T07:27:35.312 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T07:27:35.313 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T07:27:35.314 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T07:27:35.315 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T07:27:35.316 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T07:27:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T07:27:35.318 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T07:27:35.319 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T07:27:35.320 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T07:27:35.321 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T07:27:35.322 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T07:27:35.323 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T07:27:35.324 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T07:27:38.118 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:38.119 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.120 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.121 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T07:27:38.122 INFO:teuthology.orchestra.run.smithi152.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: jq-1.6-16.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T07:27:38.123 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T07:27:38.124 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T07:27:38.125 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T07:27:38.126 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T07:27:38.127 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T07:27:38.128 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T07:27:38.129 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-04T07:27:38.130 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:27:38.131 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-04T07:27:38.733 DEBUG:teuthology.parallel:result is None 2024-03-04T07:27:38.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:27:38.950 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T07:27:39.017 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-04T07:27:39.018 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-04T07:27:39.018 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-04T07:27:39.020 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:27:39.234 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T07:27:39.259 INFO:teuthology.orchestra.run.smithi133.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-04T07:27:39.259 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-04T07:27:39.259 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-04T07:27:39.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:27:39.396 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T07:27:39.422 INFO:teuthology.orchestra.run.smithi152.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-04T07:27:39.422 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-04T07:27:39.422 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-04T07:27:39.424 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T07:27:39.424 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:27:39.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T07:27:39.529 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:27:39.529 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T07:27:39.564 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:27:39.564 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T07:27:39.599 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T07:27:39.599 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:27:39.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T07:27:39.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T07:27:39.794 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:27:39.794 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T07:27:39.826 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T07:27:39.896 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:27:39.897 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T07:27:39.929 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T07:27:39.999 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T07:27:39.999 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:27:39.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T07:27:40.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T07:27:40.234 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:27:40.234 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T07:27:40.260 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T07:27:40.327 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:27:40.328 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T07:27:40.359 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T07:27:40.431 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T07:27:40.431 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:27:40.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T07:27:40.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T07:27:40.624 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:27:40.624 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T07:27:40.651 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T07:27:40.717 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:27:40.717 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T07:27:40.748 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T07:27:40.819 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-04T07:27:40.912 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-04T07:27:40.912 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:27:40.913 INFO:tasks.cephadm:Cluster fsid is af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:27:40.913 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-04T07:27:40.913 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.116', 'mon.b': '172.21.15.133', 'mon.c': '172.21.15.152'} 2024-03-04T07:27:40.913 INFO:tasks.cephadm:First mon is mon.a on smithi116 2024-03-04T07:27:40.913 INFO:tasks.cephadm:First mgr is x 2024-03-04T07:27:40.913 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-04T07:27:40.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-04T07:27:41.014 DEBUG:teuthology.orchestra.run.smithi133:> sudo hostname $(hostname -s) 2024-03-04T07:27:41.042 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-03-04T07:27:41.074 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-04T07:27:41.074 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:27:41.211 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77279/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1739-g1bc250c7', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1739.g1bc250c7'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-04 00:06:26.635980', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-qdb', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-04T07:27:41.351 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-lusov-qdb, sha1 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:27:41.353 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/x86_64/flavors/default/cephadm 2024-03-04T07:27:41.353 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:42.595 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 766895 Mar 4 07:27 /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:42.596 DEBUG:teuthology.orchestra.run.smithi133:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:44.965 INFO:teuthology.orchestra.run.smithi133.stdout:-rw-r--r--. 1 ubuntu ubuntu 766895 Mar 4 07:27 /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:44.966 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:46.767 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-r--r--. 1 ubuntu ubuntu 766895 Mar 4 07:27 /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:46.768 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:46.830 DEBUG:teuthology.orchestra.run.smithi133:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:46.848 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T07:27:46.882 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd on all hosts... 2024-03-04T07:27:46.883 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd pull 2024-03-04T07:27:46.889 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd pull 2024-03-04T07:27:46.892 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd pull 2024-03-04T07:27:47.635 INFO:teuthology.orchestra.run.smithi133.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd... 2024-03-04T07:27:47.743 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd... 2024-03-04T07:27:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd... 2024-03-04T07:28:14.083 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-04T07:28:14.083 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-04T07:28:14.083 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba", 2024-03-04T07:28:14.083 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-03-04T07:28:14.083 INFO:teuthology.orchestra.run.smithi152.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5" 2024-03-04T07:28:14.084 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-03-04T07:28:14.084 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-04T07:28:19.211 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-03-04T07:28:19.211 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-04T07:28:19.211 INFO:teuthology.orchestra.run.smithi133.stdout: "image_id": "f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba", 2024-03-04T07:28:19.211 INFO:teuthology.orchestra.run.smithi133.stdout: "repo_digests": [ 2024-03-04T07:28:19.211 INFO:teuthology.orchestra.run.smithi133.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5" 2024-03-04T07:28:19.211 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2024-03-04T07:28:19.211 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-03-04T07:28:27.196 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:28:27.196 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-04T07:28:27.196 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba", 2024-03-04T07:28:27.196 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-04T07:28:27.196 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5" 2024-03-04T07:28:27.196 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:28:27.196 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:28:27.262 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-04T07:28:27.361 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph 2024-03-04T07:28:27.394 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-03-04T07:28:27.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-04T07:28:27.527 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /etc/ceph 2024-03-04T07:28:27.556 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-03-04T07:28:27.588 INFO:tasks.cephadm:Writing seed config... 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-04T07:28:27.589 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-04T07:28:27.590 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-04T07:28:27.591 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-04T07:28:27.592 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:28:27.592 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-04T07:28:27.645 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = af2ccd48-d9f8-11ee-95c6-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-04T07:28:27.646 DEBUG:teuthology.orchestra.run.smithi116:mon.a> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a.service 2024-03-04T07:28:27.693 DEBUG:teuthology.orchestra.run.smithi116:mgr.x> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.x.service 2024-03-04T07:28:27.738 INFO:tasks.cephadm:Bootstrapping... 2024-03-04T07:28:27.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd -v bootstrap --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.116 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:28:28.010 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-04T07:28:28.010 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', '-v', 'bootstrap', '--fsid', 'af2ccd48-d9f8-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.116', '--skip-admin-label'] 2024-03-04T07:28:28.069 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T07:28:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-04T07:28:28.070 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying podman|docker is present... 2024-03-04T07:28:28.120 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T07:28:28.121 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying lvm2 is present... 2024-03-04T07:28:28.121 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying time synchronization is in place... 2024-03-04T07:28:28.135 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T07:28:28.135 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T07:28:28.148 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T07:28:28.148 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T07:28:28.164 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-04T07:28:28.179 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-04T07:28:28.179 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-04T07:28:28.179 INFO:teuthology.orchestra.run.smithi116.stdout:Repeating the final host check... 2024-03-04T07:28:28.231 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T07:28:28.231 INFO:teuthology.orchestra.run.smithi116.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-04T07:28:28.231 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl is present 2024-03-04T07:28:28.231 INFO:teuthology.orchestra.run.smithi116.stdout:lvcreate is present 2024-03-04T07:28:28.245 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T07:28:28.245 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T07:28:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T07:28:28.258 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T07:28:28.272 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-04T07:28:28.288 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-04T07:28:28.289 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-04T07:28:28.289 INFO:teuthology.orchestra.run.smithi116.stdout:Host looks OK 2024-03-04T07:28:28.289 INFO:teuthology.orchestra.run.smithi116.stdout:Cluster fsid: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:28.290 INFO:teuthology.orchestra.run.smithi116.stdout:Acquiring lock 139623262330832 on /run/cephadm/af2ccd48-d9f8-11ee-95c6-87774f69a715.lock 2024-03-04T07:28:28.290 INFO:teuthology.orchestra.run.smithi116.stdout:Lock 139623262330832 acquired on /run/cephadm/af2ccd48-d9f8-11ee-95c6-87774f69a715.lock 2024-03-04T07:28:28.291 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 3300 ... 2024-03-04T07:28:28.293 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 6789 ... 2024-03-04T07:28:28.293 INFO:teuthology.orchestra.run.smithi116.stdout:Base mon IP(s) is [172.21.15.116:3300, 172.21.15.116:6789], mon addrv is [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-04T07:28:28.307 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.116 metric 100 2024-03-04T07:28:28.307 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.116 metric 100 2024-03-04T07:28:28.314 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-04T07:28:28.314 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-04T07:28:28.314 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-03-04T07:28:28.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-04T07:28:28.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-04T07:28:28.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T07:28:28.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-04T07:28:28.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link proto kernel_ll 2024-03-04T07:28:28.321 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T07:28:28.322 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-04T07:28:28.322 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-04T07:28:28.322 INFO:teuthology.orchestra.run.smithi116.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-04T07:28:28.323 INFO:teuthology.orchestra.run.smithi116.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-04T07:28:28.325 INFO:teuthology.orchestra.run.smithi116.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd... 2024-03-04T07:28:38.016 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba 2024-03-04T07:28:38.017 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd... 2024-03-04T07:28:38.017 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Getting image source signatures 2024-03-04T07:28:38.017 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-04T07:28:38.017 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:5d74734bfa4d1f8e0a72f8c221d93ca0d3a68fb40764a44763479cacf5cffcb4 2024-03-04T07:28:38.017 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying config sha256:f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba 2024-03-04T07:28:38.017 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-04T07:28:39.895 INFO:teuthology.orchestra.run.smithi116.stdout:ceph: stdout ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev) 2024-03-04T07:28:39.896 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph version: ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev) 2024-03-04T07:28:39.896 INFO:teuthology.orchestra.run.smithi116.stdout:Extracting ceph user uid/gid from container image... 2024-03-04T07:28:41.123 INFO:teuthology.orchestra.run.smithi116.stdout:stat: stdout 167 167 2024-03-04T07:28:41.123 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial keys... 2024-03-04T07:28:42.205 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQApeOVlKwS8IBAA8z73I6ToOYqak/pCvfGhvA== 2024-03-04T07:28:43.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQAqeOVlTPJlKRAACgFfO6KuqDYBF2UK1o+exQ== 2024-03-04T07:28:44.470 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQAreOVlFai4KxAATFOlyX4bhgEFTQjDlts9jQ== 2024-03-04T07:28:44.471 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial monmap... 2024-03-04T07:28:45.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T07:28:45.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-04T07:28:45.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:45.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T07:28:45.615 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool for a [v2:172.21.15.116:3300,v1:172.21.15.116:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T07:28:45.616 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = quincy 2024-03-04T07:28:45.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: set fsid to af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:45.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T07:28:45.616 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:28:45.616 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mon... 2024-03-04T07:28:46.696 INFO:teuthology.orchestra.run.smithi116.stdout:create mon.a on 2024-03-04T07:28:47.060 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-04T07:28:47.290 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-04T07:28:47.567 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715.target. 2024-03-04T07:28:47.568 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715.target. 2024-03-04T07:28:47.914 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a 2024-03-04T07:28:47.914 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-04T07:28:48.206 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715.target.wants/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@.service. 2024-03-04T07:28:48.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:48 smithi116 podman[31760]: 2024-03-04 07:28:48.602225278 +0000 UTC m=+0.141091177 container create 1da75f11c245757b1a79c2da06789bc1eb68e2336b6f800d2a69e5a1d2ea57ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:28:48.873 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:48 smithi116 podman[31760]: 2024-03-04 07:28:48.872072737 +0000 UTC m=+0.410938679 container init 1da75f11c245757b1a79c2da06789bc1eb68e2336b6f800d2a69e5a1d2ea57ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222) 2024-03-04T07:28:49.020 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T07:28:49.020 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T07:28:49.039 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T07:28:49.039 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T07:28:49.039 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T07:28:49.039 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T07:28:49.041 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon to start... 2024-03-04T07:28:49.042 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon... 2024-03-04T07:28:49.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:48 smithi116 podman[31760]: 2024-03-04 07:28:48.882573811 +0000 UTC m=+0.421439717 container start 1da75f11c245757b1a79c2da06789bc1eb68e2336b6f800d2a69e5a1d2ea57ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, ceph=True) 2024-03-04T07:28:49.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:48 smithi116 ceph-mon[31779]: mkfs af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:49.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:48 smithi116 ceph-mon[31779]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T07:28:49.193 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:48 smithi116 bash[31760]: 1da75f11c245757b1a79c2da06789bc1eb68e2336b6f800d2a69e5a1d2ea57ca 2024-03-04T07:28:49.194 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:48 smithi116 systemd[1]: Started Ceph mon.a for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:28:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:49 smithi116 ceph-mon[31779]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T07:28:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:49 smithi116 ceph-mon[31779]: monmap e1: 1 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:28:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:49 smithi116 ceph-mon[31779]: fsmap 2024-03-04T07:28:50.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:49 smithi116 ceph-mon[31779]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T07:28:50.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:49 smithi116 ceph-mon[31779]: mgrmap e1: no daemons active 2024-03-04T07:28:50.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:49 smithi116 ceph-mon[31779]: from='client.? 172.21.15.116:0/4120999890' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-04T07:28:50.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout cluster: 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout id: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout services: 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.726808s) 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout data: 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-04T07:28:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pgs: 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.595+0000 7fa3b5200700 1 Processor -- start 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.595+0000 7fa3b5200700 1 -- start start 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.595+0000 7fa3b5200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b0107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.595+0000 7fa3b5200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b01082f0 con 0x7fa3b0107900 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.596+0000 7fa3ade00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b0107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.596+0000 7fa3ade00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b0107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33056/0 (socket says 172.21.15.116:33056) 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.596+0000 7fa3ade00700 1 -- 172.21.15.116:0/96427093 learned_addr learned my addr 172.21.15.116:0/96427093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.597+0000 7fa3ade00700 1 -- 172.21.15.116:0/96427093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0108ab0 con 0x7fa3b0107900 2024-03-04T07:28:50.388 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.597+0000 7fa3ade00700 1 --2- 172.21.15.116:0/96427093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b0107d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00098a0 tx=0x7fa3a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d916609333e6d44e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.597+0000 7fa3aca00700 1 -- 172.21.15.116:0/96427093 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3a0004160 con 0x7fa3b0107900 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.597+0000 7fa3aca00700 1 -- 172.21.15.116:0/96427093 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0004300 con 0x7fa3b0107900 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.597+0000 7fa3aca00700 1 -- 172.21.15.116:0/96427093 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3a00036a0 con 0x7fa3b0107900 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.598+0000 7fa3b5200700 1 -- 172.21.15.116:0/96427093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 msgr2=0x7fa3b0107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.598+0000 7fa3b5200700 1 --2- 172.21.15.116:0/96427093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b0107d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00098a0 tx=0x7fa3a0009bb0 comp rx=0 tx=0).stop 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.598+0000 7fa3b5200700 1 -- 172.21.15.116:0/96427093 shutdown_connections 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.598+0000 7fa3b5200700 1 --2- 172.21.15.116:0/96427093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b0107d20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.598+0000 7fa3b5200700 1 -- 172.21.15.116:0/96427093 >> 172.21.15.116:0/96427093 conn(0x7fa3b0102f60 msgr2=0x7fa3b0105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.598+0000 7fa3b5200700 1 -- 172.21.15.116:0/96427093 shutdown_connections 2024-03-04T07:28:50.389 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.598+0000 7fa3b5200700 1 -- 172.21.15.116:0/96427093 wait complete. 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.599+0000 7fa3b5200700 1 Processor -- start 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.599+0000 7fa3b5200700 1 -- start start 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.600+0000 7fa3b5200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.600+0000 7fa3b5200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b0108fe0 con 0x7fa3b0107900 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.600+0000 7fa3ade00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.600+0000 7fa3ade00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b019d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33058/0 (socket says 172.21.15.116:33058) 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.600+0000 7fa3ade00700 1 -- 172.21.15.116:0/4120999890 learned_addr learned my addr 172.21.15.116:0/4120999890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3ade00700 1 -- 172.21.15.116:0/4120999890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b019da30 con 0x7fa3b0107900 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3ade00700 1 --2- 172.21.15.116:0/4120999890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b019d4f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000c00 tx=0x7fa3a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3a6a00700 1 -- 172.21.15.116:0/4120999890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3a0003db0 con 0x7fa3b0107900 2024-03-04T07:28:50.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3a6a00700 1 -- 172.21.15.116:0/4120999890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0011630 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b019dcc0 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3a6a00700 1 -- 172.21.15.116:0/4120999890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3a0011950 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b01a09b0 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.601+0000 7fa3a6a00700 1 -- 172.21.15.116:0/4120999890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa3a0011af0 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.602+0000 7fa3a6a00700 1 -- 172.21.15.116:0/4120999890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa3a001c450 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.602+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b0069160 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.605+0000 7fa3a6a00700 1 -- 172.21.15.116:0/4120999890 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa3a0032020 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.684+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa3b002d9b0 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.685+0000 7fa3a6a00700 1 -- 172.21.15.116:0/4120999890 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fa3a0004260 con 0x7fa3b0107900 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.686+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 msgr2=0x7fa3b019d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.686+0000 7fa3b5200700 1 --2- 172.21.15.116:0/4120999890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b019d4f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000c00 tx=0x7fa3a0004750 comp rx=0 tx=0).stop 2024-03-04T07:28:50.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.686+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 shutdown_connections 2024-03-04T07:28:50.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.686+0000 7fa3b5200700 1 --2- 172.21.15.116:0/4120999890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b0107900 0x7fa3b019d4f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:50.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.686+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 >> 172.21.15.116:0/4120999890 conn(0x7fa3b0102f60 msgr2=0x7fa3b0193fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:50.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.686+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 shutdown_connections 2024-03-04T07:28:50.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:49.686+0000 7fa3b5200700 1 -- 172.21.15.116:0/4120999890 wait complete. 2024-03-04T07:28:50.392 INFO:teuthology.orchestra.run.smithi116.stdout:mon is available 2024-03-04T07:28:50.392 INFO:teuthology.orchestra.run.smithi116.stdout:Assimilating anything we can from ceph.conf... 2024-03-04T07:28:51.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:51 smithi116 ceph-mon[31779]: from='client.? 172.21.15.116:0/558377074' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T07:28:51.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:51 smithi116 ceph-mon[31779]: from='client.? 172.21.15.116:0/558377074' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T07:28:51.702 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:28:51.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T07:28:51.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:51.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-04T07:28:51.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T07:28:51.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T07:28:51.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T07:28:51.703 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T07:28:51.705 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.972+0000 7f3886600700 1 Processor -- start 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.972+0000 7f3886600700 1 -- start start 2024-03-04T07:28:51.706 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.973+0000 7f3886600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f3880107b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:51.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.973+0000 7f3886600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880108110 con 0x7f3880107720 2024-03-04T07:28:51.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.973+0000 7f3885200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f3880107b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:51.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.973+0000 7f3885200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f3880107b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33066/0 (socket says 172.21.15.116:33066) 2024-03-04T07:28:51.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.973+0000 7f3885200700 1 -- 172.21.15.116:0/3936845516 learned_addr learned my addr 172.21.15.116:0/3936845516 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:51.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.974+0000 7f3885200700 1 -- 172.21.15.116:0/3936845516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880108920 con 0x7f3880107720 2024-03-04T07:28:51.707 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.974+0000 7f3885200700 1 --2- 172.21.15.116:0/3936845516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f3880107b40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f38700098a0 tx=0x7f3870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909f46c1cf4219f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:51.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.974+0000 7f387fe00700 1 -- 172.21.15.116:0/3936845516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3870004160 con 0x7f3880107720 2024-03-04T07:28:51.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.974+0000 7f387fe00700 1 -- 172.21.15.116:0/3936845516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870004300 con 0x7f3880107720 2024-03-04T07:28:51.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.974+0000 7f387fe00700 1 -- 172.21.15.116:0/3936845516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38700036a0 con 0x7f3880107720 2024-03-04T07:28:51.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.975+0000 7f3886600700 1 -- 172.21.15.116:0/3936845516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 msgr2=0x7f3880107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:51.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.975+0000 7f3886600700 1 --2- 172.21.15.116:0/3936845516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f3880107b40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f38700098a0 tx=0x7f3870009bb0 comp rx=0 tx=0).stop 2024-03-04T07:28:51.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.975+0000 7f3886600700 1 -- 172.21.15.116:0/3936845516 shutdown_connections 2024-03-04T07:28:51.711 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.975+0000 7f3886600700 1 --2- 172.21.15.116:0/3936845516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f3880107b40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.975+0000 7f3886600700 1 -- 172.21.15.116:0/3936845516 >> 172.21.15.116:0/3936845516 conn(0x7f3880102da0 msgr2=0x7f38801051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.975+0000 7f3886600700 1 -- 172.21.15.116:0/3936845516 shutdown_connections 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.975+0000 7f3886600700 1 -- 172.21.15.116:0/3936845516 wait complete. 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.976+0000 7f3886600700 1 Processor -- start 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.976+0000 7f3886600700 1 -- start start 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.977+0000 7f3886600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f388019d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.977+0000 7f3886600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3880108e50 con 0x7f3880107720 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.977+0000 7f3885200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f388019d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.977+0000 7f3885200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f388019d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33072/0 (socket says 172.21.15.116:33072) 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.977+0000 7f3885200700 1 -- 172.21.15.116:0/558377074 learned_addr learned my addr 172.21.15.116:0/558377074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.978+0000 7f3885200700 1 -- 172.21.15.116:0/558377074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388019d8c0 con 0x7f3880107720 2024-03-04T07:28:51.712 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.978+0000 7f3885200700 1 --2- 172.21.15.116:0/558377074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f388019d380 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3870000c00 tx=0x7f3870004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.978+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3870003db0 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.978+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870011630 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.978+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3870011950 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.978+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388019db50 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.979+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38801a0840 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.979+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3870011af0 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.979+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f387001c450 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.980+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3880069160 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:50.983+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3870032020 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.058+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f38801a1b90 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.063+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f3870004260 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.063+0000 7f387e000700 1 -- 172.21.15.116:0/558377074 <== mon.0 v2:172.21.15.116:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f387001cb60 con 0x7f3880107720 2024-03-04T07:28:51.713 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.065+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 msgr2=0x7f388019d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:51.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.065+0000 7f3886600700 1 --2- 172.21.15.116:0/558377074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f388019d380 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3870000c00 tx=0x7f3870004750 comp rx=0 tx=0).stop 2024-03-04T07:28:51.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.065+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 shutdown_connections 2024-03-04T07:28:51.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.065+0000 7f3886600700 1 --2- 172.21.15.116:0/558377074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3880107720 0x7f388019d380 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:51.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.065+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 >> 172.21.15.116:0/558377074 conn(0x7f3880102da0 msgr2=0x7f3880193f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:51.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.065+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 shutdown_connections 2024-03-04T07:28:51.714 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:51.065+0000 7f3886600700 1 -- 172.21.15.116:0/558377074 wait complete. 2024-03-04T07:28:51.714 INFO:teuthology.orchestra.run.smithi116.stdout:Generating new minimal ceph.conf... 2024-03-04T07:28:52.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:52 smithi116 ceph-mon[31779]: from='client.? 172.21.15.116:0/838183479' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:28:53.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.312+0000 7fdb91e00700 1 Processor -- start 2024-03-04T07:28:53.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.312+0000 7fdb91e00700 1 -- start start 2024-03-04T07:28:53.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.313+0000 7fdb91e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:53.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.313+0000 7fdb91e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c1082f0 con 0x7fdb8c107900 2024-03-04T07:28:53.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.313+0000 7fdb8aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.313+0000 7fdb8aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33076/0 (socket says 172.21.15.116:33076) 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.313+0000 7fdb8aa00700 1 -- 172.21.15.116:0/1623328927 learned_addr learned my addr 172.21.15.116:0/1623328927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.314+0000 7fdb8aa00700 1 -- 172.21.15.116:0/1623328927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c108ab0 con 0x7fdb8c107900 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.314+0000 7fdb8aa00700 1 --2- 172.21.15.116:0/1623328927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c107d20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0098a0 tx=0x7fdb7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227dc516d43622ae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.314+0000 7fdb89600700 1 -- 172.21.15.116:0/1623328927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb7c004160 con 0x7fdb8c107900 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.314+0000 7fdb89600700 1 -- 172.21.15.116:0/1623328927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb7c004300 con 0x7fdb8c107900 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.314+0000 7fdb89600700 1 -- 172.21.15.116:0/1623328927 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb7c011b50 con 0x7fdb8c107900 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.315+0000 7fdb91e00700 1 -- 172.21.15.116:0/1623328927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 msgr2=0x7fdb8c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.315+0000 7fdb91e00700 1 --2- 172.21.15.116:0/1623328927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c107d20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0098a0 tx=0x7fdb7c009bb0 comp rx=0 tx=0).stop 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.315+0000 7fdb91e00700 1 -- 172.21.15.116:0/1623328927 shutdown_connections 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.315+0000 7fdb91e00700 1 --2- 172.21.15.116:0/1623328927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c107d20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:53.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.315+0000 7fdb91e00700 1 -- 172.21.15.116:0/1623328927 >> 172.21.15.116:0/1623328927 conn(0x7fdb8c102f60 msgr2=0x7fdb8c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.315+0000 7fdb91e00700 1 -- 172.21.15.116:0/1623328927 shutdown_connections 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.315+0000 7fdb91e00700 1 -- 172.21.15.116:0/1623328927 wait complete. 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.316+0000 7fdb91e00700 1 Processor -- start 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.316+0000 7fdb91e00700 1 -- start start 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.317+0000 7fdb91e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.317+0000 7fdb91e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c10be60 con 0x7fdb8c107900 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.317+0000 7fdb8aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.317+0000 7fdb8aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c19ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33080/0 (socket says 172.21.15.116:33080) 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.317+0000 7fdb8aa00700 1 -- 172.21.15.116:0/838183479 learned_addr learned my addr 172.21.15.116:0/838183479 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.317+0000 7fdb8aa00700 1 -- 172.21.15.116:0/838183479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c19e310 con 0x7fdb8c107900 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.318+0000 7fdb8aa00700 1 --2- 172.21.15.116:0/838183479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c19ddd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c000c00 tx=0x7fdb7c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.318+0000 7fdb90a00700 1 -- 172.21.15.116:0/838183479 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb7c003b20 con 0x7fdb8c107900 2024-03-04T07:28:53.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.318+0000 7fdb90a00700 1 -- 172.21.15.116:0/838183479 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb7c003cc0 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.318+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c19e5a0 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.318+0000 7fdb90a00700 1 -- 172.21.15.116:0/838183479 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb7c024910 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.318+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c1a1290 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.318+0000 7fdb90a00700 1 -- 172.21.15.116:0/838183479 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fdb7c022020 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.319+0000 7fdb90a00700 1 -- 172.21.15.116:0/838183479 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdb7c011e00 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.319+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8c069160 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.322+0000 7fdb90a00700 1 -- 172.21.15.116:0/838183479 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fdb7c02ddf0 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.398+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fdb8c02d9b0 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.399+0000 7fdb90a00700 1 -- 172.21.15.116:0/838183479 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fdb7c032030 con 0x7fdb8c107900 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.400+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 msgr2=0x7fdb8c19ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.400+0000 7fdb91e00700 1 --2- 172.21.15.116:0/838183479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c19ddd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c000c00 tx=0x7fdb7c011c10 comp rx=0 tx=0).stop 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.400+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 shutdown_connections 2024-03-04T07:28:53.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.400+0000 7fdb91e00700 1 --2- 172.21.15.116:0/838183479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8c107900 0x7fdb8c19ddd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:53.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.400+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 >> 172.21.15.116:0/838183479 conn(0x7fdb8c102f60 msgr2=0x7fdb8c103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:53.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.401+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 shutdown_connections 2024-03-04T07:28:53.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:52.401+0000 7fdb91e00700 1 -- 172.21.15.116:0/838183479 wait complete. 2024-03-04T07:28:53.118 INFO:teuthology.orchestra.run.smithi116.stdout:Restarting the monitor... 2024-03-04T07:28:53.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:53 smithi116 systemd[1]: Stopping Ceph mon.a for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:28:53.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a[31775]: 2024-03-04T07:28:53.612+0000 7f274c800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:28:53.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a[31775]: 2024-03-04T07:28:53.612+0000 7f274c800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T07:28:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:53 smithi116 podman[32023]: 2024-03-04 07:28:53.880749351 +0000 UTC m=+0.426507440 container died 1da75f11c245757b1a79c2da06789bc1eb68e2336b6f800d2a69e5a1d2ea57ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-04T07:28:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:55 smithi116 podman[32023]: 2024-03-04 07:28:55.019157808 +0000 UTC m=+1.564915924 container cleanup 1da75f11c245757b1a79c2da06789bc1eb68e2336b6f800d2a69e5a1d2ea57ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-04T07:28:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:55 smithi116 bash[32023]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a 2024-03-04T07:28:55.754 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:55 smithi116 podman[32035]: 2024-03-04 07:28:55.269819772 +0000 UTC m=+1.367419366 container remove 1da75f11c245757b1a79c2da06789bc1eb68e2336b6f800d2a69e5a1d2ea57ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T07:28:56.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:55 smithi116 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-04T07:28:56.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:55 smithi116 systemd[1]: Stopped Ceph mon.a for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:28:56.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:55 smithi116 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a.service: Consumed 1.885s CPU time. 2024-03-04T07:28:56.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:55 smithi116 systemd[1]: Starting Ceph mon.a for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:28:56.432 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 podman[32131]: 2024-03-04 07:28:56.050571052 +0000 UTC m=+0.033858715 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:28:56.432 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 podman[32131]: 2024-03-04 07:28:56.179117007 +0000 UTC m=+0.162404699 container create e0bdece57fe415014a210d8f463ccf707b43a325b45140edf8f73462217462e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:28:56.508 INFO:teuthology.orchestra.run.smithi116.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-04T07:28:56.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 podman[32131]: 2024-03-04 07:28:56.432578512 +0000 UTC m=+0.415866237 container init e0bdece57fe415014a210d8f463ccf707b43a325b45140edf8f73462217462e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-04T07:28:56.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 podman[32131]: 2024-03-04 07:28:56.442035545 +0000 UTC m=+0.425323237 container start e0bdece57fe415014a210d8f463ccf707b43a325b45140edf8f73462217462e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev), process ceph-mon, pid 2 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: pidfile_write: ignore empty --pid-file 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: load: jerasure load: lrc 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: RocksDB version: 7.9.2 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Git sha 0 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Compile date 2024-03-03 23:16:35 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: DB SUMMARY 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: DB Session ID: D8NUAGTCVW4ZC0GQHC01 2024-03-04T07:28:56.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: CURRENT file: CURRENT 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99314 ; 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.error_if_exists: 0 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.create_if_missing: 0 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.paranoid_checks: 1 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.env: 0x56494b0b4b20 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T07:28:56.719 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.info_log: 0x56494d4d5960 2024-03-04T07:28:56.720 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T07:28:56.720 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.statistics: (nil) 2024-03-04T07:28:56.720 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.use_fsync: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_log_file_size: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.allow_fallocate: 1 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.use_direct_reads: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T07:28:56.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.db_log_dir: 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.wal_dir: 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.write_buffer_manager: 0x56494d4a9400 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T07:28:56.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.unordered_write: 0 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.row_cache: None 2024-03-04T07:28:56.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.wal_filter: None 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.two_write_queues: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.wal_compression: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.atomic_flush: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.log_readahead_size: 0 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T07:28:56.724 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_background_jobs: 2 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 bash[32131]: e0bdece57fe415014a210d8f463ccf707b43a325b45140edf8f73462217462e0 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_background_compactions: -1 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_subcompactions: 1 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T07:28:56.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_open_files: -1 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_background_flushes: -1 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Compression algorithms supported: 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T07:28:56.726 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kZSTD supported: 0 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kXpressCompression supported: 0 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kZlibCompression supported: 1 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kSnappyCompression supported: 1 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kLZ4Compression supported: 1 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: kBZip2Compression supported: 0 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.merge_operator: 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_filter: None 2024-03-04T07:28:56.727 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56494d4d5a60) 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_type: 0 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_index_type: 0 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_shortening: 1 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T07:28:56.728 INFO:journalctl@ceph.mon.a.smithi116.stdout: checksum: 4 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: no_block_cache: 0 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache: 0x56494d4d9090 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_options: 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: capacity : 536870912 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_shard_bits : 4 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: strict_capacity_limit : 0 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_compressed: (nil) 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: persistent_cache: (nil) 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size: 4096 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size_deviation: 10 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_restart_interval: 16 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_block_restart_interval: 1 2024-03-04T07:28:56.729 INFO:journalctl@ceph.mon.a.smithi116.stdout: metadata_block_size: 4096 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: partition_filters: 0 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: use_delta_encoding: 1 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: filter_policy: bloomfilter 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: whole_key_filtering: 1 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: verify_compression: 0 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: format_version: 5 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_index_compression: 1 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_align: 0 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: prepopulate_block_cache: 0 2024-03-04T07:28:56.730 INFO:journalctl@ceph.mon.a.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-04T07:28:56.731 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T07:28:56.731 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T07:28:56.731 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression: NoCompression 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.num_levels: 7 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T07:28:56.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T07:28:56.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T07:28:56.734 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T07:28:56.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.table_properties_collectors: 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.inplace_update_support: 0 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T07:28:56.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.bloom_locality: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.max_successive_merges: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.ttl: 2592000 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.enable_blob_files: false 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.min_blob_size: 0 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T07:28:56.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5a146c71-d462-437b-9ff1-a1ef8791ff37 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709537336496703, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709537336499725, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94498, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92509, "index_size": 271, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13660, "raw_average_key_size": 50, "raw_value_size": 85105, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709537336, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5a146c71-d462-437b-9ff1-a1ef8791ff37", "db_session_id": "D8NUAGTCVW4ZC0GQHC01", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709537336499949, "job": 1, "event": "recovery_finished"} 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-04T07:28:56.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56494d5ca000 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: DB pointer 0x56494d5b4000 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** DB Stats ** 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-04T07:28:56.739 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: L0 2/0 94.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 36.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: Sum 2/0 94.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 36.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 36.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 36.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T07:28:56.740 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative compaction: 0.00 GB write, 10.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval compaction: 0.00 GB write, 10.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache BinnedLRUCache@0x56494d4d9090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T07:28:56.741 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: starting mon.a rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 systemd[1]: Started Ceph mon.a for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???) e1 preinit fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???).mds e1 new map 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???).mds e1 print_map 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout: e1 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout: legacy client fscid: -1 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout: No filesystems configured 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T07:28:56.742 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-04T07:28:56.743 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T07:28:56.743 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: monmap e1: 1 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:28:56.743 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: fsmap 2024-03-04T07:28:56.743 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T07:28:56.743 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:56 smithi116 ceph-mon[32150]: mgrmap e1: no daemons active 2024-03-04T07:28:58.006 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.202+0000 7f6a1c000700 1 Processor -- start 2024-03-04T07:28:58.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.202+0000 7f6a1c000700 1 -- start start 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.203+0000 7f6a1c000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a14106c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.203+0000 7f6a1c000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141071e0 con 0x7f6a141067f0 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.204+0000 7f6a19600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a14106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.204+0000 7f6a19600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a14106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51856/0 (socket says 172.21.15.116:51856) 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.204+0000 7f6a19600700 1 -- 172.21.15.116:0/1323632549 learned_addr learned my addr 172.21.15.116:0/1323632549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.204+0000 7f6a19600700 1 -- 172.21.15.116:0/1323632549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a141079f0 con 0x7f6a141067f0 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.205+0000 7f6a19600700 1 --2- 172.21.15.116:0/1323632549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a14106c10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a080098a0 tx=0x7f6a08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ccc84405aeff51 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.205+0000 7f6a13e00700 1 -- 172.21.15.116:0/1323632549 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a08004160 con 0x7f6a141067f0 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.205+0000 7f6a13e00700 1 -- 172.21.15.116:0/1323632549 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a08004300 con 0x7f6a141067f0 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.205+0000 7f6a13e00700 1 -- 172.21.15.116:0/1323632549 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a08011b50 con 0x7f6a141067f0 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.206+0000 7f6a1c000700 1 -- 172.21.15.116:0/1323632549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 msgr2=0x7f6a14106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:58.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.206+0000 7f6a1c000700 1 --2- 172.21.15.116:0/1323632549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a14106c10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a080098a0 tx=0x7f6a08009bb0 comp rx=0 tx=0).stop 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.206+0000 7f6a1c000700 1 -- 172.21.15.116:0/1323632549 shutdown_connections 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.206+0000 7f6a1c000700 1 --2- 172.21.15.116:0/1323632549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a14106c10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.206+0000 7f6a1c000700 1 -- 172.21.15.116:0/1323632549 >> 172.21.15.116:0/1323632549 conn(0x7f6a14101e70 msgr2=0x7f6a14104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.206+0000 7f6a1c000700 1 -- 172.21.15.116:0/1323632549 shutdown_connections 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.206+0000 7f6a1c000700 1 -- 172.21.15.116:0/1323632549 wait complete. 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.207+0000 7f6a1c000700 1 Processor -- start 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.207+0000 7f6a1c000700 1 -- start start 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.208+0000 7f6a1c000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a1419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.208+0000 7f6a1c000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a14079fd0 con 0x7f6a141067f0 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.208+0000 7f6a19600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a1419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.208+0000 7f6a19600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a1419dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51866/0 (socket says 172.21.15.116:51866) 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.208+0000 7f6a19600700 1 -- 172.21.15.116:0/2932906149 learned_addr learned my addr 172.21.15.116:0/2932906149 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:28:58.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.208+0000 7f6a19600700 1 -- 172.21.15.116:0/2932906149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1419e0e0 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a19600700 1 --2- 172.21.15.116:0/2932906149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a1419dba0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a08000c00 tx=0x7f6a08011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a12000700 1 -- 172.21.15.116:0/2932906149 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a08003b20 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a12000700 1 -- 172.21.15.116:0/2932906149 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a08003cc0 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1419e370 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a12000700 1 -- 172.21.15.116:0/2932906149 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a08024910 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1419ee90 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a12000700 1 -- 172.21.15.116:0/2932906149 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6a08022020 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.209+0000 7f6a12000700 1 -- 172.21.15.116:0/2932906149 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6a08011e00 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.210+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a140636f0 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.213+0000 7f6a12000700 1 -- 172.21.15.116:0/2932906149 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6a08042dd0 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.289+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6a140684b0 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.293+0000 7f6a12000700 1 -- 172.21.15.116:0/2932906149 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6a08031020 con 0x7f6a141067f0 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.294+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 msgr2=0x7f6a1419dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:28:58.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.294+0000 7f6a1c000700 1 --2- 172.21.15.116:0/2932906149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a1419dba0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a08000c00 tx=0x7f6a08011c10 comp rx=0 tx=0).stop 2024-03-04T07:28:58.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.295+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 shutdown_connections 2024-03-04T07:28:58.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.295+0000 7f6a1c000700 1 --2- 172.21.15.116:0/2932906149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a141067f0 0x7f6a1419dba0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:28:58.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.295+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 >> 172.21.15.116:0/2932906149 conn(0x7f6a14101e70 msgr2=0x7f6a141940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:28:58.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.295+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 shutdown_connections 2024-03-04T07:28:58.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:28:57.295+0000 7f6a1c000700 1 -- 172.21.15.116:0/2932906149 wait complete. 2024-03-04T07:28:58.015 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-04T07:28:58.029 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:28:58.029 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mgr... 2024-03-04T07:28:58.031 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-04T07:28:58.032 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-04T07:28:58.425 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.x 2024-03-04T07:28:58.426 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.x.service: Unit ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.x.service not loaded. 2024-03-04T07:28:58.677 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715.target.wants/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@.service. 2024-03-04T07:28:58.705 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:28:58 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2932906149' entity='client.admin' 2024-03-04T07:28:59.481 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T07:28:59.481 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T07:28:59.499 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T07:28:59.499 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T07:28:59.499 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T07:28:59.499 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T07:28:59.522 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T07:28:59.522 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T07:28:59.539 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T07:28:59.539 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T07:28:59.539 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T07:28:59.539 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-04T07:28:59.539 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr to start... 2024-03-04T07:28:59.539 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr... 2024-03-04T07:28:59.690 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:28:59 smithi116 systemd[1]: Started Ceph mgr.x for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:29:00.270 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:00 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:00.134+0000 7f80a60c4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T07:29:00.270 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:00 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:00.258+0000 7f80a60c4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T07:29:00.601 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:00 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4148714991' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "af2ccd48-d9f8-11ee-95c6-87774f69a715", 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:00.910 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:00.911 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T07:29:00.912 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T07:29:00.912 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T07:29:00.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T07:29:00.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T07:29:00.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:00.920 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T07:28:48.961340+0000", 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.162+0000 7f0d3c600700 1 Processor -- start 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.162+0000 7f0d3c600700 1 -- start start 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.162+0000 7f0d3c600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d34106c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.162+0000 7f0d3c600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341071e0 con 0x7f0d341067f0 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.163+0000 7f0d39c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d34106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.163+0000 7f0d39c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d34106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51878/0 (socket says 172.21.15.116:51878) 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.163+0000 7f0d39c00700 1 -- 172.21.15.116:0/38111637 learned_addr learned my addr 172.21.15.116:0/38111637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:00.922 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.163+0000 7f0d39c00700 1 -- 172.21.15.116:0/38111637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d341079f0 con 0x7f0d341067f0 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.164+0000 7f0d39c00700 1 --2- 172.21.15.116:0/38111637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d34106c10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0d280098a0 tx=0x7f0d28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274a48fcded2fa16 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.164+0000 7f0d33e00700 1 -- 172.21.15.116:0/38111637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d28004160 con 0x7f0d341067f0 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.164+0000 7f0d33e00700 1 -- 172.21.15.116:0/38111637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d28004300 con 0x7f0d341067f0 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.164+0000 7f0d33e00700 1 -- 172.21.15.116:0/38111637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d28011b50 con 0x7f0d341067f0 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.165+0000 7f0d3c600700 1 -- 172.21.15.116:0/38111637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 msgr2=0x7f0d34106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.165+0000 7f0d3c600700 1 --2- 172.21.15.116:0/38111637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d34106c10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0d280098a0 tx=0x7f0d28009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.165+0000 7f0d3c600700 1 -- 172.21.15.116:0/38111637 shutdown_connections 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.165+0000 7f0d3c600700 1 --2- 172.21.15.116:0/38111637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d34106c10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.165+0000 7f0d3c600700 1 -- 172.21.15.116:0/38111637 >> 172.21.15.116:0/38111637 conn(0x7f0d34101e70 msgr2=0x7f0d34104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.165+0000 7f0d3c600700 1 -- 172.21.15.116:0/38111637 shutdown_connections 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.165+0000 7f0d3c600700 1 -- 172.21.15.116:0/38111637 wait complete. 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.166+0000 7f0d3c600700 1 Processor -- start 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.166+0000 7f0d3c600700 1 -- start start 2024-03-04T07:29:00.923 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.166+0000 7f0d3c600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d3419bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.166+0000 7f0d3c600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3410ada0 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.167+0000 7f0d39c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d3419bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.167+0000 7f0d39c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d3419bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51886/0 (socket says 172.21.15.116:51886) 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.167+0000 7f0d39c00700 1 -- 172.21.15.116:0/4148714991 learned_addr learned my addr 172.21.15.116:0/4148714991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.167+0000 7f0d39c00700 1 -- 172.21.15.116:0/4148714991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3419c230 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.167+0000 7f0d39c00700 1 --2- 172.21.15.116:0/4148714991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d3419bcf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009fb0 tx=0x7f0d28011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.168+0000 7f0d32000700 1 -- 172.21.15.116:0/4148714991 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d280038b0 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.168+0000 7f0d32000700 1 -- 172.21.15.116:0/4148714991 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d28011e50 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.168+0000 7f0d32000700 1 -- 172.21.15.116:0/4148714991 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d28019550 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.168+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3419c4c0 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.168+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3419f110 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.168+0000 7f0d32000700 1 -- 172.21.15.116:0/4148714991 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0d28019b90 con 0x7f0d341067f0 2024-03-04T07:29:00.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.168+0000 7f0d32000700 1 -- 172.21.15.116:0/4148714991 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0d28024730 con 0x7f0d341067f0 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.169+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d34069160 con 0x7f0d341067f0 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.172+0000 7f0d32000700 1 -- 172.21.15.116:0/4148714991 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0d28003a50 con 0x7f0d341067f0 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.250+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0d3402d9b0 con 0x7f0d341067f0 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.251+0000 7f0d32000700 1 -- 172.21.15.116:0/4148714991 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f0d28024930 con 0x7f0d341067f0 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.253+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 msgr2=0x7f0d3419bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.253+0000 7f0d3c600700 1 --2- 172.21.15.116:0/4148714991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d3419bcf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009fb0 tx=0x7f0d28011bf0 comp rx=0 tx=0).stop 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.253+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 shutdown_connections 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.253+0000 7f0d3c600700 1 --2- 172.21.15.116:0/4148714991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d341067f0 0x7f0d3419bcf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.253+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 >> 172.21.15.116:0/4148714991 conn(0x7f0d34101e70 msgr2=0x7f0d34102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.253+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 shutdown_connections 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:00.253+0000 7f0d3c600700 1 -- 172.21.15.116:0/4148714991 wait complete. 2024-03-04T07:29:00.925 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (1/15)... 2024-03-04T07:29:01.198 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:01 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:01.079+0000 7f80a60c4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T07:29:01.490 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:01 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:01.197+0000 7f80a60c4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T07:29:01.490 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:01 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:01.368+0000 7f80a60c4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T07:29:01.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:01 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:01.489+0000 7f80a60c4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T07:29:01.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:01 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:01.595+0000 7f80a60c4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T07:29:02.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:01 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:01.918+0000 7f80a60c4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T07:29:02.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:02 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:02.054+0000 7f80a60c4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T07:29:02.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:02 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:02.164+0000 7f80a60c4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T07:29:02.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:02 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:02.282+0000 7f80a60c4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T07:29:02.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:02 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:02.494+0000 7f80a60c4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T07:29:04.000 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:03 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3201285140' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T07:29:04.000 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:03 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:03.827+0000 7f80a60c4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T07:29:04.000 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:04 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:03.998+0000 7f80a60c4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T07:29:04.419 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:29:04.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:04.420 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "af2ccd48-d9f8-11ee-95c6-87774f69a715", 2024-03-04T07:29:04.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T07:29:04.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T07:29:04.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T07:29:04.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T07:29:04.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:04.421 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T07:29:04.423 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T07:29:04.424 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:04.424 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T07:29:04.424 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:04.424 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T07:29:04.424 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T07:29:04.424 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T07:29:04.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T07:29:04.428 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T07:29:04.429 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:04.430 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T07:28:48.961340+0000", 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.545+0000 7f2f30c00700 1 Processor -- start 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.545+0000 7f2f30c00700 1 -- start start 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.545+0000 7f2f30c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.545+0000 7f2f30c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c1082f0 con 0x7f2f2c107900 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.546+0000 7f2f29e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.546+0000 7f2f29e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51902/0 (socket says 172.21.15.116:51902) 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.546+0000 7f2f29e00700 1 -- 172.21.15.116:0/2141367589 learned_addr learned my addr 172.21.15.116:0/2141367589 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.546+0000 7f2f29e00700 1 -- 172.21.15.116:0/2141367589 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c108ab0 con 0x7f2f2c107900 2024-03-04T07:29:04.431 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.547+0000 7f2f29e00700 1 --2- 172.21.15.116:0/2141367589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c107d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0098a0 tx=0x7f2f1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a28e615bdc7642 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:04.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.547+0000 7f2f28a00700 1 -- 172.21.15.116:0/2141367589 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f1c004160 con 0x7f2f2c107900 2024-03-04T07:29:04.435 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.547+0000 7f2f28a00700 1 -- 172.21.15.116:0/2141367589 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f1c004300 con 0x7f2f2c107900 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.547+0000 7f2f28a00700 1 -- 172.21.15.116:0/2141367589 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f1c011b50 con 0x7f2f2c107900 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.548+0000 7f2f30c00700 1 -- 172.21.15.116:0/2141367589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 msgr2=0x7f2f2c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.548+0000 7f2f30c00700 1 --2- 172.21.15.116:0/2141367589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c107d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0098a0 tx=0x7f2f1c009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.548+0000 7f2f30c00700 1 -- 172.21.15.116:0/2141367589 shutdown_connections 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.548+0000 7f2f30c00700 1 --2- 172.21.15.116:0/2141367589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c107d20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.548+0000 7f2f30c00700 1 -- 172.21.15.116:0/2141367589 >> 172.21.15.116:0/2141367589 conn(0x7f2f2c102f60 msgr2=0x7f2f2c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.548+0000 7f2f30c00700 1 -- 172.21.15.116:0/2141367589 shutdown_connections 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.548+0000 7f2f30c00700 1 -- 172.21.15.116:0/2141367589 wait complete. 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.549+0000 7f2f30c00700 1 Processor -- start 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.549+0000 7f2f30c00700 1 -- start start 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.549+0000 7f2f30c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.549+0000 7f2f30c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c07b0e0 con 0x7f2f2c107900 2024-03-04T07:29:04.436 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.550+0000 7f2f29e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.550+0000 7f2f29e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51906/0 (socket says 172.21.15.116:51906) 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.550+0000 7f2f29e00700 1 -- 172.21.15.116:0/3201285140 learned_addr learned my addr 172.21.15.116:0/3201285140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.550+0000 7f2f29e00700 1 -- 172.21.15.116:0/3201285140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c19e2e0 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.550+0000 7f2f29e00700 1 --2- 172.21.15.116:0/3201285140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c19dda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c000c00 tx=0x7f2f1c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.551+0000 7f2f22a00700 1 -- 172.21.15.116:0/3201285140 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f1c003b20 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.551+0000 7f2f22a00700 1 -- 172.21.15.116:0/3201285140 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f1c003cc0 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.551+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2c19e570 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.551+0000 7f2f22a00700 1 -- 172.21.15.116:0/3201285140 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f1c024910 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.551+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c1a1260 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.551+0000 7f2f22a00700 1 -- 172.21.15.116:0/3201285140 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2f1c022020 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.551+0000 7f2f22a00700 1 -- 172.21.15.116:0/3201285140 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2f1c011e00 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.552+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c069160 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.555+0000 7f2f22a00700 1 -- 172.21.15.116:0/3201285140 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2f1c01acd0 con 0x7f2f2c107900 2024-03-04T07:29:04.437 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.633+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2f2c02d9b0 con 0x7f2f2c107900 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.634+0000 7f2f22a00700 1 -- 172.21.15.116:0/3201285140 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f2f1c032030 con 0x7f2f2c107900 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.635+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 msgr2=0x7f2f2c19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.635+0000 7f2f30c00700 1 --2- 172.21.15.116:0/3201285140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c19dda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c000c00 tx=0x7f2f1c011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.635+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 shutdown_connections 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.635+0000 7f2f30c00700 1 --2- 172.21.15.116:0/3201285140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f2c107900 0x7f2f2c19dda0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.635+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 >> 172.21.15.116:0/3201285140 conn(0x7f2f2c102f60 msgr2=0x7f2f2c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.635+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 shutdown_connections 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:03.635+0000 7f2f30c00700 1 -- 172.21.15.116:0/3201285140 wait complete. 2024-03-04T07:29:04.438 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (2/15)... 2024-03-04T07:29:04.759 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:04 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:04.413+0000 7f80a60c4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T07:29:04.760 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:04 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:04.535+0000 7f80a60c4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T07:29:05.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:04 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:04.796+0000 7f80a60c4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:29:05.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:05 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:05.035+0000 7f80a60c4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T07:29:05.709 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:05 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:05.397+0000 7f80a60c4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:29:06.005 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:05 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:05.708+0000 7f80a60c4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T07:29:06.006 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:06 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:06.004+0000 7f80a60c4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T07:29:07.106 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:07 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:07.039+0000 7f80a60c4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T07:29:07.462 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:07 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1942724978' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "af2ccd48-d9f8-11ee-95c6-87774f69a715", 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T07:29:07.814 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T07:29:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T07:29:07.816 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:07.817 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T07:28:48.961340+0000", 2024-03-04T07:29:07.818 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:06.999+0000 7f0429c00700 1 Processor -- start 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:06.999+0000 7f0429c00700 1 -- start start 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.000+0000 7f0429c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f0424107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.000+0000 7f0429c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04241082f0 con 0x7f0424107900 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.000+0000 7f0422a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f0424107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.000+0000 7f0422a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f0424107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51630/0 (socket says 172.21.15.116:51630) 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.000+0000 7f0422a00700 1 -- 172.21.15.116:0/707749311 learned_addr learned my addr 172.21.15.116:0/707749311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.000+0000 7f0422a00700 1 -- 172.21.15.116:0/707749311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0424108ab0 con 0x7f0424107900 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.001+0000 7f0422a00700 1 --2- 172.21.15.116:0/707749311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f0424107d20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f04140098a0 tx=0x7f0414009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b40ca074847d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:07.819 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.001+0000 7f0421600700 1 -- 172.21.15.116:0/707749311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0414004160 con 0x7f0424107900 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.001+0000 7f0421600700 1 -- 172.21.15.116:0/707749311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0414004300 con 0x7f0424107900 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.001+0000 7f0421600700 1 -- 172.21.15.116:0/707749311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0414011b50 con 0x7f0424107900 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.002+0000 7f0429c00700 1 -- 172.21.15.116:0/707749311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 msgr2=0x7f0424107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.002+0000 7f0429c00700 1 --2- 172.21.15.116:0/707749311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f0424107d20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f04140098a0 tx=0x7f0414009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.002+0000 7f0429c00700 1 -- 172.21.15.116:0/707749311 shutdown_connections 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.002+0000 7f0429c00700 1 --2- 172.21.15.116:0/707749311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f0424107d20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.002+0000 7f0429c00700 1 -- 172.21.15.116:0/707749311 >> 172.21.15.116:0/707749311 conn(0x7f0424102f60 msgr2=0x7f0424105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.002+0000 7f0429c00700 1 -- 172.21.15.116:0/707749311 shutdown_connections 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.002+0000 7f0429c00700 1 -- 172.21.15.116:0/707749311 wait complete. 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.004+0000 7f0429c00700 1 Processor -- start 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.004+0000 7f0429c00700 1 -- start start 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.004+0000 7f0429c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f042419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:07.820 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.004+0000 7f0429c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042410be60 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.005+0000 7f0422a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f042419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.005+0000 7f0422a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f042419ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51646/0 (socket says 172.21.15.116:51646) 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.005+0000 7f0422a00700 1 -- 172.21.15.116:0/1942724978 learned_addr learned my addr 172.21.15.116:0/1942724978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.005+0000 7f0422a00700 1 -- 172.21.15.116:0/1942724978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042419e310 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.005+0000 7f0422a00700 1 --2- 172.21.15.116:0/1942724978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f042419ddd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0414000c00 tx=0x7f0414011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.006+0000 7f041b400700 1 -- 172.21.15.116:0/1942724978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0414003b20 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.006+0000 7f041b400700 1 -- 172.21.15.116:0/1942724978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0414003cc0 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.006+0000 7f041b400700 1 -- 172.21.15.116:0/1942724978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0414024910 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.006+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042419e5a0 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.006+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04241a1290 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.006+0000 7f041b400700 1 -- 172.21.15.116:0/1942724978 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0414022020 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.006+0000 7f041b400700 1 -- 172.21.15.116:0/1942724978 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0414011e00 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.007+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0424069160 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.009+0000 7f041b400700 1 -- 172.21.15.116:0/1942724978 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0414042dd0 con 0x7f0424107900 2024-03-04T07:29:07.821 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.088+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f042402d9b0 con 0x7f0424107900 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.089+0000 7f041b400700 1 -- 172.21.15.116:0/1942724978 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f0414031020 con 0x7f0424107900 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.090+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 msgr2=0x7f042419ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.090+0000 7f0429c00700 1 --2- 172.21.15.116:0/1942724978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f042419ddd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0414000c00 tx=0x7f0414011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.090+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 shutdown_connections 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.090+0000 7f0429c00700 1 --2- 172.21.15.116:0/1942724978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0424107900 0x7f042419ddd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.090+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 >> 172.21.15.116:0/1942724978 conn(0x7f0424102f60 msgr2=0x7f0424103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.091+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 shutdown_connections 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:07.091+0000 7f0429c00700 1 -- 172.21.15.116:0/1942724978 wait complete. 2024-03-04T07:29:07.822 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (3/15)... 2024-03-04T07:29:08.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:07 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:07.741+0000 7f80a60c4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T07:29:08.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:07 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:07.871+0000 7f80a60c4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T07:29:08.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:08.144+0000 7f80a60c4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T07:29:08.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: Activating manager daemon x 2024-03-04T07:29:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: mgrmap e2: x(active, starting, since 0.0110073s) 2024-03-04T07:29:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T07:29:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T07:29:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T07:29:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:29:08.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T07:29:08.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: Manager daemon x is now available 2024-03-04T07:29:08.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T07:29:08.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T07:29:08.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' 2024-03-04T07:29:08.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:08 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' 2024-03-04T07:29:09.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:09 smithi116 ceph-mon[32150]: from='mgr.14100 172.21.15.116:0/4093265512' entity='mgr.x' 2024-03-04T07:29:09.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:09 smithi116 ceph-mon[32150]: mgrmap e3: x(active, since 1.02073s) 2024-03-04T07:29:11.429 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:11 smithi116 ceph-mon[32150]: mgrmap e4: x(active, since 2s) 2024-03-04T07:29:11.429 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1692202020' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T07:29:11.748 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "af2ccd48-d9f8-11ee-95c6-87774f69a715", 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T07:29:11.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:11.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-04T07:29:11.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T07:29:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T07:29:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T07:29:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T07:29:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T07:29:11.754 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T07:29:11.754 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T07:29:11.754 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T07:29:11.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T07:29:11.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T07:28:48.961340+0000", 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-04T07:29:11.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.452+0000 7f598a600700 1 Processor -- start 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.453+0000 7f598a600700 1 -- start start 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.453+0000 7f598a600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f598407b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.453+0000 7f598a600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598407b910 con 0x7f598407ce90 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.454+0000 7f5983400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f598407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.454+0000 7f5983400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f598407b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51718/0 (socket says 172.21.15.116:51718) 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.454+0000 7f5983400700 1 -- 172.21.15.116:0/799305854 learned_addr learned my addr 172.21.15.116:0/799305854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.454+0000 7f5983400700 1 -- 172.21.15.116:0/799305854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598407ba90 con 0x7f598407ce90 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.455+0000 7f5983400700 1 --2- 172.21.15.116:0/799305854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f598407b340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f59740098a0 tx=0x7f5974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b0bc26bf622774 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.455+0000 7f5982000700 1 -- 172.21.15.116:0/799305854 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5974004160 con 0x7f598407ce90 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.455+0000 7f5982000700 1 -- 172.21.15.116:0/799305854 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5974004300 con 0x7f598407ce90 2024-03-04T07:29:11.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.455+0000 7f5982000700 1 -- 172.21.15.116:0/799305854 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5974011b50 con 0x7f598407ce90 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.455+0000 7f598a600700 1 -- 172.21.15.116:0/799305854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 msgr2=0x7f598407b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.455+0000 7f598a600700 1 --2- 172.21.15.116:0/799305854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f598407b340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f59740098a0 tx=0x7f5974009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.456+0000 7f598a600700 1 -- 172.21.15.116:0/799305854 shutdown_connections 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.456+0000 7f598a600700 1 --2- 172.21.15.116:0/799305854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f598407b340 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.456+0000 7f598a600700 1 -- 172.21.15.116:0/799305854 >> 172.21.15.116:0/799305854 conn(0x7f5984102f60 msgr2=0x7f5984105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.456+0000 7f598a600700 1 -- 172.21.15.116:0/799305854 shutdown_connections 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.456+0000 7f598a600700 1 -- 172.21.15.116:0/799305854 wait complete. 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.457+0000 7f598a600700 1 Processor -- start 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.457+0000 7f598a600700 1 -- start start 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.457+0000 7f598a600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f59841a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.457+0000 7f598a600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984109ac0 con 0x7f598407ce90 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.457+0000 7f5983400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f59841a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:11.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.457+0000 7f5983400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f59841a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51728/0 (socket says 172.21.15.116:51728) 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.457+0000 7f5983400700 1 -- 172.21.15.116:0/1692202020 learned_addr learned my addr 172.21.15.116:0/1692202020 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.458+0000 7f5983400700 1 -- 172.21.15.116:0/1692202020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59841a2670 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.458+0000 7f5983400700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f59841a2130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5974011b60 tx=0x7f5974011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.458+0000 7f597be00700 1 -- 172.21.15.116:0/1692202020 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5974003b20 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.458+0000 7f597be00700 1 -- 172.21.15.116:0/1692202020 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5974003cc0 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.459+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59841a2900 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.459+0000 7f597be00700 1 -- 172.21.15.116:0/1692202020 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5974024840 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.459+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59841a55f0 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.460+0000 7f597be00700 1 -- 172.21.15.116:0/1692202020 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45917+0+0 (secure 0 0 0) 0x7f5974022020 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.460+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984069160 con 0x7f598407ce90 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.460+0000 7f597be00700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f5968038b50 0x7f596803b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:11.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.460+0000 7f597be00700 1 -- 172.21.15.116:0/1692202020 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5974011e00 con 0x7f598407ce90 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.460+0000 7f5982a00700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f5968038b50 0x7f596803b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.461+0000 7f5982a00700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f5968038b50 0x7f596803b010 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5970006fd0 tx=0x7f5970006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.466+0000 7f597be00700 1 -- 172.21.15.116:0/1692202020 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f597400b3e0 con 0x7f598407ce90 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.769+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f598402d9b0 con 0x7f598407ce90 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.770+0000 7f597be00700 1 -- 172.21.15.116:0/1692202020 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f5974016260 con 0x7f598407ce90 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.773+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f5968038b50 msgr2=0x7f596803b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.773+0000 7f598a600700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f5968038b50 0x7f596803b010 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5970006fd0 tx=0x7f5970006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.773+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 msgr2=0x7f59841a2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.773+0000 7f598a600700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f59841a2130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5974011b60 tx=0x7f5974011c40 comp rx=0 tx=0).stop 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.773+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 shutdown_connections 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.773+0000 7f598a600700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f5968038b50 0x7f596803b010 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:11.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.774+0000 7f598a600700 1 --2- 172.21.15.116:0/1692202020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598407ce90 0x7f59841a2130 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:11.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.774+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 >> 172.21.15.116:0/1692202020 conn(0x7f5984102f60 msgr2=0x7f5984198bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:11.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.774+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 shutdown_connections 2024-03-04T07:29:11.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:10.774+0000 7f598a600700 1 -- 172.21.15.116:0/1692202020 wait complete. 2024-03-04T07:29:11.770 INFO:teuthology.orchestra.run.smithi116.stdout:mgr is available 2024-03-04T07:29:12.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:12 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3200193541' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T07:29:13.289 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:29:13.290 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T07:29:13.290 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T07:29:13.291 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-04T07:29:13.297 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.355+0000 7f229c600700 1 Processor -- start 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.355+0000 7f229c600700 1 -- start start 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.356+0000 7f229c600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f2294109f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.356+0000 7f229c600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229410a520 con 0x7f2294109b30 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.356+0000 7f2299c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f2294109f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.356+0000 7f2299c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f2294109f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51742/0 (socket says 172.21.15.116:51742) 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.356+0000 7f2299c00700 1 -- 172.21.15.116:0/3423341939 learned_addr learned my addr 172.21.15.116:0/3423341939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.357+0000 7f2299c00700 1 -- 172.21.15.116:0/3423341939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229410ace0 con 0x7f2294109b30 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.357+0000 7f2299c00700 1 --2- 172.21.15.116:0/3423341939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f2294109f50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f22880098a0 tx=0x7f2288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a995096448ebc5a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.357+0000 7f2293e00700 1 -- 172.21.15.116:0/3423341939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2288004160 con 0x7f2294109b30 2024-03-04T07:29:13.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.357+0000 7f2293e00700 1 -- 172.21.15.116:0/3423341939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2288004300 con 0x7f2294109b30 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.357+0000 7f2293e00700 1 -- 172.21.15.116:0/3423341939 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2288011b50 con 0x7f2294109b30 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.358+0000 7f229c600700 1 -- 172.21.15.116:0/3423341939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 msgr2=0x7f2294109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.358+0000 7f229c600700 1 --2- 172.21.15.116:0/3423341939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f2294109f50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f22880098a0 tx=0x7f2288009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.358+0000 7f229c600700 1 -- 172.21.15.116:0/3423341939 shutdown_connections 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.358+0000 7f229c600700 1 --2- 172.21.15.116:0/3423341939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f2294109f50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.358+0000 7f229c600700 1 -- 172.21.15.116:0/3423341939 >> 172.21.15.116:0/3423341939 conn(0x7f22941054c0 msgr2=0x7f22941078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.358+0000 7f229c600700 1 -- 172.21.15.116:0/3423341939 shutdown_connections 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.358+0000 7f229c600700 1 -- 172.21.15.116:0/3423341939 wait complete. 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.359+0000 7f229c600700 1 Processor -- start 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.359+0000 7f229c600700 1 -- start start 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.360+0000 7f229c600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f229419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.360+0000 7f229c600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229410e090 con 0x7f2294109b30 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.360+0000 7f2299c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f229419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:13.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.360+0000 7f2299c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f229419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51744/0 (socket says 172.21.15.116:51744) 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.360+0000 7f2299c00700 1 -- 172.21.15.116:0/3200193541 learned_addr learned my addr 172.21.15.116:0/3200193541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.360+0000 7f2299c00700 1 -- 172.21.15.116:0/3200193541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229419e380 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.360+0000 7f2299c00700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f229419de40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2288009fb0 tx=0x7f2288011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.361+0000 7f2292000700 1 -- 172.21.15.116:0/3200193541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22880038b0 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.361+0000 7f2292000700 1 -- 172.21.15.116:0/3200193541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2288011e50 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.361+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229419e610 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.361+0000 7f2292000700 1 -- 172.21.15.116:0/3200193541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2288019550 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.361+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22941a1300 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.362+0000 7f2292000700 1 -- 172.21.15.116:0/3200193541 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45917+0+0 (secure 0 0 0) 0x7f2288003a50 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.362+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2294069160 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.363+0000 7f2292000700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f227c038b50 0x7f227c03b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.363+0000 7f2292000700 1 -- 172.21.15.116:0/3200193541 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f228804cbc0 con 0x7f2294109b30 2024-03-04T07:29:13.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.363+0000 7f2299200700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f227c038b50 0x7f227c03b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.364+0000 7f2299200700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f227c038b50 0x7f227c03b010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2280006fd0 tx=0x7f2280006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.368+0000 7f2292000700 1 -- 172.21.15.116:0/3200193541 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f228804c5e0 con 0x7f2294109b30 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.585+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f22941a2650 con 0x7f2294109b30 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.586+0000 7f2292000700 1 -- 172.21.15.116:0/3200193541 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f2288023390 con 0x7f2294109b30 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.589+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f227c038b50 msgr2=0x7f227c03b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.589+0000 7f229c600700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f227c038b50 0x7f227c03b010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2280006fd0 tx=0x7f2280006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.589+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 msgr2=0x7f229419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.589+0000 7f229c600700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f229419de40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2288009fb0 tx=0x7f2288011bf0 comp rx=0 tx=0).stop 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.590+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 shutdown_connections 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.590+0000 7f229c600700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f227c038b50 0x7f227c03b010 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.590+0000 7f229c600700 1 --2- 172.21.15.116:0/3200193541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2294109b30 0x7f229419de40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.590+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 >> 172.21.15.116:0/3200193541 conn(0x7f22941054c0 msgr2=0x7f2294107330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.590+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 shutdown_connections 2024-03-04T07:29:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:12.590+0000 7f229c600700 1 -- 172.21.15.116:0/3200193541 wait complete. 2024-03-04T07:29:13.302 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling cephadm module... 2024-03-04T07:29:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/523738797' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-04T07:29:14.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: ignoring --setuser ceph since I am not root 2024-03-04T07:29:14.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: ignoring --setgroup ceph since I am not root 2024-03-04T07:29:15.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.881+0000 7fb093c00700 1 Processor -- start 2024-03-04T07:29:15.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.881+0000 7fb093c00700 1 -- start start 2024-03-04T07:29:15.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.882+0000 7fb093c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c109f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:15.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.882+0000 7fb093c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c10a510 con 0x7fb08c109b20 2024-03-04T07:29:15.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.882+0000 7fb091200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c109f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.882+0000 7fb091200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c109f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51754/0 (socket says 172.21.15.116:51754) 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.882+0000 7fb091200700 1 -- 172.21.15.116:0/1525539122 learned_addr learned my addr 172.21.15.116:0/1525539122 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.882+0000 7fb091200700 1 -- 172.21.15.116:0/1525539122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c10acd0 con 0x7fb08c109b20 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.883+0000 7fb091200700 1 --2- 172.21.15.116:0/1525539122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c109f40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0800098a0 tx=0x7fb080009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a151d65585892b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.883+0000 7fb08b400700 1 -- 172.21.15.116:0/1525539122 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb080004160 con 0x7fb08c109b20 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.883+0000 7fb08b400700 1 -- 172.21.15.116:0/1525539122 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb080004300 con 0x7fb08c109b20 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.883+0000 7fb08b400700 1 -- 172.21.15.116:0/1525539122 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb080011b50 con 0x7fb08c109b20 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.884+0000 7fb093c00700 1 -- 172.21.15.116:0/1525539122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 msgr2=0x7fb08c109f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.884+0000 7fb093c00700 1 --2- 172.21.15.116:0/1525539122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c109f40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0800098a0 tx=0x7fb080009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.884+0000 7fb093c00700 1 -- 172.21.15.116:0/1525539122 shutdown_connections 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.884+0000 7fb093c00700 1 --2- 172.21.15.116:0/1525539122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c109f40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.884+0000 7fb093c00700 1 -- 172.21.15.116:0/1525539122 >> 172.21.15.116:0/1525539122 conn(0x7fb08c1054c0 msgr2=0x7fb08c1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.884+0000 7fb093c00700 1 -- 172.21.15.116:0/1525539122 shutdown_connections 2024-03-04T07:29:15.015 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.884+0000 7fb093c00700 1 -- 172.21.15.116:0/1525539122 wait complete. 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.885+0000 7fb093c00700 1 Processor -- start 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.885+0000 7fb093c00700 1 -- start start 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.886+0000 7fb093c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.886+0000 7fb093c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c10e080 con 0x7fb08c109b20 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.886+0000 7fb091200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.886+0000 7fb091200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c19db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51758/0 (socket says 172.21.15.116:51758) 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.886+0000 7fb091200700 1 -- 172.21.15.116:0/523738797 learned_addr learned my addr 172.21.15.116:0/523738797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.886+0000 7fb091200700 1 -- 172.21.15.116:0/523738797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c19e0a0 con 0x7fb08c109b20 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.886+0000 7fb091200700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c19db60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb080000c00 tx=0x7fb080011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.887+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb080003b20 con 0x7fb08c109b20 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.887+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb080003cc0 con 0x7fb08c109b20 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.887+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08c19e330 con 0x7fb08c109b20 2024-03-04T07:29:15.016 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.887+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb080024910 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.887+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08c19ee50 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.888+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 45917+0+0 (secure 0 0 0) 0x7fb080022020 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.888+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08c0636f0 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.888+0000 7fb089600700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7fb074038b50 0x7fb07403b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.888+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb080011e00 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.889+0000 7fb08be00700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7fb074038b50 0x7fb07403b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.889+0000 7fb08be00700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7fb074038b50 0x7fb07403b010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb078006fd0 tx=0x7fb078006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:13.894+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb080016260 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.171+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fb08c0684b0 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.225+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fb08c0684b0 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.229+0000 7fb089600700 1 -- 172.21.15.116:0/523738797 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 5) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fb08001a950 con 0x7fb08c109b20 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7fb074038b50 msgr2=0x7fb07403b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7fb074038b50 0x7fb07403b010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb078006fd0 tx=0x7fb078006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 msgr2=0x7fb08c19db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:15.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c19db60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb080000c00 tx=0x7fb080011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 shutdown_connections 2024-03-04T07:29:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7fb074038b50 0x7fb07403b010 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 --2- 172.21.15.116:0/523738797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08c109b20 0x7fb08c19db60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 >> 172.21.15.116:0/523738797 conn(0x7fb08c1054c0 msgr2=0x7fb08c106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 shutdown_connections 2024-03-04T07:29:15.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:14.231+0000 7fb093c00700 1 -- 172.21.15.116:0/523738797 wait complete. 2024-03-04T07:29:15.256 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/523738797' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-04T07:29:15.257 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:15 smithi116 ceph-mon[32150]: mgrmap e5: x(active, since 6s) 2024-03-04T07:29:15.257 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:15 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:15.001+0000 7f0856a9e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T07:29:15.257 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:15 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:15.125+0000 7f0856a9e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T07:29:16.071 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:15 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:15.950+0000 7f0856a9e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T07:29:16.377 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:16 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3930653336' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T07:29:16.378 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:16.070+0000 7f0856a9e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T07:29:16.378 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:16.254+0000 7f0856a9e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T07:29:16.378 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:16.376+0000 7f0856a9e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T07:29:16.599 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:16.599 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-04T07:29:16.599 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T07:29:16.599 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.571+0000 7f7c78a00700 1 Processor -- start 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.571+0000 7f7c78a00700 1 -- start start 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.571+0000 7f7c78a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c74107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.571+0000 7f7c78a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c741082f0 con 0x7f7c74107900 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.571+0000 7f7c71e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c74107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.572+0000 7f7c71e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c74107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51798/0 (socket says 172.21.15.116:51798) 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.572+0000 7f7c71e00700 1 -- 172.21.15.116:0/2911205601 learned_addr learned my addr 172.21.15.116:0/2911205601 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.572+0000 7f7c71e00700 1 -- 172.21.15.116:0/2911205601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c74108ab0 con 0x7f7c74107900 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.572+0000 7f7c71e00700 1 --2- 172.21.15.116:0/2911205601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c74107d20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098a0 tx=0x7f7c64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e1a363ffd6c492 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:16.600 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.573+0000 7f7c70a00700 1 -- 172.21.15.116:0/2911205601 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c64004160 con 0x7f7c74107900 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.573+0000 7f7c70a00700 1 -- 172.21.15.116:0/2911205601 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c64004300 con 0x7f7c74107900 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.573+0000 7f7c70a00700 1 -- 172.21.15.116:0/2911205601 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c64011b50 con 0x7f7c74107900 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.573+0000 7f7c78a00700 1 -- 172.21.15.116:0/2911205601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 msgr2=0x7f7c74107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.573+0000 7f7c78a00700 1 --2- 172.21.15.116:0/2911205601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c74107d20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c640098a0 tx=0x7f7c64009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.574+0000 7f7c78a00700 1 -- 172.21.15.116:0/2911205601 shutdown_connections 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.574+0000 7f7c78a00700 1 --2- 172.21.15.116:0/2911205601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c74107d20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.574+0000 7f7c78a00700 1 -- 172.21.15.116:0/2911205601 >> 172.21.15.116:0/2911205601 conn(0x7f7c74102f60 msgr2=0x7f7c74105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.574+0000 7f7c78a00700 1 -- 172.21.15.116:0/2911205601 shutdown_connections 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.574+0000 7f7c78a00700 1 -- 172.21.15.116:0/2911205601 wait complete. 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.575+0000 7f7c78a00700 1 Processor -- start 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.576+0000 7f7c78a00700 1 -- start start 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.576+0000 7f7c78a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c7419bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.576+0000 7f7c78a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7410be60 con 0x7f7c74107900 2024-03-04T07:29:16.601 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.576+0000 7f7c71e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c7419bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.576+0000 7f7c71e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c7419bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51814/0 (socket says 172.21.15.116:51814) 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.576+0000 7f7c71e00700 1 -- 172.21.15.116:0/3930653336 learned_addr learned my addr 172.21.15.116:0/3930653336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.577+0000 7f7c71e00700 1 -- 172.21.15.116:0/3930653336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7419c130 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.577+0000 7f7c71e00700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c7419bbf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.577+0000 7f7c6aa00700 1 -- 172.21.15.116:0/3930653336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c64003b20 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.577+0000 7f7c6aa00700 1 -- 172.21.15.116:0/3930653336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c64003cc0 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.577+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7419c3c0 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.577+0000 7f7c6aa00700 1 -- 172.21.15.116:0/3930653336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c64024910 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.577+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7419f060 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.578+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c74069160 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.578+0000 7f7c6aa00700 1 -- 172.21.15.116:0/3930653336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f7c64022020 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.579+0000 7f7c6aa00700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 0x7f7c5803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.579+0000 7f7c6aa00700 1 -- 172.21.15.116:0/3930653336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7c64011e00 con 0x7f7c74107900 2024-03-04T07:29:16.602 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.579+0000 7f7c71400700 1 -- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 msgr2=0x7f7c5803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1181502267 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.579+0000 7f7c71400700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 0x7f7c5803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.584+0000 7f7c6aa00700 1 -- 172.21.15.116:0/3930653336 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7c64016260 con 0x7f7c74107900 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.780+0000 7f7c71400700 1 -- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 msgr2=0x7f7c5803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1181502267 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.780+0000 7f7c71400700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 0x7f7c5803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.850+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7c7402d9b0 con 0x7f7c74107900 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.850+0000 7f7c6aa00700 1 -- 172.21.15.116:0/3930653336 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7c6401ad80 con 0x7f7c74107900 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 msgr2=0x7f7c5803b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 0x7f7c5803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 msgr2=0x7f7c7419bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c7419bbf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c64000c00 tx=0x7f7c64011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 shutdown_connections 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f7c58038ba0 0x7f7c5803b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:16.603 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 --2- 172.21.15.116:0/3930653336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c74107900 0x7f7c7419bbf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:16.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 >> 172.21.15.116:0/3930653336 conn(0x7f7c74102f60 msgr2=0x7f7c74103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:16.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 shutdown_connections 2024-03-04T07:29:16.604 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:15.853+0000 7f7c78a00700 1 -- 172.21.15.116:0/3930653336 wait complete. 2024-03-04T07:29:16.604 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-04T07:29:16.604 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 5... 2024-03-04T07:29:16.747 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:16.485+0000 7f0856a9e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T07:29:17.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:16.830+0000 7f0856a9e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T07:29:17.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:16.977+0000 7f0856a9e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T07:29:17.386 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:17 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:17.092+0000 7f0856a9e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T07:29:17.386 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:17 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:17.218+0000 7f0856a9e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T07:29:17.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:17 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:17.456+0000 7f0856a9e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T07:29:18.864 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:18.696+0000 7f0856a9e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T07:29:18.864 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:18.862+0000 7f0856a9e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T07:29:19.634 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:19.268+0000 7f0856a9e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T07:29:19.634 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:19.383+0000 7f0856a9e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T07:29:19.634 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:19.633+0000 7f0856a9e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:29:20.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:19.869+0000 7f0856a9e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T07:29:20.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:20.233+0000 7f0856a9e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:29:20.841 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:20.544+0000 7f0856a9e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T07:29:20.841 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:20.839+0000 7f0856a9e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T07:29:22.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:21 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:21.829+0000 7f0856a9e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T07:29:22.908 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:22.507+0000 7f0856a9e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T07:29:22.908 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:22.634+0000 7f0856a9e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:22.907+0000 7f0856a9e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: Active manager daemon x restarted 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: Activating manager daemon x 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: osdmap e2: 0 total, 0 up, 0 in 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: mgrmap e6: x(active, starting, since 0.00378033s) 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T07:29:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T07:29:23.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T07:29:23.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: Manager daemon x is now available 2024-03-04T07:29:23.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:23.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:22 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: Found migration_current of "None". Setting to last migration. 2024-03-04T07:29:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T07:29:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T07:29:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:24.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:23 smithi116 ceph-mon[32150]: mgrmap e7: x(active, since 1.01438s) 2024-03-04T07:29:24.683 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:24.683 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-04T07:29:24.683 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T07:29:24.683 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:24.683 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.176+0000 7f234e200700 1 Processor -- start 2024-03-04T07:29:24.683 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.176+0000 7f234e200700 1 -- start start 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.176+0000 7f234e200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234810a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.176+0000 7f234e200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234807add0 con 0x7f2348108440 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.177+0000 7f2347400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234810a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.177+0000 7f2347400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234810a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42904/0 (socket says 172.21.15.116:42904) 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.177+0000 7f2347400700 1 -- 172.21.15.116:0/3378717654 learned_addr learned my addr 172.21.15.116:0/3378717654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.177+0000 7f2347400700 1 -- 172.21.15.116:0/3378717654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2348106230 con 0x7f2348108440 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.178+0000 7f2347400700 1 --2- 172.21.15.116:0/3378717654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234810a830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f23380098a0 tx=0x7f2338009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f3d8a54e59b3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.178+0000 7f2346000700 1 -- 172.21.15.116:0/3378717654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2338004160 con 0x7f2348108440 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.178+0000 7f2346000700 1 -- 172.21.15.116:0/3378717654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2338004300 con 0x7f2348108440 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.178+0000 7f2346000700 1 -- 172.21.15.116:0/3378717654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2338011b50 con 0x7f2348108440 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.179+0000 7f234e200700 1 -- 172.21.15.116:0/3378717654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 msgr2=0x7f234810a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:24.684 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.179+0000 7f234e200700 1 --2- 172.21.15.116:0/3378717654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234810a830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f23380098a0 tx=0x7f2338009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.179+0000 7f234e200700 1 -- 172.21.15.116:0/3378717654 shutdown_connections 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.179+0000 7f234e200700 1 --2- 172.21.15.116:0/3378717654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234810a830 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.179+0000 7f234e200700 1 -- 172.21.15.116:0/3378717654 >> 172.21.15.116:0/3378717654 conn(0x7f2348101e70 msgr2=0x7f2348104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.179+0000 7f234e200700 1 -- 172.21.15.116:0/3378717654 shutdown_connections 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.179+0000 7f234e200700 1 -- 172.21.15.116:0/3378717654 wait complete. 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.180+0000 7f234e200700 1 Processor -- start 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.181+0000 7f234e200700 1 -- start start 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.181+0000 7f234e200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.181+0000 7f234e200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2348107310 con 0x7f2348108440 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.181+0000 7f2347400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.181+0000 7f2347400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234819dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42908/0 (socket says 172.21.15.116:42908) 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.181+0000 7f2347400700 1 -- 172.21.15.116:0/2273249089 learned_addr learned my addr 172.21.15.116:0/2273249089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.182+0000 7f2347400700 1 -- 172.21.15.116:0/2273249089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234819e320 con 0x7f2348108440 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.182+0000 7f2347400700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234819dde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2338011b60 tx=0x7f2338011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:24.685 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.182+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2338003970 con 0x7f2348108440 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.182+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2338003b10 con 0x7f2348108440 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.182+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2338024610 con 0x7f2348108440 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.182+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234819e5b0 con 0x7f2348108440 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.183+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23481a12a0 con 0x7f2348108440 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.184+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f233801a9f0 con 0x7f2348108440 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.184+0000 7f233fe00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.184+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 --> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f232c03b7c0 con 0x7f232c038bf0 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.184+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2338011e00 con 0x7f2348108440 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.184+0000 7f2346a00700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 msgr2=0x7f232c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1181502267 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.184+0000 7f2346a00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.385+0000 7f2346a00700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 msgr2=0x7f232c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1181502267 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.385+0000 7f2346a00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T07:29:24.686 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.786+0000 7f2346a00700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 msgr2=0x7f232c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1181502267 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:17.786+0000 7f2346a00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:18.587+0000 7f2346a00700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 msgr2=0x7f232c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1181502267 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:18.587+0000 7f2346a00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:20.189+0000 7f2346a00700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 msgr2=0x7f232c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1181502267 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:20.189+0000 7f2346a00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:22.914+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 6) v1 ==== 45695+0+0 (secure 0 0 0) 0x7f2338024c40 con 0x7f2348108440 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:22.915+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 msgr2=0x7f232c03b0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:22.915+0000 7f233fe00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.918+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 7) v1 ==== 45822+0+0 (secure 0 0 0) 0x7f233804eab0 con 0x7f2348108440 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.919+0000 7f233fe00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f232c03c520 0x7f232c03e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.919+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2338012ea0 con 0x7f232c03c520 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.921+0000 7f2346a00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f232c03c520 0x7f232c03e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.924+0000 7f2346a00700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f232c03c520 0x7f232c03e910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2334003a10 tx=0x7f23340092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.927+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f2338012ea0 con 0x7f232c03c520 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.934+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2348069160 con 0x7f232c03c520 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.935+0000 7f233fe00700 1 -- 172.21.15.116:0/2273249089 <== mgr.14120 v2:172.21.15.116:6800/3218338096 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2348069160 con 0x7f232c03c520 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.935+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f232c03c520 msgr2=0x7f232c03e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.935+0000 7f234e200700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f232c03c520 0x7f232c03e910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2334003a10 tx=0x7f23340092b0 comp rx=0 tx=0).stop 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.935+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 msgr2=0x7f234819dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.935+0000 7f234e200700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234819dde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2338011b60 tx=0x7f2338011c40 comp rx=0 tx=0).stop 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.936+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 shutdown_connections 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.936+0000 7f234e200700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f232c03c520 0x7f232c03e910 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.936+0000 7f234e200700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2348108440 0x7f234819dde0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.936+0000 7f234e200700 1 --2- 172.21.15.116:0/2273249089 >> [v2:172.21.15.116:6800/1181502267,v1:172.21.15.116:6801/1181502267] conn(0x7f232c038bf0 0x7f232c03b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.936+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 >> 172.21.15.116:0/2273249089 conn(0x7f2348101e70 msgr2=0x7f2348102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.936+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 shutdown_connections 2024-03-04T07:29:24.688 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:23.936+0000 7f234e200700 1 -- 172.21.15.116:0/2273249089 wait complete. 2024-03-04T07:29:24.689 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 5 is available 2024-03-04T07:29:24.689 INFO:teuthology.orchestra.run.smithi116.stdout:Setting orchestrator backend to cephadm... 2024-03-04T07:29:25.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:24 smithi116 ceph-mon[32150]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T07:29:25.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:24 smithi116 ceph-mon[32150]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T07:29:25.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:24 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:24] ENGINE Bus STARTING 2024-03-04T07:29:25.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:24 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:24] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T07:29:25.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:24 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:26.215 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:25 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:24] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T07:29:26.216 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:25 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:24] ENGINE Bus STARTED 2024-03-04T07:29:26.216 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:25 smithi116 ceph-mon[32150]: mgrmap e8: x(active, since 2s) 2024-03-04T07:29:26.216 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:25 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:26.216 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:25 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:26.237 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.309+0000 7f9aba000700 1 Processor -- start 2024-03-04T07:29:26.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.309+0000 7f9aba000700 1 -- start start 2024-03-04T07:29:26.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.310+0000 7f9aba000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab407b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:26.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.310+0000 7f9aba000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab407b8f0 con 0x7f9ab407cec0 2024-03-04T07:29:26.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.310+0000 7f9ab3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab407b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:26.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.310+0000 7f9ab3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab407b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42998/0 (socket says 172.21.15.116:42998) 2024-03-04T07:29:26.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.310+0000 7f9ab3400700 1 -- 172.21.15.116:0/1384326469 learned_addr learned my addr 172.21.15.116:0/1384326469 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.311+0000 7f9ab3400700 1 -- 172.21.15.116:0/1384326469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab407ba70 con 0x7f9ab407cec0 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.311+0000 7f9ab3400700 1 --2- 172.21.15.116:0/1384326469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab407b320 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009e80 tx=0x7f9aa400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e905e37d5f29740 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.311+0000 7f9ab2a00700 1 -- 172.21.15.116:0/1384326469 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aa4017070 con 0x7f9ab407cec0 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9ab2a00700 1 -- 172.21.15.116:0/1384326469 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9aa400b8c0 con 0x7f9ab407cec0 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9ab2a00700 1 -- 172.21.15.116:0/1384326469 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aa40045c0 con 0x7f9ab407cec0 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9aba000700 1 -- 172.21.15.116:0/1384326469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 msgr2=0x7f9ab407b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9aba000700 1 --2- 172.21.15.116:0/1384326469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab407b320 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009e80 tx=0x7f9aa400b330 comp rx=0 tx=0).stop 2024-03-04T07:29:26.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9aba000700 1 -- 172.21.15.116:0/1384326469 shutdown_connections 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9aba000700 1 --2- 172.21.15.116:0/1384326469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab407b320 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9aba000700 1 -- 172.21.15.116:0/1384326469 >> 172.21.15.116:0/1384326469 conn(0x7f9ab4102f60 msgr2=0x7f9ab4105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.312+0000 7f9aba000700 1 -- 172.21.15.116:0/1384326469 shutdown_connections 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.313+0000 7f9aba000700 1 -- 172.21.15.116:0/1384326469 wait complete. 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.313+0000 7f9aba000700 1 Processor -- start 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.314+0000 7f9aba000700 1 -- start start 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.314+0000 7f9aba000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:26.244 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.314+0000 7f9aba000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab410def0 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.314+0000 7f9ab3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.314+0000 7f9ab3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab419db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43000/0 (socket says 172.21.15.116:43000) 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.314+0000 7f9ab3400700 1 -- 172.21.15.116:0/2883255413 learned_addr learned my addr 172.21.15.116:0/2883255413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.315+0000 7f9ab3400700 1 -- 172.21.15.116:0/2883255413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab419e0d0 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.315+0000 7f9ab3400700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab419db90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009e50 tx=0x7f9aa400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.315+0000 7f9ab0c00700 1 -- 172.21.15.116:0/2883255413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aa4020020 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.315+0000 7f9ab0c00700 1 -- 172.21.15.116:0/2883255413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9aa4003b50 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.315+0000 7f9ab0c00700 1 -- 172.21.15.116:0/2883255413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aa4017040 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.315+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab419e360 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.315+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab419ee80 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.317+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab40636f0 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.317+0000 7f9ab0c00700 1 -- 172.21.15.116:0/2883255413 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f9aa4023020 con 0x7f9ab407cec0 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.317+0000 7f9ab0c00700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f9a98038ba0 0x7f9a9803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.317+0000 7f9ab0c00700 1 -- 172.21.15.116:0/2883255413 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9aa400f050 con 0x7f9ab407cec0 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.318+0000 7f9aaaa00700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f9a98038ba0 0x7f9a9803b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.318+0000 7f9aaaa00700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f9a98038ba0 0x7f9a9803b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.323+0000 7f9ab0c00700 1 -- 172.21.15.116:0/2883255413 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9aa402a020 con 0x7f9ab407cec0 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.556+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ab4104180 con 0x7f9a98038ba0 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.566+0000 7f9ab0c00700 1 -- 172.21.15.116:0/2883255413 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ab4104180 con 0x7f9a98038ba0 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f9a98038ba0 msgr2=0x7f9a9803b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f9a98038ba0 0x7f9a9803b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 msgr2=0x7f9ab419db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab419db90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009e50 tx=0x7f9aa400bd30 comp rx=0 tx=0).stop 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 shutdown_connections 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f9a98038ba0 0x7f9a9803b060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 --2- 172.21.15.116:0/2883255413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ab407cec0 0x7f9ab419db90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 >> 172.21.15.116:0/2883255413 conn(0x7f9ab4102f60 msgr2=0x7f9ab4103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.569+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 shutdown_connections 2024-03-04T07:29:26.247 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:25.570+0000 7f9aba000700 1 -- 172.21.15.116:0/2883255413 wait complete. 2024-03-04T07:29:27.086 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:26 smithi116 ceph-mon[32150]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:29:27.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-04T07:29:27.884 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.828+0000 7ff77d200700 1 Processor -- start 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.828+0000 7ff77d200700 1 -- start start 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.829+0000 7ff77d200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff778109f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.829+0000 7ff77d200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77810a530 con 0x7ff778109b40 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.829+0000 7ff775e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff778109f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.829+0000 7ff775e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff778109f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40170/0 (socket says 172.21.15.116:40170) 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.829+0000 7ff775e00700 1 -- 172.21.15.116:0/3418415366 learned_addr learned my addr 172.21.15.116:0/3418415366 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.830+0000 7ff775e00700 1 -- 172.21.15.116:0/3418415366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77810acf0 con 0x7ff778109b40 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.830+0000 7ff775e00700 1 --2- 172.21.15.116:0/3418415366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff778109f60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7680098a0 tx=0x7ff768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581b0f77d0be41a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.830+0000 7ff774a00700 1 -- 172.21.15.116:0/3418415366 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff768004160 con 0x7ff778109b40 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.830+0000 7ff774a00700 1 -- 172.21.15.116:0/3418415366 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff768004300 con 0x7ff778109b40 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.830+0000 7ff774a00700 1 -- 172.21.15.116:0/3418415366 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff768011b50 con 0x7ff778109b40 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.831+0000 7ff77d200700 1 -- 172.21.15.116:0/3418415366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 msgr2=0x7ff778109f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:27.885 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.831+0000 7ff77d200700 1 --2- 172.21.15.116:0/3418415366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff778109f60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7680098a0 tx=0x7ff768009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.831+0000 7ff77d200700 1 -- 172.21.15.116:0/3418415366 shutdown_connections 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.831+0000 7ff77d200700 1 --2- 172.21.15.116:0/3418415366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff778109f60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.831+0000 7ff77d200700 1 -- 172.21.15.116:0/3418415366 >> 172.21.15.116:0/3418415366 conn(0x7ff7781054c0 msgr2=0x7ff7781078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.831+0000 7ff77d200700 1 -- 172.21.15.116:0/3418415366 shutdown_connections 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.831+0000 7ff77d200700 1 -- 172.21.15.116:0/3418415366 wait complete. 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.832+0000 7ff77d200700 1 Processor -- start 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.832+0000 7ff77d200700 1 -- start start 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.833+0000 7ff77d200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff77819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.833+0000 7ff77d200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77810e0a0 con 0x7ff778109b40 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.833+0000 7ff775e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff77819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.833+0000 7ff775e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff77819de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40176/0 (socket says 172.21.15.116:40176) 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.833+0000 7ff775e00700 1 -- 172.21.15.116:0/3246290460 learned_addr learned my addr 172.21.15.116:0/3246290460 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.833+0000 7ff775e00700 1 -- 172.21.15.116:0/3246290460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77819e390 con 0x7ff778109b40 2024-03-04T07:29:27.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.833+0000 7ff775e00700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff77819de50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff768011850 tx=0x7ff768011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.834+0000 7ff76ea00700 1 -- 172.21.15.116:0/3246290460 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff768003af0 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.834+0000 7ff76ea00700 1 -- 172.21.15.116:0/3246290460 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff768020070 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.834+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77819e620 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.834+0000 7ff76ea00700 1 -- 172.21.15.116:0/3246290460 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7680195a0 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.834+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7781a1310 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.835+0000 7ff76ea00700 1 -- 172.21.15.116:0/3246290460 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7ff768019850 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.835+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff778069160 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.835+0000 7ff76ea00700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff75c038c50 0x7ff75c03b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.835+0000 7ff76ea00700 1 -- 172.21.15.116:0/3246290460 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff768011e00 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.836+0000 7ff775400700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff75c038c50 0x7ff75c03b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.836+0000 7ff775400700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff75c038c50 0x7ff75c03b110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff764006fd0 tx=0x7ff764006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:26.842+0000 7ff76ea00700 1 -- 172.21.15.116:0/3246290460 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff768019b40 con 0x7ff778109b40 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.066+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff77802da20 con 0x7ff75c038c50 2024-03-04T07:29:27.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.067+0000 7ff76ea00700 1 -- 172.21.15.116:0/3246290460 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff77802da20 con 0x7ff75c038c50 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff75c038c50 msgr2=0x7ff75c03b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff75c038c50 0x7ff75c03b110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff764006fd0 tx=0x7ff764006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 msgr2=0x7ff77819de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff77819de50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff768011850 tx=0x7ff768011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 shutdown_connections 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff75c038c50 0x7ff75c03b110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 --2- 172.21.15.116:0/3246290460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff778109b40 0x7ff77819de50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 >> 172.21.15.116:0/3246290460 conn(0x7ff7781054c0 msgr2=0x7ff778107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 shutdown_connections 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:27.070+0000 7ff77d200700 1 -- 172.21.15.116:0/3246290460 wait complete. 2024-03-04T07:29:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:Generating ssh key... 2024-03-04T07:29:28.007 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:27 smithi116 ceph-mon[32150]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:29:29.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: Generating public/private rsa key pair. 2024-03-04T07:29:29.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: Your identification has been saved in /tmp/tmpojs0yoef/key. 2024-03-04T07:29:29.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: Your public key has been saved in /tmp/tmpojs0yoef/key.pub. 2024-03-04T07:29:29.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: The key fingerprint is: 2024-03-04T07:29:29.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: SHA256:i25HoRhvXFbMyY3aQJ/nEJNauEmzrWE2GFA6uE5dOwQ ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:29:29.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: The key's randomart image is: 2024-03-04T07:29:29.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: +---[RSA 3072]----+ 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | Eo...=o= | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | . o. =.X+. | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | . o o+ &+ . | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | o.+..# o+ | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | o .=o*S= . | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | o . =oo. | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | . .... | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | .. . | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: | ... | 2024-03-04T07:29:29.510 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: +----[SHA256]-----+ 2024-03-04T07:29:29.880 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.504+0000 7f18b6400700 1 Processor -- start 2024-03-04T07:29:29.881 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.504+0000 7f18b6400700 1 -- start start 2024-03-04T07:29:29.881 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.505+0000 7f18b6400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:29.881 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.505+0000 7f18b6400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b007b800 con 0x7f18b007ce10 2024-03-04T07:29:29.881 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.505+0000 7f18af400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:29.881 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.505+0000 7f18af400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b007b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40190/0 (socket says 172.21.15.116:40190) 2024-03-04T07:29:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.505+0000 7f18af400700 1 -- 172.21.15.116:0/1199576096 learned_addr learned my addr 172.21.15.116:0/1199576096 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.506+0000 7f18af400700 1 -- 172.21.15.116:0/1199576096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b007b980 con 0x7f18b007ce10 2024-03-04T07:29:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.506+0000 7f18af400700 1 --2- 172.21.15.116:0/1199576096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b007b2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18a00098a0 tx=0x7f18a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802d4cec3b774f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.506+0000 7f18ae000700 1 -- 172.21.15.116:0/1199576096 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18a0004160 con 0x7f18b007ce10 2024-03-04T07:29:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.506+0000 7f18ae000700 1 -- 172.21.15.116:0/1199576096 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a0004300 con 0x7f18b007ce10 2024-03-04T07:29:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.506+0000 7f18ae000700 1 -- 172.21.15.116:0/1199576096 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18a0011b50 con 0x7f18b007ce10 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.507+0000 7f18b6400700 1 -- 172.21.15.116:0/1199576096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 msgr2=0x7f18b007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.507+0000 7f18b6400700 1 --2- 172.21.15.116:0/1199576096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b007b2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18a00098a0 tx=0x7f18a0009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.507+0000 7f18b6400700 1 -- 172.21.15.116:0/1199576096 shutdown_connections 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.507+0000 7f18b6400700 1 --2- 172.21.15.116:0/1199576096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b007b2c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.507+0000 7f18b6400700 1 -- 172.21.15.116:0/1199576096 >> 172.21.15.116:0/1199576096 conn(0x7f18b0102f60 msgr2=0x7f18b0105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.507+0000 7f18b6400700 1 -- 172.21.15.116:0/1199576096 shutdown_connections 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.507+0000 7f18b6400700 1 -- 172.21.15.116:0/1199576096 wait complete. 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.508+0000 7f18b6400700 1 Processor -- start 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.508+0000 7f18b6400700 1 -- start start 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.509+0000 7f18b6400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.509+0000 7f18b6400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b010def0 con 0x7f18b007ce10 2024-03-04T07:29:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.509+0000 7f18af400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.509+0000 7f18af400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b019dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40196/0 (socket says 172.21.15.116:40196) 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.509+0000 7f18af400700 1 -- 172.21.15.116:0/2574463225 learned_addr learned my addr 172.21.15.116:0/2574463225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.509+0000 7f18af400700 1 -- 172.21.15.116:0/2574463225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b019e2a0 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.510+0000 7f18af400700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b019dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f18a0011b60 tx=0x7f18a0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.510+0000 7f18a7e00700 1 -- 172.21.15.116:0/2574463225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18a0003b20 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.510+0000 7f18a7e00700 1 -- 172.21.15.116:0/2574463225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a0003cc0 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.510+0000 7f18a7e00700 1 -- 172.21.15.116:0/2574463225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18a0024840 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.510+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b019e530 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.510+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b01a1220 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.511+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b0069160 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.512+0000 7f18a7e00700 1 -- 172.21.15.116:0/2574463225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f18a0022020 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.512+0000 7f18a7e00700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1894038ba0 0x7f189403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.512+0000 7f18a7e00700 1 -- 172.21.15.116:0/2574463225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f18a0011e00 con 0x7f18b007ce10 2024-03-04T07:29:29.888 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.512+0000 7f18aea00700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1894038ba0 0x7f189403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.513+0000 7f18aea00700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1894038ba0 0x7f189403b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.517+0000 7f18a7e00700 1 -- 172.21.15.116:0/2574463225 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f18a004c550 con 0x7f18b007ce10 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:28.737+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b002da20 con 0x7f1894038ba0 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.153+0000 7f18a7e00700 1 -- 172.21.15.116:0/2574463225 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f18b002da20 con 0x7f1894038ba0 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1894038ba0 msgr2=0x7f189403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1894038ba0 0x7f189403b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 msgr2=0x7f18b019dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b019dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f18a0011b60 tx=0x7f18a0011c40 comp rx=0 tx=0).stop 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 shutdown_connections 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1894038ba0 0x7f189403b060 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 --2- 172.21.15.116:0/2574463225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b007ce10 0x7f18b019dd60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 >> 172.21.15.116:0/2574463225 conn(0x7f18b0102f60 msgr2=0x7f18b0103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:29.889 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 shutdown_connections 2024-03-04T07:29:29.890 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:29.157+0000 7f18b6400700 1 -- 172.21.15.116:0/2574463225 wait complete. 2024-03-04T07:29:30.464 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:30 smithi116 ceph-mon[32150]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:29:30.464 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:30 smithi116 ceph-mon[32150]: Generating ssh key... 2024-03-04T07:29:30.464 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:30 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:30.464 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:30 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:31.463 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBNLlSw5nqlO2AanaUItBKtxH16Auf0M5q2uFeeCPgfLIaiNEOlyC5eyeid3NGBeJiFTH6EyTyJYzo3GkdnOC5nBczvZ58qzR1YRLtV36n+cWsA7P0qwVH6BSSrEuTirKg5PiBkyFduaX/Q4ksiD7f7q4VsAjpuZh+hI/Q5aXUwWHo+3eJzeG9Y68GZQhbpWoQEpw5l/oN5lZCcCPI2zRYec2VIOg81ajFk+oXmXX8cipIg+jNP5frLfbHcqHKI9iIZz5riUspAKhSwNbVw/0g14AQ5fEZglwIqoHua7R6WncHFsEZdw+LnYQxcwKIimrD22A4gssmLwIaiNE8hh4bB81646jk+m0HKZZy3DX8vGDR9m8lKsV2JpeDPlAn+7epReMjsaOQCxk8/AuwHo3Hmi8IxqE+qnCoWIpWGgpObL1jTNHpYLqtomMbPdkn5FLqThaJ2acfCVREPT/m/+3KXNyJJ4lCch/WgX4uhsHz8OIFzmYDG/f5LlOw3aRRE0= ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:29:31.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.520+0000 7fbe6be00700 1 Processor -- start 2024-03-04T07:29:31.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.520+0000 7fbe6be00700 1 -- start start 2024-03-04T07:29:31.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.521+0000 7fbe6be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe641065f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:31.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.521+0000 7fbe6be00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6407ae30 con 0x7fbe641061d0 2024-03-04T07:29:31.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.521+0000 7fbe69400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe641065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:31.464 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.521+0000 7fbe69400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe641065f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40204/0 (socket says 172.21.15.116:40204) 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.521+0000 7fbe69400700 1 -- 172.21.15.116:0/2106131955 learned_addr learned my addr 172.21.15.116:0/2106131955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.522+0000 7fbe69400700 1 -- 172.21.15.116:0/2106131955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64106b30 con 0x7fbe641061d0 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.522+0000 7fbe69400700 1 --2- 172.21.15.116:0/2106131955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe641065f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009e80 tx=0x7fbe5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683aed59c87f1b87 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.522+0000 7fbe63e00700 1 -- 172.21.15.116:0/2106131955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe58017070 con 0x7fbe641061d0 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.522+0000 7fbe63e00700 1 -- 172.21.15.116:0/2106131955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe5800b8c0 con 0x7fbe641061d0 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.522+0000 7fbe63e00700 1 -- 172.21.15.116:0/2106131955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe580045c0 con 0x7fbe641061d0 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.523+0000 7fbe6be00700 1 -- 172.21.15.116:0/2106131955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 msgr2=0x7fbe641065f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.523+0000 7fbe6be00700 1 --2- 172.21.15.116:0/2106131955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe641065f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009e80 tx=0x7fbe5800b330 comp rx=0 tx=0).stop 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.523+0000 7fbe6be00700 1 -- 172.21.15.116:0/2106131955 shutdown_connections 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.523+0000 7fbe6be00700 1 --2- 172.21.15.116:0/2106131955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe641065f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.523+0000 7fbe6be00700 1 -- 172.21.15.116:0/2106131955 >> 172.21.15.116:0/2106131955 conn(0x7fbe64101e70 msgr2=0x7fbe64104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.523+0000 7fbe6be00700 1 -- 172.21.15.116:0/2106131955 shutdown_connections 2024-03-04T07:29:31.465 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.523+0000 7fbe6be00700 1 -- 172.21.15.116:0/2106131955 wait complete. 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.524+0000 7fbe6be00700 1 Processor -- start 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.525+0000 7fbe6be00700 1 -- start start 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.525+0000 7fbe6be00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe6419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.525+0000 7fbe6be00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64107bc0 con 0x7fbe641061d0 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.525+0000 7fbe69400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe6419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.525+0000 7fbe69400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe6419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40218/0 (socket says 172.21.15.116:40218) 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.525+0000 7fbe69400700 1 -- 172.21.15.116:0/202373879 learned_addr learned my addr 172.21.15.116:0/202373879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.525+0000 7fbe69400700 1 -- 172.21.15.116:0/202373879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6419e340 con 0x7fbe641061d0 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.526+0000 7fbe69400700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe6419de00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009530 tx=0x7fbe5800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.526+0000 7fbe62000700 1 -- 172.21.15.116:0/202373879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe58020020 con 0x7fbe641061d0 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.526+0000 7fbe62000700 1 -- 172.21.15.116:0/202373879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe58003b50 con 0x7fbe641061d0 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.526+0000 7fbe62000700 1 -- 172.21.15.116:0/202373879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe58017040 con 0x7fbe641061d0 2024-03-04T07:29:31.466 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.526+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6419e5d0 con 0x7fbe641061d0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.526+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe641a1270 con 0x7fbe641061d0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.527+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64069160 con 0x7fbe641061d0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.527+0000 7fbe62000700 1 -- 172.21.15.116:0/202373879 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fbe58023020 con 0x7fbe641061d0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.528+0000 7fbe62000700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fbe4c038bf0 0x7fbe4c03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.528+0000 7fbe62000700 1 -- 172.21.15.116:0/202373879 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbe5800f050 con 0x7fbe641061d0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.528+0000 7fbe68a00700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fbe4c038bf0 0x7fbe4c03b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.529+0000 7fbe68a00700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fbe4c038bf0 0x7fbe4c03b0b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe50006fd0 tx=0x7fbe50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.533+0000 7fbe62000700 1 -- 172.21.15.116:0/202373879 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbe5802a020 con 0x7fbe641061d0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.752+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe6402da20 con 0x7fbe4c038bf0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.753+0000 7fbe62000700 1 -- 172.21.15.116:0/202373879 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbe6402da20 con 0x7fbe4c038bf0 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fbe4c038bf0 msgr2=0x7fbe4c03b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fbe4c038bf0 0x7fbe4c03b0b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe50006fd0 tx=0x7fbe50006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 msgr2=0x7fbe6419de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe6419de00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009530 tx=0x7fbe5800bd50 comp rx=0 tx=0).stop 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 shutdown_connections 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fbe4c038bf0 0x7fbe4c03b0b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 --2- 172.21.15.116:0/202373879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe641061d0 0x7fbe6419de00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.756+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 >> 172.21.15.116:0/202373879 conn(0x7fbe64101e70 msgr2=0x7fbe64194680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.757+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 shutdown_connections 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:30.757+0000 7fbe6be00700 1 -- 172.21.15.116:0/202373879 wait complete. 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-04T07:29:31.468 INFO:teuthology.orchestra.run.smithi116.stdout:Adding key to root@localhost authorized_keys... 2024-03-04T07:29:31.469 INFO:teuthology.orchestra.run.smithi116.stdout:Adding host smithi116... 2024-03-04T07:29:31.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:31 smithi116 ceph-mon[32150]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:29:32.751 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:32 smithi116 ceph-mon[32150]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:29:34.703 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:34 smithi116 ceph-mon[32150]: Deploying cephadm binary to smithi116 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Added host 'smithi116' with addr '172.21.15.116' 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.084+0000 7f120ce00700 1 Processor -- start 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.084+0000 7f120ce00700 1 -- start start 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.085+0000 7f120ce00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120810b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.085+0000 7f120ce00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1208109780 con 0x7f120810b1f0 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.085+0000 7f1205e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120810b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.085+0000 7f1205e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120810b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40232/0 (socket says 172.21.15.116:40232) 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.085+0000 7f1205e00700 1 -- 172.21.15.116:0/2774625857 learned_addr learned my addr 172.21.15.116:0/2774625857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.086+0000 7f1205e00700 1 -- 172.21.15.116:0/2774625857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1208109900 con 0x7f120810b1f0 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.086+0000 7f1205e00700 1 --2- 172.21.15.116:0/2774625857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120810b610 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f11f80098a0 tx=0x7f11f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bf7d34266d1b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.086+0000 7f1204a00700 1 -- 172.21.15.116:0/2774625857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11f8004160 con 0x7f120810b1f0 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.087+0000 7f1204a00700 1 -- 172.21.15.116:0/2774625857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11f8004300 con 0x7f120810b1f0 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.087+0000 7f1204a00700 1 -- 172.21.15.116:0/2774625857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11f8011b50 con 0x7f120810b1f0 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.087+0000 7f120ce00700 1 -- 172.21.15.116:0/2774625857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 msgr2=0x7f120810b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.087+0000 7f120ce00700 1 --2- 172.21.15.116:0/2774625857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120810b610 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f11f80098a0 tx=0x7f11f8009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.087+0000 7f120ce00700 1 -- 172.21.15.116:0/2774625857 shutdown_connections 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.087+0000 7f120ce00700 1 --2- 172.21.15.116:0/2774625857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120810b610 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.087+0000 7f120ce00700 1 -- 172.21.15.116:0/2774625857 >> 172.21.15.116:0/2774625857 conn(0x7f12081054c0 msgr2=0x7f12081078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.088+0000 7f120ce00700 1 -- 172.21.15.116:0/2774625857 shutdown_connections 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.088+0000 7f120ce00700 1 -- 172.21.15.116:0/2774625857 wait complete. 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.089+0000 7f120ce00700 1 Processor -- start 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.089+0000 7f120ce00700 1 -- start start 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.089+0000 7f120ce00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:39.348 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.089+0000 7f120ce00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120810e0c0 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.089+0000 7f1205e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.089+0000 7f1205e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120819dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40234/0 (socket says 172.21.15.116:40234) 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.089+0000 7f1205e00700 1 -- 172.21.15.116:0/2299269251 learned_addr learned my addr 172.21.15.116:0/2299269251 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.090+0000 7f1205e00700 1 -- 172.21.15.116:0/2299269251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120819e520 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.090+0000 7f1205e00700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120819dfe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f11f8011850 tx=0x7f11f8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.090+0000 7f11fea00700 1 -- 172.21.15.116:0/2299269251 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11f8003af0 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.090+0000 7f11fea00700 1 -- 172.21.15.116:0/2299269251 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11f8020070 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.090+0000 7f11fea00700 1 -- 172.21.15.116:0/2299269251 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11f80195a0 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.090+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120819e7b0 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.091+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120819f2d0 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.092+0000 7f11fea00700 1 -- 172.21.15.116:0/2299269251 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f11f801a430 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.092+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12080636f0 con 0x7f120810b1f0 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.092+0000 7f11fea00700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f11ec038bf0 0x7f11ec03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:39.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.092+0000 7f11fea00700 1 -- 172.21.15.116:0/2299269251 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f11f8011e00 con 0x7f120810b1f0 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.092+0000 7f1205400700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f11ec038bf0 0x7f11ec03b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.093+0000 7f1205400700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f11ec038bf0 0x7f11ec03b0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f11f4006fd0 tx=0x7f11f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.098+0000 7f11fea00700 1 -- 172.21.15.116:0/2299269251 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f11f8021270 con 0x7f120810b1f0 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:32.323+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}) v1 -- 0x7f120819f550 con 0x7f11ec038bf0 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.537+0000 7f11fea00700 1 -- 172.21.15.116:0/2299269251 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f120819f550 con 0x7f11ec038bf0 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f11ec038bf0 msgr2=0x7f11ec03b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f11ec038bf0 0x7f11ec03b0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f11f4006fd0 tx=0x7f11f4006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 msgr2=0x7f120819dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120819dfe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f11f8011850 tx=0x7f11f8011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 shutdown_connections 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f11ec038bf0 0x7f11ec03b0b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 --2- 172.21.15.116:0/2299269251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f120810b1f0 0x7f120819dfe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 >> 172.21.15.116:0/2299269251 conn(0x7f12081054c0 msgr2=0x7f1208107240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:39.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 shutdown_connections 2024-03-04T07:29:39.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:38.541+0000 7f120ce00700 1 -- 172.21.15.116:0/2299269251 wait complete. 2024-03-04T07:29:39.351 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mon service... 2024-03-04T07:29:39.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:39 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:39.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:39 smithi116 ceph-mon[32150]: Added host smithi116 2024-03-04T07:29:39.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:39 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:41.550 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-04T07:29:41.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.044+0000 7f6ec5c00700 1 Processor -- start 2024-03-04T07:29:41.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.044+0000 7f6ec5c00700 1 -- start start 2024-03-04T07:29:41.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.045+0000 7f6ec5c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec010f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:41.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.045+0000 7f6ec5c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec010fe70 con 0x7f6ec010f510 2024-03-04T07:29:41.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.045+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec010f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:41.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.045+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec010f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51228/0 (socket says 172.21.15.116:51228) 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.045+0000 7f6ebea00700 1 -- 172.21.15.116:0/356523860 learned_addr learned my addr 172.21.15.116:0/356523860 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.046+0000 7f6ebea00700 1 -- 172.21.15.116:0/356523860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec010dcd0 con 0x7f6ec010f510 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.046+0000 7f6ebea00700 1 --2- 172.21.15.116:0/356523860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec010f930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00098a0 tx=0x7f6eb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a46824d8d4d6c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.046+0000 7f6ebe000700 1 -- 172.21.15.116:0/356523860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eb0004160 con 0x7f6ec010f510 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.046+0000 7f6ebe000700 1 -- 172.21.15.116:0/356523860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6eb0004300 con 0x7f6ec010f510 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.046+0000 7f6ebe000700 1 -- 172.21.15.116:0/356523860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eb0011b50 con 0x7f6ec010f510 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.047+0000 7f6ec5c00700 1 -- 172.21.15.116:0/356523860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 msgr2=0x7f6ec010f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.047+0000 7f6ec5c00700 1 --2- 172.21.15.116:0/356523860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec010f930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00098a0 tx=0x7f6eb0009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.047+0000 7f6ec5c00700 1 -- 172.21.15.116:0/356523860 shutdown_connections 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.047+0000 7f6ec5c00700 1 --2- 172.21.15.116:0/356523860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec010f930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.047+0000 7f6ec5c00700 1 -- 172.21.15.116:0/356523860 >> 172.21.15.116:0/356523860 conn(0x7f6ec007d0b0 msgr2=0x7f6ec007f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:41.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.047+0000 7f6ec5c00700 1 -- 172.21.15.116:0/356523860 shutdown_connections 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.047+0000 7f6ec5c00700 1 -- 172.21.15.116:0/356523860 wait complete. 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.048+0000 7f6ec5c00700 1 Processor -- start 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.049+0000 7f6ec5c00700 1 -- start start 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.049+0000 7f6ec5c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec01a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.049+0000 7f6ec5c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01124c0 con 0x7f6ec010f510 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.049+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec01a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.049+0000 7f6ebea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec01a23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51232/0 (socket says 172.21.15.116:51232) 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.049+0000 7f6ebea00700 1 -- 172.21.15.116:0/3513391932 learned_addr learned my addr 172.21.15.116:0/3513391932 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.049+0000 7f6ebea00700 1 -- 172.21.15.116:0/3513391932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec01a2930 con 0x7f6ec010f510 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.050+0000 7f6ebea00700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec01a23f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0011850 tx=0x7f6eb0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.050+0000 7f6eb7400700 1 -- 172.21.15.116:0/3513391932 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eb0003af0 con 0x7f6ec010f510 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.050+0000 7f6eb7400700 1 -- 172.21.15.116:0/3513391932 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6eb0020070 con 0x7f6ec010f510 2024-03-04T07:29:41.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.050+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec01a2bc0 con 0x7f6ec010f510 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.050+0000 7f6eb7400700 1 -- 172.21.15.116:0/3513391932 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eb00195a0 con 0x7f6ec010f510 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.050+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec01a36e0 con 0x7f6ec010f510 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.051+0000 7f6eb7400700 1 -- 172.21.15.116:0/3513391932 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f6eb0019850 con 0x7f6ec010f510 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.051+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec0112640 con 0x7f6ec010f510 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.052+0000 7f6eb7400700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f6ea4038bf0 0x7f6ea403b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.052+0000 7f6eb7400700 1 -- 172.21.15.116:0/3513391932 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6eb0011e00 con 0x7f6ec010f510 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.052+0000 7f6eb7e00700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f6ea4038bf0 0x7f6ea403b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.053+0000 7f6eb7e00700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f6ea4038bf0 0x7f6ea403b0b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8006fd0 tx=0x7f6ea8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.057+0000 7f6eb7400700 1 -- 172.21.15.116:0/3513391932 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6eb0019b40 con 0x7f6ec010f510 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.312+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6ec0075b20 con 0x7f6ea4038bf0 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.321+0000 7f6eb7400700 1 -- 172.21.15.116:0/3513391932 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6ec0075b20 con 0x7f6ea4038bf0 2024-03-04T07:29:41.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.323+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f6ea4038bf0 msgr2=0x7f6ea403b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.323+0000 7f6ec5c00700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f6ea4038bf0 0x7f6ea403b0b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8006fd0 tx=0x7f6ea8006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.323+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 msgr2=0x7f6ec01a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.323+0000 7f6ec5c00700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec01a23f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0011850 tx=0x7f6eb0011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.323+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 shutdown_connections 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.323+0000 7f6ec5c00700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f6ea4038bf0 0x7f6ea403b0b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.324+0000 7f6ec5c00700 1 --2- 172.21.15.116:0/3513391932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ec010f510 0x7f6ec01a23f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.324+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 >> 172.21.15.116:0/3513391932 conn(0x7f6ec007d0b0 msgr2=0x7f6ec007de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.324+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 shutdown_connections 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:40.324+0000 7f6ec5c00700 1 -- 172.21.15.116:0/3513391932 wait complete. 2024-03-04T07:29:41.560 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mgr service... 2024-03-04T07:29:41.652 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:41 smithi116 ceph-mon[32150]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:29:41.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:41 smithi116 ceph-mon[32150]: Saving service mon spec with placement count:5 2024-03-04T07:29:41.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:41 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:43.172 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:43 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:43.172 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:43 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:43.172 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:43 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.561+0000 7f75f4200700 1 Processor -- start 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.561+0000 7f75f4200700 1 -- start start 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.562+0000 7f75f4200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec07b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.562+0000 7f75f4200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec07b8d0 con 0x7f75ec07cea0 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.562+0000 7f75f1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec07b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.562+0000 7f75f1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec07b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51234/0 (socket says 172.21.15.116:51234) 2024-03-04T07:29:43.490 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.562+0000 7f75f1800700 1 -- 172.21.15.116:0/3585192102 learned_addr learned my addr 172.21.15.116:0/3585192102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.563+0000 7f75f1800700 1 -- 172.21.15.116:0/3585192102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec07ba50 con 0x7f75ec07cea0 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.563+0000 7f75f1800700 1 --2- 172.21.15.116:0/3585192102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec07b300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75e00098a0 tx=0x7f75e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea973aac32c0bc7c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.563+0000 7f75ebe00700 1 -- 172.21.15.116:0/3585192102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75e0004160 con 0x7f75ec07cea0 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.563+0000 7f75ebe00700 1 -- 172.21.15.116:0/3585192102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e0004300 con 0x7f75ec07cea0 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.563+0000 7f75ebe00700 1 -- 172.21.15.116:0/3585192102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75e0011b20 con 0x7f75ec07cea0 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.564+0000 7f75f4200700 1 -- 172.21.15.116:0/3585192102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 msgr2=0x7f75ec07b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.564+0000 7f75f4200700 1 --2- 172.21.15.116:0/3585192102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec07b300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75e00098a0 tx=0x7f75e0009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.564+0000 7f75f4200700 1 -- 172.21.15.116:0/3585192102 shutdown_connections 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.564+0000 7f75f4200700 1 --2- 172.21.15.116:0/3585192102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec07b300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.564+0000 7f75f4200700 1 -- 172.21.15.116:0/3585192102 >> 172.21.15.116:0/3585192102 conn(0x7f75ec102f60 msgr2=0x7f75ec105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.564+0000 7f75f4200700 1 -- 172.21.15.116:0/3585192102 shutdown_connections 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.564+0000 7f75f4200700 1 -- 172.21.15.116:0/3585192102 wait complete. 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.565+0000 7f75f4200700 1 Processor -- start 2024-03-04T07:29:43.491 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.566+0000 7f75f4200700 1 -- start start 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.566+0000 7f75f4200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec19db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.566+0000 7f75f4200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec10df60 con 0x7f75ec07cea0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.566+0000 7f75f1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec19db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.566+0000 7f75f1800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec19db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51248/0 (socket says 172.21.15.116:51248) 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.566+0000 7f75f1800700 1 -- 172.21.15.116:0/2003521906 learned_addr learned my addr 172.21.15.116:0/2003521906 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.567+0000 7f75f1800700 1 -- 172.21.15.116:0/2003521906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec19e0d0 con 0x7f75ec07cea0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.567+0000 7f75f1800700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec19db90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75e0000c00 tx=0x7f75e0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.567+0000 7f75ea000700 1 -- 172.21.15.116:0/2003521906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75e0003b60 con 0x7f75ec07cea0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.567+0000 7f75ea000700 1 -- 172.21.15.116:0/2003521906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e001a910 con 0x7f75ec07cea0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.567+0000 7f75ea000700 1 -- 172.21.15.116:0/2003521906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75e0024930 con 0x7f75ec07cea0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.567+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ec19e360 con 0x7f75ec07cea0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.567+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec19ee80 con 0x7f75ec07cea0 2024-03-04T07:29:43.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.569+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ec0636f0 con 0x7f75ec07cea0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.569+0000 7f75ea000700 1 -- 172.21.15.116:0/2003521906 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f75e0022020 con 0x7f75ec07cea0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.569+0000 7f75ea000700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f75d4038bf0 0x7f75d403b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.569+0000 7f75ea000700 1 -- 172.21.15.116:0/2003521906 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f75e004cc30 con 0x7f75ec07cea0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.570+0000 7f75f0e00700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f75d4038bf0 0x7f75d403b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.570+0000 7f75f0e00700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f75d4038bf0 0x7f75d403b0b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f75d8006fd0 tx=0x7f75d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.575+0000 7f75ea000700 1 -- 172.21.15.116:0/2003521906 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f75e002ddb0 con 0x7f75ec07cea0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.809+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f75ec1041c0 con 0x7f75d4038bf0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.816+0000 7f75ea000700 1 -- 172.21.15.116:0/2003521906 <== mgr.14120 v2:172.21.15.116:6800/3218338096 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f75ec1041c0 con 0x7f75d4038bf0 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.819+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f75d4038bf0 msgr2=0x7f75d403b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.819+0000 7f75f4200700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f75d4038bf0 0x7f75d403b0b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f75d8006fd0 tx=0x7f75d8006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.819+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 msgr2=0x7f75ec19db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.819+0000 7f75f4200700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec19db90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75e0000c00 tx=0x7f75e0011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.820+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 shutdown_connections 2024-03-04T07:29:43.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.820+0000 7f75f4200700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f75d4038bf0 0x7f75d403b0b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:43.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.820+0000 7f75f4200700 1 --2- 172.21.15.116:0/2003521906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ec07cea0 0x7f75ec19db90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:43.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.820+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 >> 172.21.15.116:0/2003521906 conn(0x7f75ec102f60 msgr2=0x7f75ec103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:43.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.820+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 shutdown_connections 2024-03-04T07:29:43.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:42.820+0000 7f75f4200700 1 -- 172.21.15.116:0/2003521906 wait complete. 2024-03-04T07:29:44.357 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:44 smithi116 ceph-mon[32150]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:29:44.357 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:44 smithi116 ceph-mon[32150]: Saving service mgr spec with placement count:2 2024-03-04T07:29:44.357 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:44 smithi116 ceph-mon[32150]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:29:45.231 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.088+0000 7f8122400700 1 Processor -- start 2024-03-04T07:29:45.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.088+0000 7f8122400700 1 -- start start 2024-03-04T07:29:45.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.088+0000 7f8122400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c10a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:45.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.088+0000 7f8122400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c07aeb0 con 0x7f811c108360 2024-03-04T07:29:45.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.089+0000 7f811b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c10a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:45.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.089+0000 7f811b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c10a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51260/0 (socket says 172.21.15.116:51260) 2024-03-04T07:29:45.238 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.089+0000 7f811b400700 1 -- 172.21.15.116:0/2233759828 learned_addr learned my addr 172.21.15.116:0/2233759828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.089+0000 7f811b400700 1 -- 172.21.15.116:0/2233759828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c10ac90 con 0x7f811c108360 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.090+0000 7f811b400700 1 --2- 172.21.15.116:0/2233759828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c10a750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098a0 tx=0x7f810c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962a5f3316f9c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.090+0000 7f811a000700 1 -- 172.21.15.116:0/2233759828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c004160 con 0x7f811c108360 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.090+0000 7f811a000700 1 -- 172.21.15.116:0/2233759828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f810c004300 con 0x7f811c108360 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.090+0000 7f811a000700 1 -- 172.21.15.116:0/2233759828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c011b20 con 0x7f811c108360 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.091+0000 7f8122400700 1 -- 172.21.15.116:0/2233759828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 msgr2=0x7f811c10a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.091+0000 7f8122400700 1 --2- 172.21.15.116:0/2233759828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c10a750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098a0 tx=0x7f810c009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.091+0000 7f8122400700 1 -- 172.21.15.116:0/2233759828 shutdown_connections 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.091+0000 7f8122400700 1 --2- 172.21.15.116:0/2233759828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c10a750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.091+0000 7f8122400700 1 -- 172.21.15.116:0/2233759828 >> 172.21.15.116:0/2233759828 conn(0x7f811c101e70 msgr2=0x7f811c104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.091+0000 7f8122400700 1 -- 172.21.15.116:0/2233759828 shutdown_connections 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.091+0000 7f8122400700 1 -- 172.21.15.116:0/2233759828 wait complete. 2024-03-04T07:29:45.239 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.093+0000 7f8122400700 1 Processor -- start 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.093+0000 7f8122400700 1 -- start start 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.093+0000 7f8122400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c19bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.093+0000 7f8122400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c10bd00 con 0x7f811c108360 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.093+0000 7f811b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c19bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.093+0000 7f811b400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c19bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51272/0 (socket says 172.21.15.116:51272) 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.093+0000 7f811b400700 1 -- 172.21.15.116:0/2690850167 learned_addr learned my addr 172.21.15.116:0/2690850167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.094+0000 7f811b400700 1 -- 172.21.15.116:0/2690850167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c19c0b0 con 0x7f811c108360 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.094+0000 7f811b400700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c19bb70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f810c011b60 tx=0x7f810c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.094+0000 7f8113e00700 1 -- 172.21.15.116:0/2690850167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c003b60 con 0x7f811c108360 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.094+0000 7f8113e00700 1 -- 172.21.15.116:0/2690850167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f810c01a910 con 0x7f811c108360 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.094+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811c19c340 con 0x7f811c108360 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.095+0000 7f8113e00700 1 -- 172.21.15.116:0/2690850167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c024930 con 0x7f811c108360 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.095+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811c19c7a0 con 0x7f811c108360 2024-03-04T07:29:45.240 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.096+0000 7f8113e00700 1 -- 172.21.15.116:0/2690850167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f810c022020 con 0x7f811c108360 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.096+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811c069160 con 0x7f811c108360 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.096+0000 7f8113e00700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f8100038ba0 0x7f810003b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.096+0000 7f8113e00700 1 -- 172.21.15.116:0/2690850167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f810c04ccb0 con 0x7f811c108360 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.097+0000 7f811aa00700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f8100038ba0 0x7f810003b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.098+0000 7f811aa00700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f8100038ba0 0x7f810003b060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8108006fd0 tx=0x7f8108006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.102+0000 7f8113e00700 1 -- 172.21.15.116:0/2690850167 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f810c019c70 con 0x7f811c108360 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.357+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f811c02db70 con 0x7f811c108360 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.363+0000 7f8113e00700 1 -- 172.21.15.116:0/2690850167 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f810c04c600 con 0x7f811c108360 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.365+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f8100038ba0 msgr2=0x7f810003b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.365+0000 7f8122400700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f8100038ba0 0x7f810003b060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8108006fd0 tx=0x7f8108006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.365+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 msgr2=0x7f811c19bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.365+0000 7f8122400700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c19bb70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f810c011b60 tx=0x7f810c011c40 comp rx=0 tx=0).stop 2024-03-04T07:29:45.241 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.365+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 shutdown_connections 2024-03-04T07:29:45.242 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.366+0000 7f8122400700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f8100038ba0 0x7f810003b060 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:45.242 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.366+0000 7f8122400700 1 --2- 172.21.15.116:0/2690850167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f811c108360 0x7f811c19bb70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:45.242 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.366+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 >> 172.21.15.116:0/2690850167 conn(0x7f811c101e70 msgr2=0x7f811c102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:45.242 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.366+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 shutdown_connections 2024-03-04T07:29:45.242 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:44.366+0000 7f8122400700 1 -- 172.21.15.116:0/2690850167 wait complete. 2024-03-04T07:29:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:45 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2690850167' entity='client.admin' 2024-03-04T07:29:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:45 smithi116 ceph-mon[32150]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.948+0000 7ff340800700 1 Processor -- start 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.949+0000 7ff340800700 1 -- start start 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.949+0000 7ff340800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff33810f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.949+0000 7ff340800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33810fe70 con 0x7ff33810f510 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.950+0000 7ff33de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff33810f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.950+0000 7ff33de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff33810f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46670/0 (socket says 172.21.15.116:46670) 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.950+0000 7ff33de00700 1 -- 172.21.15.116:0/3253774541 learned_addr learned my addr 172.21.15.116:0/3253774541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:46.843 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.950+0000 7ff33de00700 1 -- 172.21.15.116:0/3253774541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33810dcd0 con 0x7ff33810f510 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.950+0000 7ff33de00700 1 --2- 172.21.15.116:0/3253774541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff33810f930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0098a0 tx=0x7ff32c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd422ba7919d462 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.951+0000 7ff33ca00700 1 -- 172.21.15.116:0/3253774541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c004160 con 0x7ff33810f510 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.951+0000 7ff33ca00700 1 -- 172.21.15.116:0/3253774541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff32c004300 con 0x7ff33810f510 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.951+0000 7ff33ca00700 1 -- 172.21.15.116:0/3253774541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c011b20 con 0x7ff33810f510 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.951+0000 7ff340800700 1 -- 172.21.15.116:0/3253774541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 msgr2=0x7ff33810f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.951+0000 7ff340800700 1 --2- 172.21.15.116:0/3253774541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff33810f930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0098a0 tx=0x7ff32c009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.951+0000 7ff340800700 1 -- 172.21.15.116:0/3253774541 shutdown_connections 2024-03-04T07:29:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.952+0000 7ff340800700 1 --2- 172.21.15.116:0/3253774541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff33810f930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.952+0000 7ff340800700 1 -- 172.21.15.116:0/3253774541 >> 172.21.15.116:0/3253774541 conn(0x7ff33807d0b0 msgr2=0x7ff33807f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.952+0000 7ff340800700 1 -- 172.21.15.116:0/3253774541 shutdown_connections 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.952+0000 7ff340800700 1 -- 172.21.15.116:0/3253774541 wait complete. 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.953+0000 7ff340800700 1 Processor -- start 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.953+0000 7ff340800700 1 -- start start 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.953+0000 7ff340800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff3381a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.953+0000 7ff340800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3381124c0 con 0x7ff33810f510 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.953+0000 7ff33de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff3381a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.953+0000 7ff33de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff3381a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46678/0 (socket says 172.21.15.116:46678) 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.953+0000 7ff33de00700 1 -- 172.21.15.116:0/2818821823 learned_addr learned my addr 172.21.15.116:0/2818821823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:46.845 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.954+0000 7ff33de00700 1 -- 172.21.15.116:0/2818821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3381a2910 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.954+0000 7ff33de00700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff3381a23d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff32c000c00 tx=0x7ff32c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.954+0000 7ff332a00700 1 -- 172.21.15.116:0/2818821823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c003b60 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.954+0000 7ff332a00700 1 -- 172.21.15.116:0/2818821823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff32c01a910 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.954+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3381a2ba0 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.954+0000 7ff332a00700 1 -- 172.21.15.116:0/2818821823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff32c024930 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.954+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3381a3670 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.955+0000 7ff332a00700 1 -- 172.21.15.116:0/2818821823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7ff32c022020 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.956+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff338112640 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.956+0000 7ff332a00700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff320038bf0 0x7ff32003b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.956+0000 7ff332a00700 1 -- 172.21.15.116:0/2818821823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff32c04cda0 con 0x7ff33810f510 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.956+0000 7ff33d400700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff320038bf0 0x7ff32003b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.957+0000 7ff33d400700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff320038bf0 0x7ff32003b0b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff328006fd0 tx=0x7ff328006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:46.846 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:45.961+0000 7ff332a00700 1 -- 172.21.15.116:0/2818821823 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff32c00be30 con 0x7ff33810f510 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.180+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff3381a40c0 con 0x7ff33810f510 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.183+0000 7ff332a00700 1 -- 172.21.15.116:0/2818821823 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff32c04c6f0 con 0x7ff33810f510 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff320038bf0 msgr2=0x7ff32003b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff320038bf0 0x7ff32003b0b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff328006fd0 tx=0x7ff328006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 msgr2=0x7ff3381a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff3381a23d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff32c000c00 tx=0x7ff32c011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 shutdown_connections 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7ff320038bf0 0x7ff32003b0b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 --2- 172.21.15.116:0/2818821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff33810f510 0x7ff3381a23d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 >> 172.21.15.116:0/2818821823 conn(0x7ff33807d0b0 msgr2=0x7ff33807dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 shutdown_connections 2024-03-04T07:29:46.847 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:46.186+0000 7ff340800700 1 -- 172.21.15.116:0/2818821823 wait complete. 2024-03-04T07:29:46.848 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling the dashboard module... 2024-03-04T07:29:47.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:47 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2818821823' entity='client.admin' 2024-03-04T07:29:47.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:47 smithi116 ceph-mon[32150]: from='mgr.14120 172.21.15.116:0/2563209964' entity='mgr.x' 2024-03-04T07:29:48.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:48 smithi116 ceph-mon[32150]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:29:48.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:48 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/370682490' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-04T07:29:48.964 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: ignoring --setuser ceph since I am not root 2024-03-04T07:29:48.964 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: ignoring --setgroup ceph since I am not root 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.455+0000 7fea32600700 1 Processor -- start 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.455+0000 7fea32600700 1 -- start start 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.455+0000 7fea32600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.455+0000 7fea32600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c10eaa0 con 0x7fea2c10e0b0 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.456+0000 7fea31200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.456+0000 7fea31200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c10e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46686/0 (socket says 172.21.15.116:46686) 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.456+0000 7fea31200700 1 -- 172.21.15.116:0/2811641099 learned_addr learned my addr 172.21.15.116:0/2811641099 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:49.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.456+0000 7fea31200700 1 -- 172.21.15.116:0/2811641099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c10ef60 con 0x7fea2c10e0b0 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.457+0000 7fea31200700 1 --2- 172.21.15.116:0/2811641099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c10e4d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fea200098a0 tx=0x7fea20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a037038f76bbe5dd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.457+0000 7fea2b400700 1 -- 172.21.15.116:0/2811641099 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea20004160 con 0x7fea2c10e0b0 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.457+0000 7fea2b400700 1 -- 172.21.15.116:0/2811641099 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea20004300 con 0x7fea2c10e0b0 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.457+0000 7fea2b400700 1 -- 172.21.15.116:0/2811641099 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea20011b20 con 0x7fea2c10e0b0 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.457+0000 7fea32600700 1 -- 172.21.15.116:0/2811641099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 msgr2=0x7fea2c10e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.457+0000 7fea32600700 1 --2- 172.21.15.116:0/2811641099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c10e4d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fea200098a0 tx=0x7fea20009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.458+0000 7fea32600700 1 -- 172.21.15.116:0/2811641099 shutdown_connections 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.458+0000 7fea32600700 1 --2- 172.21.15.116:0/2811641099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c10e4d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.458+0000 7fea32600700 1 -- 172.21.15.116:0/2811641099 >> 172.21.15.116:0/2811641099 conn(0x7fea2c07ce90 msgr2=0x7fea2c07f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:49.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.458+0000 7fea32600700 1 -- 172.21.15.116:0/2811641099 shutdown_connections 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.458+0000 7fea32600700 1 -- 172.21.15.116:0/2811641099 wait complete. 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.459+0000 7fea32600700 1 Processor -- start 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.459+0000 7fea32600700 1 -- start start 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.460+0000 7fea32600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c1a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.460+0000 7fea32600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c112310 con 0x7fea2c10e0b0 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.460+0000 7fea31200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c1a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.460+0000 7fea31200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c1a1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46698/0 (socket says 172.21.15.116:46698) 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.460+0000 7fea31200700 1 -- 172.21.15.116:0/370682490 learned_addr learned my addr 172.21.15.116:0/370682490 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.460+0000 7fea31200700 1 -- 172.21.15.116:0/370682490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c1a24a0 con 0x7fea2c10e0b0 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.461+0000 7fea31200700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c1a1f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fea200117e0 tx=0x7fea20011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.461+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea20003bf0 con 0x7fea2c10e0b0 2024-03-04T07:29:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.461+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c1a2730 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.461+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea2001a910 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.461+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea200249a0 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.461+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c1a2b90 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.462+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fea20022020 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.462+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c069160 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.462+0000 7fea29600700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fea14038b50 0x7fea1403b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.463+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fea2004cd10 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.463+0000 7fea2be00700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fea14038b50 0x7fea1403b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.463+0000 7fea2be00700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fea14038b50 0x7fea1403b010 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fea18006fd0 tx=0x7fea18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.468+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fea20011e20 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:47.750+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fea2c02dab0 con 0x7fea2c10e0b0 2024-03-04T07:29:49.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.543+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fea2000b3e0 con 0x7fea2c10e0b0 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.545+0000 7fea29600700 1 -- 172.21.15.116:0/370682490 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 9) v1 ==== 45941+0+0 (secure 0 0 0) 0x7fea20053080 con 0x7fea2c10e0b0 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.548+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fea14038b50 msgr2=0x7fea1403b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.548+0000 7fea32600700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fea14038b50 0x7fea1403b010 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fea18006fd0 tx=0x7fea18006e40 comp rx=0 tx=0).stop 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.548+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 msgr2=0x7fea2c1a1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.548+0000 7fea32600700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c1a1f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fea200117e0 tx=0x7fea20011c10 comp rx=0 tx=0).stop 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.549+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 shutdown_connections 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.549+0000 7fea32600700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7fea14038b50 0x7fea1403b010 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.549+0000 7fea32600700 1 --2- 172.21.15.116:0/370682490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea2c10e0b0 0x7fea2c1a1f60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.549+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 >> 172.21.15.116:0/370682490 conn(0x7fea2c07ce90 msgr2=0x7fea2c07da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.549+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 shutdown_connections 2024-03-04T07:29:49.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:48.549+0000 7fea32600700 1 -- 172.21.15.116:0/370682490 wait complete. 2024-03-04T07:29:49.543 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:49.385+0000 7f5285ae2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T07:29:49.543 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:49.504+0000 7f5285ae2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T07:29:49.800 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:49 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/370682490' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-04T07:29:49.801 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:49 smithi116 ceph-mon[32150]: mgrmap e9: x(active, since 25s) 2024-03-04T07:29:50.429 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:50.309+0000 7f5285ae2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T07:29:50.430 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:50.429+0000 7f5285ae2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T07:29:50.733 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:50 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1383035731' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T07:29:50.733 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:50.608+0000 7f5285ae2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T07:29:50.733 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:50.732+0000 7f5285ae2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T07:29:51.004 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:50.842+0000 7f5285ae2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.943+0000 7f1679800700 1 Processor -- start 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.943+0000 7f1679800700 1 -- start start 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.943+0000 7f1679800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167407b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.943+0000 7f1679800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167407b820 con 0x7f167407ce30 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.944+0000 7f1672a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167407b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.944+0000 7f1672a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167407b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46722/0 (socket says 172.21.15.116:46722) 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.944+0000 7f1672a00700 1 -- 172.21.15.116:0/3566852347 learned_addr learned my addr 172.21.15.116:0/3566852347 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.944+0000 7f1672a00700 1 -- 172.21.15.116:0/3566852347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167407b9a0 con 0x7f167407ce30 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.945+0000 7f1672a00700 1 --2- 172.21.15.116:0/3566852347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167407b2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f16640098a0 tx=0x7f1664009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8e403417fbe72b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.945+0000 7f1671600700 1 -- 172.21.15.116:0/3566852347 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1664004160 con 0x7f167407ce30 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.945+0000 7f1671600700 1 -- 172.21.15.116:0/3566852347 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1664004300 con 0x7f167407ce30 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.945+0000 7f1671600700 1 -- 172.21.15.116:0/3566852347 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1664011b20 con 0x7f167407ce30 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.946+0000 7f1679800700 1 -- 172.21.15.116:0/3566852347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 msgr2=0x7f167407b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.946+0000 7f1679800700 1 --2- 172.21.15.116:0/3566852347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167407b2e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f16640098a0 tx=0x7f1664009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.946+0000 7f1679800700 1 -- 172.21.15.116:0/3566852347 shutdown_connections 2024-03-04T07:29:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.946+0000 7f1679800700 1 --2- 172.21.15.116:0/3566852347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167407b2e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.946+0000 7f1679800700 1 -- 172.21.15.116:0/3566852347 >> 172.21.15.116:0/3566852347 conn(0x7f1674102f60 msgr2=0x7f1674105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.946+0000 7f1679800700 1 -- 172.21.15.116:0/3566852347 shutdown_connections 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.946+0000 7f1679800700 1 -- 172.21.15.116:0/3566852347 wait complete. 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.947+0000 7f1679800700 1 Processor -- start 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.947+0000 7f1679800700 1 -- start start 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.947+0000 7f1679800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.947+0000 7f1679800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167410dec0 con 0x7f167407ce30 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.948+0000 7f1672a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.948+0000 7f1672a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167419dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46732/0 (socket says 172.21.15.116:46732) 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.948+0000 7f1672a00700 1 -- 172.21.15.116:0/1383035731 learned_addr learned my addr 172.21.15.116:0/1383035731 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.948+0000 7f1672a00700 1 -- 172.21.15.116:0/1383035731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167419e230 con 0x7f167407ce30 2024-03-04T07:29:51.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.948+0000 7f1672a00700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167419dcf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16640117e0 tx=0x7f1664011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.949+0000 7f166b400700 1 -- 172.21.15.116:0/1383035731 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f166401a430 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.949+0000 7f166b400700 1 -- 172.21.15.116:0/1383035731 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f166401aab0 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.949+0000 7f166b400700 1 -- 172.21.15.116:0/1383035731 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1664024a60 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.949+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167419e4c0 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.949+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167419e920 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.950+0000 7f166b400700 1 -- 172.21.15.116:0/1383035731 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f1664022020 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.950+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1674069160 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.950+0000 7f166b400700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 0x7f165803b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.950+0000 7f166b400700 1 -- 172.21.15.116:0/1383035731 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f166404d250 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.951+0000 7f1672000700 1 -- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 msgr2=0x7f165803b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3218338096 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.951+0000 7f1672000700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 0x7f165803b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:49.956+0000 7f166b400700 1 -- 172.21.15.116:0/1383035731 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f166402c3a0 con 0x7f167407ce30 2024-03-04T07:29:51.022 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.152+0000 7f1672000700 1 -- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 msgr2=0x7f165803b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3218338096 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.152+0000 7f1672000700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 0x7f165803b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.237+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f167402dab0 con 0x7f167407ce30 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.237+0000 7f166b400700 1 -- 172.21.15.116:0/1383035731 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f166402c580 con 0x7f167407ce30 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 msgr2=0x7f165803b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 0x7f165803b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 msgr2=0x7f167419dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167419dcf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16640117e0 tx=0x7f1664011ce0 comp rx=0 tx=0).stop 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 shutdown_connections 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f1658038c50 0x7f165803b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 --2- 172.21.15.116:0/1383035731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f167407ce30 0x7f167419dcf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 >> 172.21.15.116:0/1383035731 conn(0x7f1674102f60 msgr2=0x7f1674103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.240+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 shutdown_connections 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:50.241+0000 7f1679800700 1 -- 172.21.15.116:0/1383035731 wait complete. 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-04T07:29:51.023 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 9... 2024-03-04T07:29:51.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:51.174+0000 7f5285ae2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T07:29:51.615 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:51.321+0000 7f5285ae2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T07:29:51.615 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:51.438+0000 7f5285ae2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T07:29:51.615 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:51.564+0000 7f5285ae2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T07:29:51.887 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:51.785+0000 7f5285ae2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T07:29:53.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:53.036+0000 7f5285ae2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T07:29:53.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:53.204+0000 7f5285ae2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T07:29:53.986 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:53.614+0000 7f5285ae2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T07:29:53.986 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:53.731+0000 7f5285ae2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T07:29:53.986 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:53.985+0000 7f5285ae2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:29:54.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:54.226+0000 7f5285ae2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T07:29:54.915 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:54.597+0000 7f5285ae2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:29:54.915 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:54.913+0000 7f5285ae2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T07:29:55.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:55 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:55.213+0000 7f5285ae2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T07:29:56.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:56 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:56.223+0000 7f5285ae2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T07:29:57.257 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:56 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:56.914+0000 7f5285ae2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T07:29:57.257 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:57.044+0000 7f5285ae2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T07:29:57.515 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:29:57.327+0000 7f5285ae2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: Active manager daemon x restarted 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: Activating manager daemon x 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: osdmap e3: 0 total, 0 up, 0 in 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: mgrmap e10: x(active, starting, since 0.00446171s) 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T07:29:57.516 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T07:29:57.517 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T07:29:57.517 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:57 smithi116 ceph-mon[32150]: Manager daemon x is now available 2024-03-04T07:29:58.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:58.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:29:58.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T07:29:58.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T07:29:58.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:58 smithi116 ceph-mon[32150]: mgrmap e11: x(active, since 1.01204s) 2024-03-04T07:29:59.739 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:29:59 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:58] ENGINE Bus STARTING 2024-03-04T07:29:59.750 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-04T07:29:59.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-04T07:29:59.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T07:29:59.751 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-04T07:29:59.752 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.677+0000 7f414d600700 1 Processor -- start 2024-03-04T07:29:59.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.677+0000 7f414d600700 1 -- start start 2024-03-04T07:29:59.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.678+0000 7f414d600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414810a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:59.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.678+0000 7f414d600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414807ae70 con 0x7f41481083c0 2024-03-04T07:29:59.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.678+0000 7f4146a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414810a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:59.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.678+0000 7f4146a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414810a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46738/0 (socket says 172.21.15.116:46738) 2024-03-04T07:29:59.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.678+0000 7f4146a00700 1 -- 172.21.15.116:0/3205925431 learned_addr learned my addr 172.21.15.116:0/3205925431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:59.753 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.679+0000 7f4146a00700 1 -- 172.21.15.116:0/3205925431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414810acf0 con 0x7f41481083c0 2024-03-04T07:29:59.755 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.679+0000 7f4146a00700 1 --2- 172.21.15.116:0/3205925431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414810a7b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41380098a0 tx=0x7f4138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e34bc7697b07492 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.679+0000 7f4145600700 1 -- 172.21.15.116:0/3205925431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4138004160 con 0x7f41481083c0 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.679+0000 7f4145600700 1 -- 172.21.15.116:0/3205925431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4138004300 con 0x7f41481083c0 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.679+0000 7f4145600700 1 -- 172.21.15.116:0/3205925431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4138011b20 con 0x7f41481083c0 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.680+0000 7f414d600700 1 -- 172.21.15.116:0/3205925431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 msgr2=0x7f414810a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.680+0000 7f414d600700 1 --2- 172.21.15.116:0/3205925431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414810a7b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41380098a0 tx=0x7f4138009bb0 comp rx=0 tx=0).stop 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.680+0000 7f414d600700 1 -- 172.21.15.116:0/3205925431 shutdown_connections 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.680+0000 7f414d600700 1 --2- 172.21.15.116:0/3205925431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414810a7b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.680+0000 7f414d600700 1 -- 172.21.15.116:0/3205925431 >> 172.21.15.116:0/3205925431 conn(0x7f4148101e90 msgr2=0x7f41481042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.680+0000 7f414d600700 1 -- 172.21.15.116:0/3205925431 shutdown_connections 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.680+0000 7f414d600700 1 -- 172.21.15.116:0/3205925431 wait complete. 2024-03-04T07:29:59.756 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.681+0000 7f414d600700 1 Processor -- start 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.682+0000 7f414d600700 1 -- start start 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.682+0000 7f414d600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.682+0000 7f414d600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414810bd50 con 0x7f41481083c0 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.682+0000 7f4146a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.682+0000 7f4146a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414819dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46742/0 (socket says 172.21.15.116:46742) 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.682+0000 7f4146a00700 1 -- 172.21.15.116:0/336600674 learned_addr learned my addr 172.21.15.116:0/336600674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.683+0000 7f4146a00700 1 -- 172.21.15.116:0/336600674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414819e320 con 0x7f41481083c0 2024-03-04T07:29:59.757 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.683+0000 7f4146a00700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414819dde0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4138011b60 tx=0x7f4138011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.683+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4138003b60 con 0x7f41481083c0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.683+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f413801a910 con 0x7f41481083c0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.683+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4138024930 con 0x7f41481083c0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.683+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414819e5b0 con 0x7f41481083c0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.683+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414819ea10 con 0x7f41481083c0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.685+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f4138022020 con 0x7f41481083c0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.685+0000 7f413f400700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.685+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 --> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f412c03b7c0 con 0x7f412c038bf0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.685+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f413804cd60 con 0x7f41481083c0 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.685+0000 7f4146000700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 msgr2=0x7f412c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3218338096 2024-03-04T07:29:59.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.685+0000 7f4146000700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.886+0000 7f4146000700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 msgr2=0x7f412c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3218338096 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:51.886+0000 7f4146000700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:52.287+0000 7f4146000700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 msgr2=0x7f412c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3218338096 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:52.287+0000 7f4146000700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:53.088+0000 7f4146000700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 msgr2=0x7f412c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3218338096 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:53.088+0000 7f4146000700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:54.690+0000 7f4146000700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 msgr2=0x7f412c03b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3218338096 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:54.690+0000 7f4146000700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:57.334+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 10) v1 ==== 45708+0+0 (secure 0 0 0) 0x7f4138024ba0 con 0x7f41481083c0 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:57.334+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 msgr2=0x7f412c03b0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:57.334+0000 7f413f400700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.339+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 11) v1 ==== 45835+0+0 (secure 0 0 0) 0x7f413804e4b0 con 0x7f41481083c0 2024-03-04T07:29:59.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.339+0000 7f413f400700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f412c03c6b0 0x7f412c03eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.339+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f413800af30 con 0x7f412c03c6b0 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.341+0000 7f4146000700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f412c03c6b0 0x7f412c03eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.343+0000 7f4146000700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f412c03c6b0 0x7f412c03eaa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4134003a10 tx=0x7f41340092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.344+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f413800af30 con 0x7f412c03c6b0 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.357+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4148069160 con 0x7f412c03c6b0 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.358+0000 7f413f400700 1 -- 172.21.15.116:0/336600674 <== mgr.14152 v2:172.21.15.116:6800/142890918 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4148069160 con 0x7f412c03c6b0 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.358+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f412c03c6b0 msgr2=0x7f412c03eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.358+0000 7f414d600700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f412c03c6b0 0x7f412c03eaa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4134003a10 tx=0x7f41340092b0 comp rx=0 tx=0).stop 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.358+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 msgr2=0x7f414819dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.358+0000 7f414d600700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414819dde0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4138011b60 tx=0x7f4138011c40 comp rx=0 tx=0).stop 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.359+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 shutdown_connections 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.359+0000 7f414d600700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f412c03c6b0 0x7f412c03eaa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:59.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.359+0000 7f414d600700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41481083c0 0x7f414819dde0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:59.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.359+0000 7f414d600700 1 --2- 172.21.15.116:0/336600674 >> [v2:172.21.15.116:6800/3218338096,v1:172.21.15.116:6801/3218338096] conn(0x7f412c038bf0 0x7f412c03b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:29:59.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.359+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 >> 172.21.15.116:0/336600674 conn(0x7f4148101e90 msgr2=0x7f4148102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:29:59.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.359+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 shutdown_connections 2024-03-04T07:29:59.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:29:58.359+0000 7f414d600700 1 -- 172.21.15.116:0/336600674 wait complete. 2024-03-04T07:29:59.765 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 9 is available 2024-03-04T07:29:59.765 INFO:teuthology.orchestra.run.smithi116.stdout:Generating a dashboard self-signed certificate... 2024-03-04T07:30:00.663 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:00 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:58] ENGINE Serving on https://172.21.15.116:7150 2024-03-04T07:30:00.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:00 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:58] ENGINE Serving on http://172.21.15.116:8765 2024-03-04T07:30:00.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:00 smithi116 ceph-mon[32150]: [04/Mar/2024:07:29:58] ENGINE Bus STARTED 2024-03-04T07:30:00.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:00 smithi116 ceph-mon[32150]: mgrmap e12: x(active, since 2s) 2024-03-04T07:30:00.664 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:00 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:01.479 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:01 smithi116 ceph-mon[32150]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:30:01.479 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:01 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:01.479 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:01 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:01.492 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.420+0000 7f907dc00700 1 Processor -- start 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.420+0000 7f907dc00700 1 -- start start 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.420+0000 7f907dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907807b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.420+0000 7f907dc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907807b820 con 0x7f907807ce30 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.421+0000 7f9076a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907807b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.421+0000 7f9076a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907807b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50592/0 (socket says 172.21.15.116:50592) 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.421+0000 7f9076a00700 1 -- 172.21.15.116:0/951626637 learned_addr learned my addr 172.21.15.116:0/951626637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.421+0000 7f9076a00700 1 -- 172.21.15.116:0/951626637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907807b9a0 con 0x7f907807ce30 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.422+0000 7f9076a00700 1 --2- 172.21.15.116:0/951626637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907807b2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f90680098a0 tx=0x7f9068009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a85eac1520bdfb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.422+0000 7f9075600700 1 -- 172.21.15.116:0/951626637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9068004160 con 0x7f907807ce30 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.422+0000 7f9075600700 1 -- 172.21.15.116:0/951626637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9068004300 con 0x7f907807ce30 2024-03-04T07:30:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.422+0000 7f9075600700 1 -- 172.21.15.116:0/951626637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9068011b20 con 0x7f907807ce30 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.422+0000 7f907dc00700 1 -- 172.21.15.116:0/951626637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 msgr2=0x7f907807b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.422+0000 7f907dc00700 1 --2- 172.21.15.116:0/951626637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907807b2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f90680098a0 tx=0x7f9068009bb0 comp rx=0 tx=0).stop 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.423+0000 7f907dc00700 1 -- 172.21.15.116:0/951626637 shutdown_connections 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.423+0000 7f907dc00700 1 --2- 172.21.15.116:0/951626637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907807b2e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.423+0000 7f907dc00700 1 -- 172.21.15.116:0/951626637 >> 172.21.15.116:0/951626637 conn(0x7f9078102f60 msgr2=0x7f9078105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.423+0000 7f907dc00700 1 -- 172.21.15.116:0/951626637 shutdown_connections 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.423+0000 7f907dc00700 1 -- 172.21.15.116:0/951626637 wait complete. 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.424+0000 7f907dc00700 1 Processor -- start 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.424+0000 7f907dc00700 1 -- start start 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.424+0000 7f907dc00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.424+0000 7f907dc00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907810dec0 con 0x7f907807ce30 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.425+0000 7f9076a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.425+0000 7f9076a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907819dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50602/0 (socket says 172.21.15.116:50602) 2024-03-04T07:30:01.494 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.425+0000 7f9076a00700 1 -- 172.21.15.116:0/1363393155 learned_addr learned my addr 172.21.15.116:0/1363393155 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.425+0000 7f9076a00700 1 -- 172.21.15.116:0/1363393155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907819e230 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.425+0000 7f9076a00700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907819dcf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f90680117e0 tx=0x7f9068011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.426+0000 7f906f400700 1 -- 172.21.15.116:0/1363393155 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f906801a430 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.426+0000 7f906f400700 1 -- 172.21.15.116:0/1363393155 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f906801aab0 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.426+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907819e4c0 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.426+0000 7f906f400700 1 -- 172.21.15.116:0/1363393155 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9068024a60 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.426+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907819e920 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.427+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9078069160 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.427+0000 7f906f400700 1 -- 172.21.15.116:0/1363393155 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f9068024d10 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.428+0000 7f906f400700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f905c038bf0 0x7f905c03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.428+0000 7f906f400700 1 -- 172.21.15.116:0/1363393155 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f906804d360 con 0x7f907807ce30 2024-03-04T07:30:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.428+0000 7f9076000700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f905c038bf0 0x7f905c03b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.429+0000 7f9076000700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f905c038bf0 0x7f905c03b0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9064006fd0 tx=0x7f9064006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.433+0000 7f906f400700 1 -- 172.21.15.116:0/1363393155 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f906802c3a0 con 0x7f907807ce30 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.662+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f907802db20 con 0x7f905c038bf0 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.779+0000 7f906f400700 1 -- 172.21.15.116:0/1363393155 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f907802db20 con 0x7f905c038bf0 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.781+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f905c038bf0 msgr2=0x7f905c03b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.781+0000 7f907dc00700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f905c038bf0 0x7f905c03b0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9064006fd0 tx=0x7f9064006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.781+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 msgr2=0x7f907819dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.781+0000 7f907dc00700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907819dcf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f90680117e0 tx=0x7f9068011ce0 comp rx=0 tx=0).stop 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.781+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 shutdown_connections 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.782+0000 7f907dc00700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f905c038bf0 0x7f905c03b0b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.782+0000 7f907dc00700 1 --2- 172.21.15.116:0/1363393155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f907807ce30 0x7f907819dcf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.782+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 >> 172.21.15.116:0/1363393155 conn(0x7f9078102f60 msgr2=0x7f9078103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.782+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 shutdown_connections 2024-03-04T07:30:01.497 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:00.782+0000 7f907dc00700 1 -- 172.21.15.116:0/1363393155 wait complete. 2024-03-04T07:30:01.497 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial admin user... 2024-03-04T07:30:03.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:02 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:03.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:02 smithi116 ceph-mon[32150]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:30:03.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:02 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$A2u7jrZRrGw7G8Kf8WYI2OVw10OPjH0DZmWLNQmju2R.tmmGWKm8y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709537402, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.140+0000 7f1b27000700 1 Processor -- start 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.140+0000 7f1b27000700 1 -- start start 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.140+0000 7f1b27000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b20107d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.140+0000 7f1b27000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b20108320 con 0x7f1b20107930 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.141+0000 7f1b1fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b20107d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.141+0000 7f1b1fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b20107d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50604/0 (socket says 172.21.15.116:50604) 2024-03-04T07:30:03.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.141+0000 7f1b1fe00700 1 -- 172.21.15.116:0/3047624364 learned_addr learned my addr 172.21.15.116:0/3047624364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.141+0000 7f1b1fe00700 1 -- 172.21.15.116:0/3047624364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b20108ae0 con 0x7f1b20107930 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.142+0000 7f1b1fe00700 1 --2- 172.21.15.116:0/3047624364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b20107d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b100098a0 tx=0x7f1b10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afabdf61bb27745 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.142+0000 7f1b1ea00700 1 -- 172.21.15.116:0/3047624364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b10004160 con 0x7f1b20107930 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.142+0000 7f1b1ea00700 1 -- 172.21.15.116:0/3047624364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b10004300 con 0x7f1b20107930 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.142+0000 7f1b1ea00700 1 -- 172.21.15.116:0/3047624364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b10011b20 con 0x7f1b20107930 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.143+0000 7f1b27000700 1 -- 172.21.15.116:0/3047624364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 msgr2=0x7f1b20107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.143+0000 7f1b27000700 1 --2- 172.21.15.116:0/3047624364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b20107d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b100098a0 tx=0x7f1b10009bb0 comp rx=0 tx=0).stop 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.143+0000 7f1b27000700 1 -- 172.21.15.116:0/3047624364 shutdown_connections 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.143+0000 7f1b27000700 1 --2- 172.21.15.116:0/3047624364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b20107d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.143+0000 7f1b27000700 1 -- 172.21.15.116:0/3047624364 >> 172.21.15.116:0/3047624364 conn(0x7f1b20102f50 msgr2=0x7f1b20105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.143+0000 7f1b27000700 1 -- 172.21.15.116:0/3047624364 shutdown_connections 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.143+0000 7f1b27000700 1 -- 172.21.15.116:0/3047624364 wait complete. 2024-03-04T07:30:03.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.144+0000 7f1b27000700 1 Processor -- start 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.144+0000 7f1b27000700 1 -- start start 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.145+0000 7f1b27000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b2019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.145+0000 7f1b27000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2010be90 con 0x7f1b20107930 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.145+0000 7f1b1fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b2019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.145+0000 7f1b1fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b2019bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50616/0 (socket says 172.21.15.116:50616) 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.145+0000 7f1b1fe00700 1 -- 172.21.15.116:0/4104225046 learned_addr learned my addr 172.21.15.116:0/4104225046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.145+0000 7f1b1fe00700 1 -- 172.21.15.116:0/4104225046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2019c0c0 con 0x7f1b20107930 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.146+0000 7f1b1fe00700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b2019bb80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1b100117e0 tx=0x7f1b10011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.146+0000 7f1b1cc00700 1 -- 172.21.15.116:0/4104225046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b10003bf0 con 0x7f1b20107930 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.146+0000 7f1b1cc00700 1 -- 172.21.15.116:0/4104225046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b1001a910 con 0x7f1b20107930 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.146+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2019c350 con 0x7f1b20107930 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.146+0000 7f1b1cc00700 1 -- 172.21.15.116:0/4104225046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b100249a0 con 0x7f1b20107930 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.146+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2019c7b0 con 0x7f1b20107930 2024-03-04T07:30:03.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.147+0000 7f1b1cc00700 1 -- 172.21.15.116:0/4104225046 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f1b10022020 con 0x7f1b20107930 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.147+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b20069160 con 0x7f1b20107930 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.148+0000 7f1b1cc00700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1b04038c00 0x7f1b0403b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.148+0000 7f1b1cc00700 1 -- 172.21.15.116:0/4104225046 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1b1004ce40 con 0x7f1b20107930 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.148+0000 7f1b1f400700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1b04038c00 0x7f1b0403b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.149+0000 7f1b1f400700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1b04038c00 0x7f1b0403b0c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c006fd0 tx=0x7f1b0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.154+0000 7f1b1cc00700 1 -- 172.21.15.116:0/4104225046 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1b10011e20 con 0x7f1b20107930 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.391+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1b201957c0 con 0x7f1b04038c00 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.643+0000 7f1b1cc00700 1 -- 172.21.15.116:0/4104225046 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1b201957c0 con 0x7f1b04038c00 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1b04038c00 msgr2=0x7f1b0403b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1b04038c00 0x7f1b0403b0c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c006fd0 tx=0x7f1b0c006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 msgr2=0x7f1b2019bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:03.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b2019bb80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1b100117e0 tx=0x7f1b10011c10 comp rx=0 tx=0).stop 2024-03-04T07:30:03.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 shutdown_connections 2024-03-04T07:30:03.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1b04038c00 0x7f1b0403b0c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:03.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 --2- 172.21.15.116:0/4104225046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b20107930 0x7f1b2019bb80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:03.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.646+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 >> 172.21.15.116:0/4104225046 conn(0x7f1b20102f50 msgr2=0x7f1b20103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:03.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.647+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 shutdown_connections 2024-03-04T07:30:03.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:02.647+0000 7f1b27000700 1 -- 172.21.15.116:0/4104225046 wait complete. 2024-03-04T07:30:03.561 INFO:teuthology.orchestra.run.smithi116.stdout:Fetching dashboard port number... 2024-03-04T07:30:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:04 smithi116 ceph-mon[32150]: mgrmap e13: x(active, since 6s) 2024-03-04T07:30:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:04 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1804964910' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-04T07:30:05.328 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 8443 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.352+0000 7fb851200700 1 Processor -- start 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.352+0000 7fb851200700 1 -- start start 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.352+0000 7fb851200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c10b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.352+0000 7fb851200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c109780 con 0x7fb84c10b1f0 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.353+0000 7fb849e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c10b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.353+0000 7fb849e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c10b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50618/0 (socket says 172.21.15.116:50618) 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.353+0000 7fb849e00700 1 -- 172.21.15.116:0/333870620 learned_addr learned my addr 172.21.15.116:0/333870620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.353+0000 7fb849e00700 1 -- 172.21.15.116:0/333870620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c109900 con 0x7fb84c10b1f0 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.353+0000 7fb849e00700 1 --2- 172.21.15.116:0/333870620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c10b610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0098a0 tx=0x7fb83c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbf946582543960 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:05.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.354+0000 7fb848a00700 1 -- 172.21.15.116:0/333870620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb83c004160 con 0x7fb84c10b1f0 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.354+0000 7fb848a00700 1 -- 172.21.15.116:0/333870620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb83c004300 con 0x7fb84c10b1f0 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.354+0000 7fb848a00700 1 -- 172.21.15.116:0/333870620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb83c011b20 con 0x7fb84c10b1f0 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.355+0000 7fb851200700 1 -- 172.21.15.116:0/333870620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 msgr2=0x7fb84c10b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.355+0000 7fb851200700 1 --2- 172.21.15.116:0/333870620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c10b610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0098a0 tx=0x7fb83c009bb0 comp rx=0 tx=0).stop 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.355+0000 7fb851200700 1 -- 172.21.15.116:0/333870620 shutdown_connections 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.355+0000 7fb851200700 1 --2- 172.21.15.116:0/333870620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c10b610 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.355+0000 7fb851200700 1 -- 172.21.15.116:0/333870620 >> 172.21.15.116:0/333870620 conn(0x7fb84c1054c0 msgr2=0x7fb84c1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.355+0000 7fb851200700 1 -- 172.21.15.116:0/333870620 shutdown_connections 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.355+0000 7fb851200700 1 -- 172.21.15.116:0/333870620 wait complete. 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.356+0000 7fb851200700 1 Processor -- start 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.356+0000 7fb851200700 1 -- start start 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.356+0000 7fb851200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c19df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:05.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.356+0000 7fb851200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c10e0c0 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.357+0000 7fb849e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c19df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.357+0000 7fb849e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c19df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50620/0 (socket says 172.21.15.116:50620) 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.357+0000 7fb849e00700 1 -- 172.21.15.116:0/1804964910 learned_addr learned my addr 172.21.15.116:0/1804964910 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.357+0000 7fb849e00700 1 -- 172.21.15.116:0/1804964910 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c19e4c0 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.357+0000 7fb849e00700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c19df80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb83c000c00 tx=0x7fb83c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.358+0000 7fb842a00700 1 -- 172.21.15.116:0/1804964910 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb83c003b60 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.358+0000 7fb842a00700 1 -- 172.21.15.116:0/1804964910 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb83c01a910 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.358+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c19e750 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.358+0000 7fb842a00700 1 -- 172.21.15.116:0/1804964910 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb83c024930 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.358+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c19f270 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.359+0000 7fb842a00700 1 -- 172.21.15.116:0/1804964910 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7fb83c022020 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.359+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c0636f0 con 0x7fb84c10b1f0 2024-03-04T07:30:05.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.359+0000 7fb842a00700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb830038cf0 0x7fb83003b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.360+0000 7fb842a00700 1 -- 172.21.15.116:0/1804964910 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb83c04cf30 con 0x7fb84c10b1f0 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.360+0000 7fb849400700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb830038cf0 0x7fb83003b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.360+0000 7fb849400700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb830038cf0 0x7fb83003b1b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb838006fd0 tx=0x7fb838006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.365+0000 7fb842a00700 1 -- 172.21.15.116:0/1804964910 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb83c02c350 con 0x7fb84c10b1f0 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.585+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb84c075a50 con 0x7fb84c10b1f0 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.586+0000 7fb842a00700 1 -- 172.21.15.116:0/1804964910 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb83c0172b0 con 0x7fb84c10b1f0 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb830038cf0 msgr2=0x7fb83003b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb830038cf0 0x7fb83003b1b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb838006fd0 tx=0x7fb838006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 msgr2=0x7fb84c19df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c19df80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb83c000c00 tx=0x7fb83c011c10 comp rx=0 tx=0).stop 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 shutdown_connections 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb830038cf0 0x7fb83003b1b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:05.337 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 --2- 172.21.15.116:0/1804964910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb84c10b1f0 0x7fb84c19df80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:05.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 >> 172.21.15.116:0/1804964910 conn(0x7fb84c1054c0 msgr2=0x7fb84c107170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:05.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 shutdown_connections 2024-03-04T07:30:05.338 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:04.590+0000 7fb851200700 1 -- 172.21.15.116:0/1804964910 wait complete. 2024-03-04T07:30:05.361 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T07:30:05.361 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-04T07:30:05.386 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T07:30:05.386 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-04T07:30:05.386 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-04T07:30:05.386 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-04T07:30:05.390 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph Dashboard is now available at: 2024-03-04T07:30:05.390 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:05.390 INFO:teuthology.orchestra.run.smithi116.stdout: URL: https://smithi116.front.sepia.ceph.com:8443/ 2024-03-04T07:30:05.390 INFO:teuthology.orchestra.run.smithi116.stdout: User: admin 2024-03-04T07:30:05.390 INFO:teuthology.orchestra.run.smithi116.stdout: Password: be74upzvr4 2024-03-04T07:30:05.390 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:05.391 INFO:teuthology.orchestra.run.smithi116.stdout:Saving cluster configuration to /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config directory 2024-03-04T07:30:05.392 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling autotune for osd_memory_target 2024-03-04T07:30:07.371 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.391+0000 7f5a30200700 1 Processor -- start 2024-03-04T07:30:07.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.391+0000 7f5a30200700 1 -- start start 2024-03-04T07:30:07.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.392+0000 7f5a30200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2810b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:07.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.392+0000 7f5a30200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28109780 con 0x7f5a2810b1f0 2024-03-04T07:30:07.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.392+0000 7f5a2d800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2810b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:07.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.392+0000 7f5a2d800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2810b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50562/0 (socket says 172.21.15.116:50562) 2024-03-04T07:30:07.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.392+0000 7f5a2d800700 1 -- 172.21.15.116:0/1070298791 learned_addr learned my addr 172.21.15.116:0/1070298791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:07.376 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.393+0000 7f5a2d800700 1 -- 172.21.15.116:0/1070298791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28109900 con 0x7f5a2810b1f0 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.393+0000 7f5a2d800700 1 --2- 172.21.15.116:0/1070298791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2810b610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0098a0 tx=0x7f5a1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3487337c5ebf9c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.393+0000 7f5a27e00700 1 -- 172.21.15.116:0/1070298791 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a1c004160 con 0x7f5a2810b1f0 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.393+0000 7f5a27e00700 1 -- 172.21.15.116:0/1070298791 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a1c004300 con 0x7f5a2810b1f0 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.393+0000 7f5a27e00700 1 -- 172.21.15.116:0/1070298791 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a1c011b20 con 0x7f5a2810b1f0 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.394+0000 7f5a30200700 1 -- 172.21.15.116:0/1070298791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 msgr2=0x7f5a2810b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.394+0000 7f5a30200700 1 --2- 172.21.15.116:0/1070298791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2810b610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0098a0 tx=0x7f5a1c009bb0 comp rx=0 tx=0).stop 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.394+0000 7f5a30200700 1 -- 172.21.15.116:0/1070298791 shutdown_connections 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.394+0000 7f5a30200700 1 --2- 172.21.15.116:0/1070298791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2810b610 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.394+0000 7f5a30200700 1 -- 172.21.15.116:0/1070298791 >> 172.21.15.116:0/1070298791 conn(0x7f5a281054c0 msgr2=0x7f5a281078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.394+0000 7f5a30200700 1 -- 172.21.15.116:0/1070298791 shutdown_connections 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.394+0000 7f5a30200700 1 -- 172.21.15.116:0/1070298791 wait complete. 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.395+0000 7f5a30200700 1 Processor -- start 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.395+0000 7f5a30200700 1 -- start start 2024-03-04T07:30:07.377 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.396+0000 7f5a30200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.396+0000 7f5a30200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2810e0c0 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.396+0000 7f5a2d800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.396+0000 7f5a2d800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2819df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50574/0 (socket says 172.21.15.116:50574) 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.396+0000 7f5a2d800700 1 -- 172.21.15.116:0/3284087214 learned_addr learned my addr 172.21.15.116:0/3284087214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.396+0000 7f5a2d800700 1 -- 172.21.15.116:0/3284087214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2819e4c0 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.397+0000 7f5a2d800700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2819df80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c000c00 tx=0x7f5a1c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.397+0000 7f5a26000700 1 -- 172.21.15.116:0/3284087214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a1c003b60 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.397+0000 7f5a26000700 1 -- 172.21.15.116:0/3284087214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a1c01a910 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.397+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2819e750 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.397+0000 7f5a26000700 1 -- 172.21.15.116:0/3284087214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a1c024930 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.397+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2819f270 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.398+0000 7f5a26000700 1 -- 172.21.15.116:0/3284087214 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f5a1c022020 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.398+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a280636f0 con 0x7f5a2810b1f0 2024-03-04T07:30:07.378 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.399+0000 7f5a26000700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a10038cf0 0x7f5a1003b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.399+0000 7f5a26000700 1 -- 172.21.15.116:0/3284087214 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5a1c04ce90 con 0x7f5a2810b1f0 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.399+0000 7f5a2ce00700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a10038cf0 0x7f5a1003b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.399+0000 7f5a2ce00700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a10038cf0 0x7f5a1003b1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006fd0 tx=0x7f5a14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.404+0000 7f5a26000700 1 -- 172.21.15.116:0/3284087214 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5a1c050030 con 0x7f5a2810b1f0 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.622+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f5a28075ac0 con 0x7f5a2810b1f0 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.623+0000 7f5a26000700 1 -- 172.21.15.116:0/3284087214 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5a1c02d3f0 con 0x7f5a2810b1f0 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a10038cf0 msgr2=0x7f5a1003b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a10038cf0 0x7f5a1003b1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5a14006fd0 tx=0x7f5a14006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 msgr2=0x7f5a2819df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2819df80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c000c00 tx=0x7f5a1c011c10 comp rx=0 tx=0).stop 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 shutdown_connections 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a10038cf0 0x7f5a1003b1b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:07.379 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 --2- 172.21.15.116:0/3284087214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2810b1f0 0x7f5a2819df80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:07.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.627+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 >> 172.21.15.116:0/3284087214 conn(0x7f5a281054c0 msgr2=0x7f5a28107170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:07.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.628+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 shutdown_connections 2024-03-04T07:30:07.380 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:06.628+0000 7f5a30200700 1 -- 172.21.15.116:0/3284087214 wait complete. 2024-03-04T07:30:09.010 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.954+0000 7ffa29a00700 1 Processor -- start 2024-03-04T07:30:09.010 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.954+0000 7ffa29a00700 1 -- start start 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.955+0000 7ffa29a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa24109d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.955+0000 7ffa29a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2410a350 con 0x7ffa24109960 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.955+0000 7ffa23e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa24109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.955+0000 7ffa23e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa24109d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50582/0 (socket says 172.21.15.116:50582) 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.955+0000 7ffa23e00700 1 -- 172.21.15.116:0/982644732 learned_addr learned my addr 172.21.15.116:0/982644732 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.956+0000 7ffa23e00700 1 -- 172.21.15.116:0/982644732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2410ab10 con 0x7ffa24109960 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.956+0000 7ffa23e00700 1 --2- 172.21.15.116:0/982644732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa24109d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009e80 tx=0x7ffa1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c509411b3100fd67 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.956+0000 7ffa23400700 1 -- 172.21.15.116:0/982644732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa14017070 con 0x7ffa24109960 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.956+0000 7ffa23400700 1 -- 172.21.15.116:0/982644732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa1400b8c0 con 0x7ffa24109960 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa23400700 1 -- 172.21.15.116:0/982644732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa140045c0 con 0x7ffa24109960 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa29a00700 1 -- 172.21.15.116:0/982644732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 msgr2=0x7ffa24109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa29a00700 1 --2- 172.21.15.116:0/982644732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa24109d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009e80 tx=0x7ffa1400b330 comp rx=0 tx=0).stop 2024-03-04T07:30:09.011 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa29a00700 1 -- 172.21.15.116:0/982644732 shutdown_connections 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa29a00700 1 --2- 172.21.15.116:0/982644732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa24109d80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa29a00700 1 -- 172.21.15.116:0/982644732 >> 172.21.15.116:0/982644732 conn(0x7ffa2407ce90 msgr2=0x7ffa241074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa29a00700 1 -- 172.21.15.116:0/982644732 shutdown_connections 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.957+0000 7ffa29a00700 1 -- 172.21.15.116:0/982644732 wait complete. 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.958+0000 7ffa29a00700 1 Processor -- start 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.959+0000 7ffa29a00700 1 -- start start 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.959+0000 7ffa29a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa2419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.959+0000 7ffa29a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2410dec0 con 0x7ffa24109960 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.959+0000 7ffa23e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa2419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.959+0000 7ffa23e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa2419db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50586/0 (socket says 172.21.15.116:50586) 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.959+0000 7ffa23e00700 1 -- 172.21.15.116:0/501980407 learned_addr learned my addr 172.21.15.116:0/501980407 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.960+0000 7ffa23e00700 1 -- 172.21.15.116:0/501980407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2419e0b0 con 0x7ffa24109960 2024-03-04T07:30:09.012 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.960+0000 7ffa23e00700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa2419db70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009fd0 tx=0x7ffa14003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.960+0000 7ffa21600700 1 -- 172.21.15.116:0/501980407 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa14020020 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.960+0000 7ffa21600700 1 -- 172.21.15.116:0/501980407 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa1400bba0 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.960+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2419e340 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.960+0000 7ffa21600700 1 -- 172.21.15.116:0/501980407 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa14017040 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.960+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa241a1030 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.961+0000 7ffa21600700 1 -- 172.21.15.116:0/501980407 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7ffa14023020 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.961+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa24069160 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.962+0000 7ffa21600700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffa08038cf0 0x7ffa0803b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.962+0000 7ffa21600700 1 -- 172.21.15.116:0/501980407 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffa1400f050 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.962+0000 7ffa1b400700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffa08038cf0 0x7ffa0803b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.963+0000 7ffa1b400700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffa08038cf0 0x7ffa0803b1b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c006fd0 tx=0x7ffa0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:07.967+0000 7ffa21600700 1 -- 172.21.15.116:0/501980407 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffa1402a020 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.296+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7ffa2402d9b0 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.302+0000 7ffa21600700 1 -- 172.21.15.116:0/501980407 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7ffa1401bbb0 con 0x7ffa24109960 2024-03-04T07:30:09.013 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffa08038cf0 msgr2=0x7ffa0803b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffa08038cf0 0x7ffa0803b1b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c006fd0 tx=0x7ffa0c006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 msgr2=0x7ffa2419db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa2419db70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009fd0 tx=0x7ffa14003d00 comp rx=0 tx=0).stop 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 shutdown_connections 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffa08038cf0 0x7ffa0803b1b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 --2- 172.21.15.116:0/501980407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa24109960 0x7ffa2419db70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 >> 172.21.15.116:0/501980407 conn(0x7ffa2407ce90 msgr2=0x7ffa24105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 shutdown_connections 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-04T07:30:08.305+0000 7ffa29a00700 1 -- 172.21.15.116:0/501980407 wait complete. 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.014 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout:Or, if you are only running a single cluster on this host: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph telemetry on 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout:For more information see: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:09.015 INFO:teuthology.orchestra.run.smithi116.stdout:Bootstrap complete. 2024-03-04T07:30:09.102 INFO:tasks.cephadm:Fetching config... 2024-03-04T07:30:09.102 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:30:09.102 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-04T07:30:09.155 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-04T07:30:09.155 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:30:09.155 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-04T07:30:09.199 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-04T07:30:09.199 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:30:09.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-04T07:30:09.322 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-04T07:30:09.322 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:30:09.322 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-04T07:30:09.415 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-04T07:30:09.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBNLlSw5nqlO2AanaUItBKtxH16Auf0M5q2uFeeCPgfLIaiNEOlyC5eyeid3NGBeJiFTH6EyTyJYzo3GkdnOC5nBczvZ58qzR1YRLtV36n+cWsA7P0qwVH6BSSrEuTirKg5PiBkyFduaX/Q4ksiD7f7q4VsAjpuZh+hI/Q5aXUwWHo+3eJzeG9Y68GZQhbpWoQEpw5l/oN5lZCcCPI2zRYec2VIOg81ajFk+oXmXX8cipIg+jNP5frLfbHcqHKI9iIZz5riUspAKhSwNbVw/0g14AQ5fEZglwIqoHua7R6WncHFsEZdw+LnYQxcwKIimrD22A4gssmLwIaiNE8hh4bB81646jk+m0HKZZy3DX8vGDR9m8lKsV2JpeDPlAn+7epReMjsaOQCxk8/AuwHo3Hmi8IxqE+qnCoWIpWGgpObL1jTNHpYLqtomMbPdkn5FLqThaJ2acfCVREPT/m/+3KXNyJJ4lCch/WgX4uhsHz8OIFzmYDG/f5LlOw3aRRE0= ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T07:30:09.539 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBNLlSw5nqlO2AanaUItBKtxH16Auf0M5q2uFeeCPgfLIaiNEOlyC5eyeid3NGBeJiFTH6EyTyJYzo3GkdnOC5nBczvZ58qzR1YRLtV36n+cWsA7P0qwVH6BSSrEuTirKg5PiBkyFduaX/Q4ksiD7f7q4VsAjpuZh+hI/Q5aXUwWHo+3eJzeG9Y68GZQhbpWoQEpw5l/oN5lZCcCPI2zRYec2VIOg81ajFk+oXmXX8cipIg+jNP5frLfbHcqHKI9iIZz5riUspAKhSwNbVw/0g14AQ5fEZglwIqoHua7R6WncHFsEZdw+LnYQxcwKIimrD22A4gssmLwIaiNE8hh4bB81646jk+m0HKZZy3DX8vGDR9m8lKsV2JpeDPlAn+7epReMjsaOQCxk8/AuwHo3Hmi8IxqE+qnCoWIpWGgpObL1jTNHpYLqtomMbPdkn5FLqThaJ2acfCVREPT/m/+3KXNyJJ4lCch/WgX4uhsHz8OIFzmYDG/f5LlOw3aRRE0= ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:30:09.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:09 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/501980407' entity='client.admin' 2024-03-04T07:30:09.583 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBNLlSw5nqlO2AanaUItBKtxH16Auf0M5q2uFeeCPgfLIaiNEOlyC5eyeid3NGBeJiFTH6EyTyJYzo3GkdnOC5nBczvZ58qzR1YRLtV36n+cWsA7P0qwVH6BSSrEuTirKg5PiBkyFduaX/Q4ksiD7f7q4VsAjpuZh+hI/Q5aXUwWHo+3eJzeG9Y68GZQhbpWoQEpw5l/oN5lZCcCPI2zRYec2VIOg81ajFk+oXmXX8cipIg+jNP5frLfbHcqHKI9iIZz5riUspAKhSwNbVw/0g14AQ5fEZglwIqoHua7R6WncHFsEZdw+LnYQxcwKIimrD22A4gssmLwIaiNE8hh4bB81646jk+m0HKZZy3DX8vGDR9m8lKsV2JpeDPlAn+7epReMjsaOQCxk8/AuwHo3Hmi8IxqE+qnCoWIpWGgpObL1jTNHpYLqtomMbPdkn5FLqThaJ2acfCVREPT/m/+3KXNyJJ4lCch/WgX4uhsHz8OIFzmYDG/f5LlOw3aRRE0= ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T07:30:09.629 INFO:teuthology.orchestra.run.smithi133.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBNLlSw5nqlO2AanaUItBKtxH16Auf0M5q2uFeeCPgfLIaiNEOlyC5eyeid3NGBeJiFTH6EyTyJYzo3GkdnOC5nBczvZ58qzR1YRLtV36n+cWsA7P0qwVH6BSSrEuTirKg5PiBkyFduaX/Q4ksiD7f7q4VsAjpuZh+hI/Q5aXUwWHo+3eJzeG9Y68GZQhbpWoQEpw5l/oN5lZCcCPI2zRYec2VIOg81ajFk+oXmXX8cipIg+jNP5frLfbHcqHKI9iIZz5riUspAKhSwNbVw/0g14AQ5fEZglwIqoHua7R6WncHFsEZdw+LnYQxcwKIimrD22A4gssmLwIaiNE8hh4bB81646jk+m0HKZZy3DX8vGDR9m8lKsV2JpeDPlAn+7epReMjsaOQCxk8/AuwHo3Hmi8IxqE+qnCoWIpWGgpObL1jTNHpYLqtomMbPdkn5FLqThaJ2acfCVREPT/m/+3KXNyJJ4lCch/WgX4uhsHz8OIFzmYDG/f5LlOw3aRRE0= ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:30:09.647 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBNLlSw5nqlO2AanaUItBKtxH16Auf0M5q2uFeeCPgfLIaiNEOlyC5eyeid3NGBeJiFTH6EyTyJYzo3GkdnOC5nBczvZ58qzR1YRLtV36n+cWsA7P0qwVH6BSSrEuTirKg5PiBkyFduaX/Q4ksiD7f7q4VsAjpuZh+hI/Q5aXUwWHo+3eJzeG9Y68GZQhbpWoQEpw5l/oN5lZCcCPI2zRYec2VIOg81ajFk+oXmXX8cipIg+jNP5frLfbHcqHKI9iIZz5riUspAKhSwNbVw/0g14AQ5fEZglwIqoHua7R6WncHFsEZdw+LnYQxcwKIimrD22A4gssmLwIaiNE8hh4bB81646jk+m0HKZZy3DX8vGDR9m8lKsV2JpeDPlAn+7epReMjsaOQCxk8/AuwHo3Hmi8IxqE+qnCoWIpWGgpObL1jTNHpYLqtomMbPdkn5FLqThaJ2acfCVREPT/m/+3KXNyJJ4lCch/WgX4uhsHz8OIFzmYDG/f5LlOw3aRRE0= ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T07:30:09.695 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBNLlSw5nqlO2AanaUItBKtxH16Auf0M5q2uFeeCPgfLIaiNEOlyC5eyeid3NGBeJiFTH6EyTyJYzo3GkdnOC5nBczvZ58qzR1YRLtV36n+cWsA7P0qwVH6BSSrEuTirKg5PiBkyFduaX/Q4ksiD7f7q4VsAjpuZh+hI/Q5aXUwWHo+3eJzeG9Y68GZQhbpWoQEpw5l/oN5lZCcCPI2zRYec2VIOg81ajFk+oXmXX8cipIg+jNP5frLfbHcqHKI9iIZz5riUspAKhSwNbVw/0g14AQ5fEZglwIqoHua7R6WncHFsEZdw+LnYQxcwKIimrD22A4gssmLwIaiNE8hh4bB81646jk+m0HKZZy3DX8vGDR9m8lKsV2JpeDPlAn+7epReMjsaOQCxk8/AuwHo3Hmi8IxqE+qnCoWIpWGgpObL1jTNHpYLqtomMbPdkn5FLqThaJ2acfCVREPT/m/+3KXNyJJ4lCch/WgX4uhsHz8OIFzmYDG/f5LlOw3aRRE0= ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:30:09.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-04T07:30:10.137 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:30:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.850+0000 7fd22d200700 1 -- 172.21.15.116:0/936650903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 msgr2=0x7fd228104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.850+0000 7fd22d200700 1 --2- 172.21.15.116:0/936650903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228104110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd218009910 tx=0x7fd218009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.850+0000 7fd22d200700 1 -- 172.21.15.116:0/936650903 shutdown_connections 2024-03-04T07:30:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.850+0000 7fd22d200700 1 --2- 172.21.15.116:0/936650903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228104110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:12.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.850+0000 7fd22d200700 1 -- 172.21.15.116:0/936650903 >> 172.21.15.116:0/936650903 conn(0x7fd2280ff290 msgr2=0x7fd2281016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:12.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.850+0000 7fd22d200700 1 -- 172.21.15.116:0/936650903 shutdown_connections 2024-03-04T07:30:12.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.850+0000 7fd22d200700 1 -- 172.21.15.116:0/936650903 wait complete. 2024-03-04T07:30:12.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.851+0000 7fd22d200700 1 Processor -- start 2024-03-04T07:30:12.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.851+0000 7fd22d200700 1 -- start start 2024-03-04T07:30:12.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.852+0000 7fd22d200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:12.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.852+0000 7fd22d200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd228109d80 con 0x7fd228103cf0 2024-03-04T07:30:12.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.852+0000 7fd225e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:12.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.852+0000 7fd225e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50594/0 (socket says 172.21.15.116:50594) 2024-03-04T07:30:12.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.852+0000 7fd225e00700 1 -- 172.21.15.116:0/1743653389 learned_addr learned my addr 172.21.15.116:0/1743653389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:12.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.852+0000 7fd225e00700 1 -- 172.21.15.116:0/1743653389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd228199940 con 0x7fd228103cf0 2024-03-04T07:30:12.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.853+0000 7fd225e00700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228199400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd218005850 tx=0x7fd218005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:12.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.853+0000 7fd21ea00700 1 -- 172.21.15.116:0/1743653389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd218026070 con 0x7fd228103cf0 2024-03-04T07:30:12.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.853+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd228199bd0 con 0x7fd228103cf0 2024-03-04T07:30:12.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.853+0000 7fd21ea00700 1 -- 172.21.15.116:0/1743653389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2180044a0 con 0x7fd228103cf0 2024-03-04T07:30:12.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.853+0000 7fd21ea00700 1 -- 172.21.15.116:0/1743653389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd218003ac0 con 0x7fd228103cf0 2024-03-04T07:30:12.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.853+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22819c8c0 con 0x7fd228103cf0 2024-03-04T07:30:12.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.854+0000 7fd21ea00700 1 -- 172.21.15.116:0/1743653389 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7fd218029070 con 0x7fd228103cf0 2024-03-04T07:30:12.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.854+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22810a500 con 0x7fd228103cf0 2024-03-04T07:30:12.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.855+0000 7fd21ea00700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd20c038c40 0x7fd20c03b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:12.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.855+0000 7fd21ea00700 1 -- 172.21.15.116:0/1743653389 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd21801e070 con 0x7fd228103cf0 2024-03-04T07:30:12.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.855+0000 7fd225400700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd20c038c40 0x7fd20c03b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:12.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.856+0000 7fd225400700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd20c038c40 0x7fd20c03b100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd214006fd0 tx=0x7fd214006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:12.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:12.860+0000 7fd21ea00700 1 -- 172.21.15.116:0/1743653389 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd218013070 con 0x7fd228103cf0 2024-03-04T07:30:13.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.083+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd22802dac0 con 0x7fd228103cf0 2024-03-04T07:30:13.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.088+0000 7fd21ea00700 1 -- 172.21.15.116:0/1743653389 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd2180300c0 con 0x7fd228103cf0 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd20c038c40 msgr2=0x7fd20c03b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd20c038c40 0x7fd20c03b100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd214006fd0 tx=0x7fd214006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 msgr2=0x7fd228199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228199400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd218005850 tx=0x7fd218005030 comp rx=0 tx=0).stop 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 shutdown_connections 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd20c038c40 0x7fd20c03b100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 --2- 172.21.15.116:0/1743653389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd228103cf0 0x7fd228199400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 >> 172.21.15.116:0/1743653389 conn(0x7fd2280ff290 msgr2=0x7fd2280ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 shutdown_connections 2024-03-04T07:30:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:13.092+0000 7fd22d200700 1 -- 172.21.15.116:0/1743653389 wait complete. 2024-03-04T07:30:13.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:13.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:13.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:30:13.450 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:13.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:30:13.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:13.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1743653389' entity='client.admin' 2024-03-04T07:30:13.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:30:13.451 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:13.872 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-04T07:30:13.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-04T07:30:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:30:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.304+0000 7f49f3400700 1 -- 172.21.15.116:0/3075002968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 msgr2=0x7f49ec104130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.304+0000 7f49f3400700 1 --2- 172.21.15.116:0/3075002968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec104130 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009910 tx=0x7f49e0009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.304+0000 7f49f3400700 1 -- 172.21.15.116:0/3075002968 shutdown_connections 2024-03-04T07:30:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.305+0000 7f49f3400700 1 --2- 172.21.15.116:0/3075002968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec104130 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.305+0000 7f49f3400700 1 -- 172.21.15.116:0/3075002968 >> 172.21.15.116:0/3075002968 conn(0x7f49ec0ff2b0 msgr2=0x7f49ec1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.305+0000 7f49f3400700 1 -- 172.21.15.116:0/3075002968 shutdown_connections 2024-03-04T07:30:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.305+0000 7f49f3400700 1 -- 172.21.15.116:0/3075002968 wait complete. 2024-03-04T07:30:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.306+0000 7f49f3400700 1 Processor -- start 2024-03-04T07:30:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.306+0000 7f49f3400700 1 -- start start 2024-03-04T07:30:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.306+0000 7f49f3400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.306+0000 7f49f3400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec109da0 con 0x7f49ec103d10 2024-03-04T07:30:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.306+0000 7f49f0a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.306+0000 7f49f0a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51254/0 (socket says 172.21.15.116:51254) 2024-03-04T07:30:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.307+0000 7f49f0a00700 1 -- 172.21.15.116:0/2880808235 learned_addr learned my addr 172.21.15.116:0/2880808235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.307+0000 7f49f0a00700 1 -- 172.21.15.116:0/2880808235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec199980 con 0x7f49ec103d10 2024-03-04T07:30:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.307+0000 7f49f0a00700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec199440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005850 tx=0x7f49e0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:16.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.307+0000 7f49e9600700 1 -- 172.21.15.116:0/2880808235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49e0026070 con 0x7f49ec103d10 2024-03-04T07:30:16.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.307+0000 7f49e9600700 1 -- 172.21.15.116:0/2880808235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49e00044a0 con 0x7f49ec103d10 2024-03-04T07:30:16.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.307+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ec199c10 con 0x7f49ec103d10 2024-03-04T07:30:16.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.307+0000 7f49e9600700 1 -- 172.21.15.116:0/2880808235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49e0003ab0 con 0x7f49ec103d10 2024-03-04T07:30:16.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.308+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ec19c900 con 0x7f49ec103d10 2024-03-04T07:30:16.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.309+0000 7f49e9600700 1 -- 172.21.15.116:0/2880808235 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f49e0029070 con 0x7f49ec103d10 2024-03-04T07:30:16.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.309+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ec10a520 con 0x7f49ec103d10 2024-03-04T07:30:16.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.309+0000 7f49e9600700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49d4038c40 0x7f49d403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:16.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.309+0000 7f49e9600700 1 -- 172.21.15.116:0/2880808235 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f49e001e070 con 0x7f49ec103d10 2024-03-04T07:30:16.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.309+0000 7f49ebe00700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49d4038c40 0x7f49d403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:16.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.310+0000 7f49ebe00700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49d4038c40 0x7f49d403b100 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f49d8006fd0 tx=0x7f49d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:16.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.314+0000 7f49e9600700 1 -- 172.21.15.116:0/2880808235 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f49e0013070 con 0x7f49ec103d10 2024-03-04T07:30:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.542+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f49ec02dac0 con 0x7f49d4038c40 2024-03-04T07:30:16.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.550+0000 7f49e9600700 1 -- 172.21.15.116:0/2880808235 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f49ec02dac0 con 0x7f49d4038c40 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49d4038c40 msgr2=0x7f49d403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49d4038c40 0x7f49d403b100 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f49d8006fd0 tx=0x7f49d8006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 msgr2=0x7f49ec199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec199440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005850 tx=0x7f49e0005030 comp rx=0 tx=0).stop 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 shutdown_connections 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49d4038c40 0x7f49d403b100 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 --2- 172.21.15.116:0/2880808235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ec103d10 0x7f49ec199440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 >> 172.21.15.116:0/2880808235 conn(0x7f49ec0ff2b0 msgr2=0x7f49ec0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 shutdown_connections 2024-03-04T07:30:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:16.553+0000 7f49f3400700 1 -- 172.21.15.116:0/2880808235 wait complete. 2024-03-04T07:30:17.353 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi133 2024-03-04T07:30:17.353 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:30:17.353 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/etc/ceph/ceph.conf 2024-03-04T07:30:17.372 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:30:17.372 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:30:17.429 INFO:tasks.cephadm:Adding host smithi133 to orchestrator... 2024-03-04T07:30:17.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch host add smithi133 2024-03-04T07:30:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:17 smithi116 ceph-mon[32150]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:30:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:17 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:17 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:30:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:17 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:30:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:17 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:30:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:17 smithi116 ceph-mon[32150]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T07:30:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:30:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:18 smithi116 ceph-mon[32150]: Updating smithi116:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:30:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:18 smithi116 ceph-mon[32150]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:19.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:19 smithi116 ceph-mon[32150]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:30:19.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:19 smithi116 ceph-mon[32150]: Updating smithi116:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T07:30:19.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:19.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:19.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:20.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.045+0000 7f745ae00700 1 -- 172.21.15.116:0/2872800913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 msgr2=0x7f7454107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:20.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.045+0000 7f745ae00700 1 --2- 172.21.15.116:0/2872800913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454107e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7444009910 tx=0x7f7444009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:20.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.045+0000 7f745ae00700 1 -- 172.21.15.116:0/2872800913 shutdown_connections 2024-03-04T07:30:20.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.045+0000 7f745ae00700 1 --2- 172.21.15.116:0/2872800913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454107e60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:20.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.045+0000 7f745ae00700 1 -- 172.21.15.116:0/2872800913 >> 172.21.15.116:0/2872800913 conn(0x7f74540fd590 msgr2=0x7f74540ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:20.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.045+0000 7f745ae00700 1 -- 172.21.15.116:0/2872800913 shutdown_connections 2024-03-04T07:30:20.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.045+0000 7f745ae00700 1 -- 172.21.15.116:0/2872800913 wait complete. 2024-03-04T07:30:20.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.046+0000 7f745ae00700 1 Processor -- start 2024-03-04T07:30:20.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.047+0000 7f745ae00700 1 -- start start 2024-03-04T07:30:20.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.047+0000 7f745ae00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:20.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.047+0000 7f745ae00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745410c0c0 con 0x7f7454107a80 2024-03-04T07:30:20.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.047+0000 7f7453e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:20.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.047+0000 7f7453e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51268/0 (socket says 172.21.15.116:51268) 2024-03-04T07:30:20.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.047+0000 7f7453e00700 1 -- 172.21.15.116:0/815754873 learned_addr learned my addr 172.21.15.116:0/815754873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:20.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.048+0000 7f7453e00700 1 -- 172.21.15.116:0/815754873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454197b50 con 0x7f7454107a80 2024-03-04T07:30:20.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.048+0000 7f7453e00700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454197610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7444005850 tx=0x7f7444005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:20.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.048+0000 7f7450c00700 1 -- 172.21.15.116:0/815754873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7444026070 con 0x7f7454107a80 2024-03-04T07:30:20.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.048+0000 7f7450c00700 1 -- 172.21.15.116:0/815754873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f744400bbd0 con 0x7f7454107a80 2024-03-04T07:30:20.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.048+0000 7f7450c00700 1 -- 172.21.15.116:0/815754873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7444003970 con 0x7f7454107a80 2024-03-04T07:30:20.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.048+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7454197e40 con 0x7f7454107a80 2024-03-04T07:30:20.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.048+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745419bee0 con 0x7f7454107a80 2024-03-04T07:30:20.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.049+0000 7f7450c00700 1 -- 172.21.15.116:0/815754873 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f7444029070 con 0x7f7454107a80 2024-03-04T07:30:20.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.049+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74541031c0 con 0x7f7454107a80 2024-03-04T07:30:20.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.050+0000 7f7450c00700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7438038ca0 0x7f743803b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:20.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.050+0000 7f7450c00700 1 -- 172.21.15.116:0/815754873 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f744401e070 con 0x7f7454107a80 2024-03-04T07:30:20.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.050+0000 7f7453400700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7438038ca0 0x7f743803b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:20.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.051+0000 7f7453400700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7438038ca0 0x7f743803b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7440006fd0 tx=0x7f7440006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:20.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.055+0000 7f7450c00700 1 -- 172.21.15.116:0/815754873 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7444013070 con 0x7f7454107a80 2024-03-04T07:30:20.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:20.279+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi133", "target": ["mon-mgr", ""]}) v1 -- 0x7f7454198970 con 0x7f7438038ca0 2024-03-04T07:30:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:20 smithi116 ceph-mon[32150]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:22.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:21 smithi116 ceph-mon[32150]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi133", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:30:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:22 smithi116 ceph-mon[32150]: Deploying cephadm binary to smithi133 2024-03-04T07:30:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:22 smithi116 ceph-mon[32150]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.917+0000 7f7450c00700 1 -- 172.21.15.116:0/815754873 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7454198970 con 0x7f7438038ca0 2024-03-04T07:30:24.917 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi133' with addr '172.21.15.133' 2024-03-04T07:30:24.918 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:24 smithi116 ceph-mon[32150]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.920+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7438038ca0 msgr2=0x7f743803b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7438038ca0 0x7f743803b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7440006fd0 tx=0x7f7440006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 msgr2=0x7f7454197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454197610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7444005850 tx=0x7f7444005030 comp rx=0 tx=0).stop 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 shutdown_connections 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7438038ca0 0x7f743803b160 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 --2- 172.21.15.116:0/815754873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7454107a80 0x7f7454197610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 >> 172.21.15.116:0/815754873 conn(0x7f74540fd590 msgr2=0x7f7454109670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 shutdown_connections 2024-03-04T07:30:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:24.921+0000 7f745ae00700 1 -- 172.21.15.116:0/815754873 wait complete. 2024-03-04T07:30:25.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T07:30:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:25 smithi116 ceph-mon[32150]: Added host smithi133 2024-03-04T07:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:25 smithi116 ceph-mon[32150]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:27.369 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:27 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:28.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:28 smithi116 ceph-mon[32150]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:28.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.338+0000 7f04eba00700 1 -- 172.21.15.116:0/2619228832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 msgr2=0x7f04e4107e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.338+0000 7f04eba00700 1 --2- 172.21.15.116:0/2619228832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e4107e50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009910 tx=0x7f04d8009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.338+0000 7f04eba00700 1 -- 172.21.15.116:0/2619228832 shutdown_connections 2024-03-04T07:30:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.338+0000 7f04eba00700 1 --2- 172.21.15.116:0/2619228832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e4107e50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.338+0000 7f04eba00700 1 -- 172.21.15.116:0/2619228832 >> 172.21.15.116:0/2619228832 conn(0x7f04e40fd5a0 msgr2=0x7f04e40ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.338+0000 7f04eba00700 1 -- 172.21.15.116:0/2619228832 shutdown_connections 2024-03-04T07:30:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.338+0000 7f04eba00700 1 -- 172.21.15.116:0/2619228832 wait complete. 2024-03-04T07:30:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.339+0000 7f04eba00700 1 Processor -- start 2024-03-04T07:30:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.339+0000 7f04eba00700 1 -- start start 2024-03-04T07:30:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.340+0000 7f04eba00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e419f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.340+0000 7f04eba00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e410c0b0 con 0x7f04e4107a70 2024-03-04T07:30:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.340+0000 7f04e9000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e419f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.340+0000 7f04e9000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e419f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43938/0 (socket says 172.21.15.116:43938) 2024-03-04T07:30:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.340+0000 7f04e9000700 1 -- 172.21.15.116:0/2612721598 learned_addr learned my addr 172.21.15.116:0/2612721598 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.340+0000 7f04e9000700 1 -- 172.21.15.116:0/2612721598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e419f920 con 0x7f04e4107a70 2024-03-04T07:30:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.341+0000 7f04e9000700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e419f3e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005850 tx=0x7f04d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.341+0000 7f04e1600700 1 -- 172.21.15.116:0/2612721598 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04d8026070 con 0x7f04e4107a70 2024-03-04T07:30:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.341+0000 7f04e1600700 1 -- 172.21.15.116:0/2612721598 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04d800bbd0 con 0x7f04e4107a70 2024-03-04T07:30:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.341+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e419fc10 con 0x7f04e4107a70 2024-03-04T07:30:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.341+0000 7f04e1600700 1 -- 172.21.15.116:0/2612721598 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04d8003970 con 0x7f04e4107a70 2024-03-04T07:30:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.341+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e41995a0 con 0x7f04e4107a70 2024-03-04T07:30:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.342+0000 7f04e1600700 1 -- 172.21.15.116:0/2612721598 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f04d8029070 con 0x7f04e4107a70 2024-03-04T07:30:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.343+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e41031b0 con 0x7f04e4107a70 2024-03-04T07:30:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.343+0000 7f04e1600700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04cc038ca0 0x7f04cc03b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.343+0000 7f04e1600700 1 -- 172.21.15.116:0/2612721598 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f04d801e070 con 0x7f04e4107a70 2024-03-04T07:30:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.343+0000 7f04e3e00700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04cc038ca0 0x7f04cc03b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.344+0000 7f04e3e00700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04cc038ca0 0x7f04cc03b160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04d0006fd0 tx=0x7f04d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.349+0000 7f04e1600700 1 -- 172.21.15.116:0/2612721598 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f04d8013070 con 0x7f04e4107a70 2024-03-04T07:30:28.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.578+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04e419aad0 con 0x7f04cc038ca0 2024-03-04T07:30:28.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.580+0000 7f04e1600700 1 -- 172.21.15.116:0/2612721598 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f04e419aad0 con 0x7f04cc038ca0 2024-03-04T07:30:28.580 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:28.580 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.133", "hostname": "smithi133", "labels": [], "status": ""}] 2024-03-04T07:30:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.583+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04cc038ca0 msgr2=0x7f04cc03b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.583+0000 7f04eba00700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04cc038ca0 0x7f04cc03b160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04d0006fd0 tx=0x7f04d0006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.583+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 msgr2=0x7f04e419f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:28.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.583+0000 7f04eba00700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e419f3e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005850 tx=0x7f04d8005030 comp rx=0 tx=0).stop 2024-03-04T07:30:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.584+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 shutdown_connections 2024-03-04T07:30:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.584+0000 7f04eba00700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04cc038ca0 0x7f04cc03b160 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.584+0000 7f04eba00700 1 --2- 172.21.15.116:0/2612721598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04e4107a70 0x7f04e419f3e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.584+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 >> 172.21.15.116:0/2612721598 conn(0x7f04e40fd5a0 msgr2=0x7f04e4075da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.584+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 shutdown_connections 2024-03-04T07:30:28.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:28.584+0000 7f04eba00700 1 -- 172.21.15.116:0/2612721598 wait complete. 2024-03-04T07:30:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:29 smithi116 ceph-mon[32150]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:30:29.341 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-03-04T07:30:29.341 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:30:29.341 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-03-04T07:30:29.360 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:30:29.360 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:30:29.417 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-03-04T07:30:29.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch host add smithi152 2024-03-04T07:30:29.831 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:30:30.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:30 smithi116 ceph-mon[32150]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.884+0000 7f20b3e00700 1 -- 172.21.15.116:0/515910537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 msgr2=0x7f20ac107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.884+0000 7f20b3e00700 1 --2- 172.21.15.116:0/515910537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac107e60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20a000b0a0 tx=0x7f20a000b3b0 comp rx=0 tx=0).stop 2024-03-04T07:30:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.884+0000 7f20b3e00700 1 -- 172.21.15.116:0/515910537 shutdown_connections 2024-03-04T07:30:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.884+0000 7f20b3e00700 1 --2- 172.21.15.116:0/515910537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac107e60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:31.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.884+0000 7f20b3e00700 1 -- 172.21.15.116:0/515910537 >> 172.21.15.116:0/515910537 conn(0x7f20ac0fd590 msgr2=0x7f20ac0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.884+0000 7f20b3e00700 1 -- 172.21.15.116:0/515910537 shutdown_connections 2024-03-04T07:30:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.884+0000 7f20b3e00700 1 -- 172.21.15.116:0/515910537 wait complete. 2024-03-04T07:30:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.886+0000 7f20b3e00700 1 Processor -- start 2024-03-04T07:30:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.886+0000 7f20b3e00700 1 -- start start 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.886+0000 7f20b3e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.886+0000 7f20b3e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac10c0c0 con 0x7f20ac107a80 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.886+0000 7f20b1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.886+0000 7f20b1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43950/0 (socket says 172.21.15.116:43950) 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.886+0000 7f20b1400700 1 -- 172.21.15.116:0/141043 learned_addr learned my addr 172.21.15.116:0/141043 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.887+0000 7f20b1400700 1 -- 172.21.15.116:0/141043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac197a60 con 0x7f20ac107a80 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.887+0000 7f20b1400700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac197520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20a0015040 tx=0x7f20a0003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.887+0000 7f20aa000700 1 -- 172.21.15.116:0/141043 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20a0017040 con 0x7f20ac107a80 2024-03-04T07:30:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.887+0000 7f20aa000700 1 -- 172.21.15.116:0/141043 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20a0022070 con 0x7f20ac107a80 2024-03-04T07:30:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.887+0000 7f20aa000700 1 -- 172.21.15.116:0/141043 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20a001d5d0 con 0x7f20ac107a80 2024-03-04T07:30:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.887+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac197cf0 con 0x7f20ac107a80 2024-03-04T07:30:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.888+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac19bdf0 con 0x7f20ac107a80 2024-03-04T07:30:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.889+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac1031c0 con 0x7f20ac107a80 2024-03-04T07:30:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.889+0000 7f20aa000700 1 -- 172.21.15.116:0/141043 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f20a0004000 con 0x7f20ac107a80 2024-03-04T07:30:31.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.890+0000 7f20aa000700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2094038ca0 0x7f209403b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:31.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.890+0000 7f20aa000700 1 -- 172.21.15.116:0/141043 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f20a000f050 con 0x7f20ac107a80 2024-03-04T07:30:31.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.890+0000 7f20b0a00700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2094038ca0 0x7f209403b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:31.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.891+0000 7f20b0a00700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2094038ca0 0x7f209403b160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2098006fd0 tx=0x7f2098006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:31.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:31.896+0000 7f20aa000700 1 -- 172.21.15.116:0/141043 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f20a0007e20 con 0x7f20ac107a80 2024-03-04T07:30:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:32.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi133", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:30:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:30:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:30:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: Updating smithi133:/etc/ceph/ceph.conf 2024-03-04T07:30:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: Updating smithi133:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:30:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:32.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:31 smithi116 ceph-mon[32150]: Updating smithi133:/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:30:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:32.128+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7f20ac0636f0 con 0x7f2094038ca0 2024-03-04T07:30:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:32 smithi116 ceph-mon[32150]: Updating smithi133:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T07:30:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:32 smithi116 ceph-mon[32150]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:30:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:33 smithi116 ceph-mon[32150]: Deploying cephadm binary to smithi152 2024-03-04T07:30:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:33 smithi116 ceph-mon[32150]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:36.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:36 smithi116 ceph-mon[32150]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:36.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.822+0000 7f20aa000700 1 -- 172.21.15.116:0/141043 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f20ac0636f0 con 0x7f2094038ca0 2024-03-04T07:30:36.822 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2094038ca0 msgr2=0x7f209403b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2094038ca0 0x7f209403b160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2098006fd0 tx=0x7f2098006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 msgr2=0x7f20ac197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac197520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20a0015040 tx=0x7f20a0003d50 comp rx=0 tx=0).stop 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 shutdown_connections 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2094038ca0 0x7f209403b160 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 --2- 172.21.15.116:0/141043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20ac107a80 0x7f20ac197520 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:36.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.825+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 >> 172.21.15.116:0/141043 conn(0x7f20ac0fd590 msgr2=0x7f20ac1095e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:36.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.826+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 shutdown_connections 2024-03-04T07:30:36.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:36.826+0000 7f20b3e00700 1 -- 172.21.15.116:0/141043 wait complete. 2024-03-04T07:30:37.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T07:30:37.989 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:30:38.139 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:38.139 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:37 smithi116 ceph-mon[32150]: Added host smithi152 2024-03-04T07:30:38.139 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:30:38.140 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:37 smithi116 ceph-mon[32150]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:38.140 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:39.201 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:40.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.083+0000 7fe467a00700 1 -- 172.21.15.116:0/2052456771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 msgr2=0x7fe4601042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:40.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.083+0000 7fe467a00700 1 --2- 172.21.15.116:0/2052456771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe4601042c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe454009910 tx=0x7fe454009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:40.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.084+0000 7fe467a00700 1 -- 172.21.15.116:0/2052456771 shutdown_connections 2024-03-04T07:30:40.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.084+0000 7fe467a00700 1 --2- 172.21.15.116:0/2052456771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe4601042c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:40.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.084+0000 7fe467a00700 1 -- 172.21.15.116:0/2052456771 >> 172.21.15.116:0/2052456771 conn(0x7fe4600ff7b0 msgr2=0x7fe460101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:40.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.084+0000 7fe467a00700 1 -- 172.21.15.116:0/2052456771 shutdown_connections 2024-03-04T07:30:40.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.084+0000 7fe467a00700 1 -- 172.21.15.116:0/2052456771 wait complete. 2024-03-04T07:30:40.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.085+0000 7fe467a00700 1 Processor -- start 2024-03-04T07:30:40.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.085+0000 7fe467a00700 1 -- start start 2024-03-04T07:30:40.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.085+0000 7fe467a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe46019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:40.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.085+0000 7fe467a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46010e2b0 con 0x7fe460103ee0 2024-03-04T07:30:40.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.085+0000 7fe465000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe46019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:40.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.085+0000 7fe465000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe46019f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55906/0 (socket says 172.21.15.116:55906) 2024-03-04T07:30:40.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.086+0000 7fe465000700 1 -- 172.21.15.116:0/3169721435 learned_addr learned my addr 172.21.15.116:0/3169721435 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:40.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.086+0000 7fe465000700 1 -- 172.21.15.116:0/3169721435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46019f970 con 0x7fe460103ee0 2024-03-04T07:30:40.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.086+0000 7fe465000700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe46019f430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe454005850 tx=0x7fe454005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:40.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.086+0000 7fe45d600700 1 -- 172.21.15.116:0/3169721435 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe454026070 con 0x7fe460103ee0 2024-03-04T07:30:40.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.086+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46019fc60 con 0x7fe460103ee0 2024-03-04T07:30:40.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.086+0000 7fe45d600700 1 -- 172.21.15.116:0/3169721435 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe45400bbd0 con 0x7fe460103ee0 2024-03-04T07:30:40.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.086+0000 7fe45d600700 1 -- 172.21.15.116:0/3169721435 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe454003970 con 0x7fe460103ee0 2024-03-04T07:30:40.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.087+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4601995f0 con 0x7fe460103ee0 2024-03-04T07:30:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.088+0000 7fe45d600700 1 -- 172.21.15.116:0/3169721435 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7fe454029070 con 0x7fe460103ee0 2024-03-04T07:30:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.088+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4601053c0 con 0x7fe460103ee0 2024-03-04T07:30:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.088+0000 7fe45d600700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe448038c40 0x7fe44803b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.088+0000 7fe45d600700 1 -- 172.21.15.116:0/3169721435 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe45401e070 con 0x7fe460103ee0 2024-03-04T07:30:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.088+0000 7fe45fe00700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe448038c40 0x7fe44803b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.089+0000 7fe45fe00700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe448038c40 0x7fe44803b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe44c006fd0 tx=0x7fe44c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.094+0000 7fe45d600700 1 -- 172.21.15.116:0/3169721435 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe454013070 con 0x7fe460103ee0 2024-03-04T07:30:40.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.323+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe46019ab20 con 0x7fe448038c40 2024-03-04T07:30:40.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.324+0000 7fe45d600700 1 -- 172.21.15.116:0/3169721435 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7fe46019ab20 con 0x7fe448038c40 2024-03-04T07:30:40.324 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:30:40.324 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.133", "hostname": "smithi133", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe448038c40 msgr2=0x7fe44803b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe448038c40 0x7fe44803b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe44c006fd0 tx=0x7fe44c006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 msgr2=0x7fe46019f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe46019f430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe454005850 tx=0x7fe454005030 comp rx=0 tx=0).stop 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 shutdown_connections 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe448038c40 0x7fe44803b100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 --2- 172.21.15.116:0/3169721435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe460103ee0 0x7fe46019f430 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 >> 172.21.15.116:0/3169721435 conn(0x7fe4600ff7b0 msgr2=0x7fe4601058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:40.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 shutdown_connections 2024-03-04T07:30:40.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:40.327+0000 7fe467a00700 1 -- 172.21.15.116:0/3169721435 wait complete. 2024-03-04T07:30:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:40 smithi116 ceph-mon[32150]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:41.135 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-04T07:30:41.136 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-04T07:30:41.570 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:30:41.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:41 smithi116 ceph-mon[32150]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:30:42.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:42 smithi116 ceph-mon[32150]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:43.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:43.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:43.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:43.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:43.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:30:43.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:30:43.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.768+0000 7f9a36400700 1 -- 172.21.15.116:0/2495645208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 msgr2=0x7f9a301042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.768+0000 7f9a36400700 1 --2- 172.21.15.116:0/2495645208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a301042c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009910 tx=0x7f9a20009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.768+0000 7f9a36400700 1 -- 172.21.15.116:0/2495645208 shutdown_connections 2024-03-04T07:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.768+0000 7f9a36400700 1 --2- 172.21.15.116:0/2495645208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a301042c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.768+0000 7f9a36400700 1 -- 172.21.15.116:0/2495645208 >> 172.21.15.116:0/2495645208 conn(0x7f9a300ff7b0 msgr2=0x7f9a30101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.769+0000 7f9a36400700 1 -- 172.21.15.116:0/2495645208 shutdown_connections 2024-03-04T07:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.769+0000 7f9a36400700 1 -- 172.21.15.116:0/2495645208 wait complete. 2024-03-04T07:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.770+0000 7f9a36400700 1 Processor -- start 2024-03-04T07:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.770+0000 7f9a36400700 1 -- start start 2024-03-04T07:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.770+0000 7f9a36400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a3019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.770+0000 7f9a36400700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3010e2b0 con 0x7f9a30103ee0 2024-03-04T07:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.771+0000 7f9a2f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a3019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.771+0000 7f9a2f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a3019f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55932/0 (socket says 172.21.15.116:55932) 2024-03-04T07:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.771+0000 7f9a2f400700 1 -- 172.21.15.116:0/2070432100 learned_addr learned my addr 172.21.15.116:0/2070432100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.771+0000 7f9a2f400700 1 -- 172.21.15.116:0/2070432100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3019fae0 con 0x7f9a30103ee0 2024-03-04T07:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.772+0000 7f9a2f400700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a3019f5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9a20006010 tx=0x7f9a20005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.772+0000 7f9a27e00700 1 -- 172.21.15.116:0/2070432100 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a20026070 con 0x7f9a30103ee0 2024-03-04T07:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.772+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3019fdd0 con 0x7f9a30103ee0 2024-03-04T07:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.772+0000 7f9a27e00700 1 -- 172.21.15.116:0/2070432100 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a2000bbd0 con 0x7f9a30103ee0 2024-03-04T07:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.772+0000 7f9a27e00700 1 -- 172.21.15.116:0/2070432100 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a20003910 con 0x7f9a30103ee0 2024-03-04T07:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.772+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a30199840 con 0x7f9a30103ee0 2024-03-04T07:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.773+0000 7f9a27e00700 1 -- 172.21.15.116:0/2070432100 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f9a20029070 con 0x7f9a30103ee0 2024-03-04T07:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.773+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a301053c0 con 0x7f9a30103ee0 2024-03-04T07:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.774+0000 7f9a27e00700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a14038c40 0x7f9a1403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.774+0000 7f9a27e00700 1 -- 172.21.15.116:0/2070432100 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9a2001e070 con 0x7f9a30103ee0 2024-03-04T07:30:43.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.774+0000 7f9a2ea00700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a14038c40 0x7f9a1403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:43.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.775+0000 7f9a2ea00700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a14038c40 0x7f9a1403b100 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c006fd0 tx=0x7f9a1c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:43.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:43.780+0000 7f9a27e00700 1 -- 172.21.15.116:0/2070432100 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9a20013070 con 0x7f9a30103ee0 2024-03-04T07:30:44.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.010+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9a3019aba0 con 0x7f9a30103ee0 2024-03-04T07:30:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.898+0000 7f9a27e00700 1 -- 172.21.15.116:0/2070432100 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9a2000fe20 con 0x7f9a30103ee0 2024-03-04T07:30:44.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a14038c40 msgr2=0x7f9a1403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:44.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a14038c40 0x7f9a1403b100 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c006fd0 tx=0x7f9a1c006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 msgr2=0x7f9a3019f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a3019f5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9a20006010 tx=0x7f9a20005030 comp rx=0 tx=0).stop 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 shutdown_connections 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a14038c40 0x7f9a1403b100 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 --2- 172.21.15.116:0/2070432100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a30103ee0 0x7f9a3019f5a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.902+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 >> 172.21.15.116:0/2070432100 conn(0x7f9a300ff7b0 msgr2=0x7f9a30106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.903+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 shutdown_connections 2024-03-04T07:30:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:30:44.903+0000 7f9a36400700 1 -- 172.21.15.116:0/2070432100 wait complete. 2024-03-04T07:30:44.904 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: Updating smithi152:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: Updating smithi152:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:45.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:44 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2070432100' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T07:30:45.674 INFO:tasks.cephadm:Adding mon.a on smithi116 2024-03-04T07:30:45.675 INFO:tasks.cephadm:Adding mon.b on smithi133 2024-03-04T07:30:45.675 INFO:tasks.cephadm:Adding mon.c on smithi152 2024-03-04T07:30:45.675 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi116:172.21.15.116=a;smithi133:172.21.15.133=b;smithi152:172.21.15.152=c' 2024-03-04T07:30:45.917 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:30:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:45 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2070432100' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T07:30:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:45 smithi116 ceph-mon[32150]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:30:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:45 smithi116 ceph-mon[32150]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:47.311 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.311+0000 7fd1ccb31700 1 -- 172.21.15.152:0/3839145261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 msgr2=0x7fd1c8100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:47.312 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.311+0000 7fd1ccb31700 1 --2- 172.21.15.152:0/3839145261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c8100740 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009910 tx=0x7fd1b8009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:47.312 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.311+0000 7fd1ccb31700 1 -- 172.21.15.152:0/3839145261 shutdown_connections 2024-03-04T07:30:47.312 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.311+0000 7fd1ccb31700 1 --2- 172.21.15.152:0/3839145261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c8100740 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:47.312 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.311+0000 7fd1ccb31700 1 -- 172.21.15.152:0/3839145261 >> 172.21.15.152:0/3839145261 conn(0x7fd1c80fbbf0 msgr2=0x7fd1c80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:47.312 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.311+0000 7fd1ccb31700 1 -- 172.21.15.152:0/3839145261 shutdown_connections 2024-03-04T07:30:47.312 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.311+0000 7fd1ccb31700 1 -- 172.21.15.152:0/3839145261 wait complete. 2024-03-04T07:30:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.312+0000 7fd1ccb31700 1 Processor -- start 2024-03-04T07:30:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.312+0000 7fd1ccb31700 1 -- start start 2024-03-04T07:30:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.312+0000 7fd1ccb31700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.312+0000 7fd1ccb31700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c810afc0 con 0x7fd1c8100360 2024-03-04T07:30:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.313+0000 7fd1c659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.313+0000 7fd1c659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c81983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:52660/0 (socket says 172.21.15.152:52660) 2024-03-04T07:30:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.313+0000 7fd1c659c700 1 -- 172.21.15.152:0/1860905734 learned_addr learned my addr 172.21.15.152:0/1860905734 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:30:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.314+0000 7fd1c659c700 1 -- 172.21.15.152:0/1860905734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c8198930 con 0x7fd1c8100360 2024-03-04T07:30:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.314+0000 7fd1c659c700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c81983f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005850 tx=0x7fd1b8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.314+0000 7fd1b77fe700 1 -- 172.21.15.152:0/1860905734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1b8026070 con 0x7fd1c8100360 2024-03-04T07:30:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.314+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c8198c20 con 0x7fd1c8100360 2024-03-04T07:30:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.314+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c81990e0 con 0x7fd1c8100360 2024-03-04T07:30:47.315 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.314+0000 7fd1b77fe700 1 -- 172.21.15.152:0/1860905734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1b800bbd0 con 0x7fd1c8100360 2024-03-04T07:30:47.315 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.314+0000 7fd1b77fe700 1 -- 172.21.15.152:0/1860905734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1b80039b0 con 0x7fd1c8100360 2024-03-04T07:30:47.315 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.315+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c8101840 con 0x7fd1c8100360 2024-03-04T07:30:47.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.316+0000 7fd1b77fe700 1 -- 172.21.15.152:0/1860905734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7fd1b8029070 con 0x7fd1c8100360 2024-03-04T07:30:47.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.316+0000 7fd1b77fe700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1b0038c40 0x7fd1b003b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:47.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.316+0000 7fd1b77fe700 1 -- 172.21.15.152:0/1860905734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd1b801e070 con 0x7fd1c8100360 2024-03-04T07:30:47.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.317+0000 7fd1c5d9b700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1b0038c40 0x7fd1b003b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:47.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.317+0000 7fd1c5d9b700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1b0038c40 0x7fd1b003b100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc006fd0 tx=0x7fd1bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:47.331 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.331+0000 7fd1b77fe700 1 -- 172.21.15.152:0/1860905734 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd1b8013070 con 0x7fd1c8100360 2024-03-04T07:30:47.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.536+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi133:172.21.15.133=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1c8109600 con 0x7fd1b0038c40 2024-03-04T07:30:47.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.552+0000 7fd1b77fe700 1 -- 172.21.15.152:0/1860905734 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd1c8109600 con 0x7fd1b0038c40 2024-03-04T07:30:47.552 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mon update... 2024-03-04T07:30:47.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1b0038c40 msgr2=0x7fd1b003b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:47.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1b0038c40 0x7fd1b003b100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc006fd0 tx=0x7fd1bc006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:47.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 msgr2=0x7fd1c81983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:47.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c81983f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005850 tx=0x7fd1b8005030 comp rx=0 tx=0).stop 2024-03-04T07:30:47.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 shutdown_connections 2024-03-04T07:30:47.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1b0038c40 0x7fd1b003b100 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:47.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 --2- 172.21.15.152:0/1860905734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1c8100360 0x7fd1c81983f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:47.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 >> 172.21.15.152:0/1860905734 conn(0x7fd1c80fbbf0 msgr2=0x7fd1c8103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:47.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 shutdown_connections 2024-03-04T07:30:47.555 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:47.554+0000 7fd1ccb31700 1 -- 172.21.15.152:0/1860905734 wait complete. 2024-03-04T07:30:48.114 DEBUG:teuthology.orchestra.run.smithi133:mon.b> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.b.service 2024-03-04T07:30:48.117 DEBUG:teuthology.orchestra.run.smithi152:mon.c> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.c.service 2024-03-04T07:30:48.119 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T07:30:48.119 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T07:30:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi133:172.21.15.133=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:30:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi133:172.21.15.133=b;smithi152:172.21.15.152=c;count:3 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:30:49.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:48 smithi116 ceph-mon[32150]: Deploying daemon mon.c on smithi152 2024-03-04T07:30:49.095 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:30:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:50 smithi116 ceph-mon[32150]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.696+0000 7f3841276700 1 -- 172.21.15.152:0/1116167374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 msgr2=0x7f383c107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.696+0000 7f3841276700 1 --2- 172.21.15.152:0/1116167374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c107e70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f382c009910 tx=0x7f382c009c20 comp rx=0 tx=0).stop 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.696+0000 7f3841276700 1 -- 172.21.15.152:0/1116167374 shutdown_connections 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.696+0000 7f3841276700 1 --2- 172.21.15.152:0/1116167374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c107e70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.696+0000 7f3841276700 1 -- 172.21.15.152:0/1116167374 >> 172.21.15.152:0/1116167374 conn(0x7f383c0fd5a0 msgr2=0x7f383c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.696+0000 7f3841276700 1 -- 172.21.15.152:0/1116167374 shutdown_connections 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.696+0000 7f3841276700 1 -- 172.21.15.152:0/1116167374 wait complete. 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.697+0000 7f3841276700 1 Processor -- start 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.697+0000 7f3841276700 1 -- start start 2024-03-04T07:30:51.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.697+0000 7f3841276700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:51.699 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.697+0000 7f3841276700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c10c0d0 con 0x7f383c107a90 2024-03-04T07:30:51.699 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.698+0000 7f383affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:51.699 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.698+0000 7f383affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:35306/0 (socket says 172.21.15.152:35306) 2024-03-04T07:30:51.699 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.698+0000 7f383affd700 1 -- 172.21.15.152:0/2121795863 learned_addr learned my addr 172.21.15.152:0/2121795863 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:30:51.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.698+0000 7f383affd700 1 -- 172.21.15.152:0/2121795863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c197ad0 con 0x7f383c107a90 2024-03-04T07:30:51.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.699+0000 7f383affd700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c197590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f382c006010 tx=0x7f382c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:51.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.699+0000 7f3823fff700 1 -- 172.21.15.152:0/2121795863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f382c026070 con 0x7f383c107a90 2024-03-04T07:30:51.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.699+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c197dc0 con 0x7f383c107a90 2024-03-04T07:30:51.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.699+0000 7f3823fff700 1 -- 172.21.15.152:0/2121795863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f382c00bbd0 con 0x7f383c107a90 2024-03-04T07:30:51.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.699+0000 7f3823fff700 1 -- 172.21.15.152:0/2121795863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f382c003910 con 0x7f383c107a90 2024-03-04T07:30:51.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.699+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c19be60 con 0x7f383c107a90 2024-03-04T07:30:51.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.700+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c1031d0 con 0x7f383c107a90 2024-03-04T07:30:51.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.700+0000 7f3823fff700 1 -- 172.21.15.152:0/2121795863 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f382c029070 con 0x7f383c107a90 2024-03-04T07:30:51.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.701+0000 7f3823fff700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3824038c40 0x7f382403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:51.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.701+0000 7f3823fff700 1 -- 172.21.15.152:0/2121795863 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f382c01e070 con 0x7f383c107a90 2024-03-04T07:30:51.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.701+0000 7f383a7fc700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3824038c40 0x7f382403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:51.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.702+0000 7f383a7fc700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3824038c40 0x7f382403b100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3830006fd0 tx=0x7f3830006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:51.715 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.716+0000 7f3823fff700 1 -- 172.21.15.152:0/2121795863 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f382c013070 con 0x7f383c107a90 2024-03-04T07:30:51.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.995+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f383c1988f0 con 0x7f383c107a90 2024-03-04T07:30:51.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.996+0000 7f3823fff700 1 -- 172.21.15.152:0/2121795863 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7f382c030030 con 0x7f383c107a90 2024-03-04T07:30:51.996 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:30:51.996 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","modified":"2024-03-04T07:28:44.925082Z","created":"2024-03-04T07:28:44.925082Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T07:30:51.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3824038c40 msgr2=0x7f382403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:51.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3824038c40 0x7f382403b100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3830006fd0 tx=0x7f3830006e40 comp rx=0 tx=0).stop 2024-03-04T07:30:51.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 msgr2=0x7f383c197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:51.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c197590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f382c006010 tx=0x7f382c005030 comp rx=0 tx=0).stop 2024-03-04T07:30:51.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 shutdown_connections 2024-03-04T07:30:51.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3824038c40 0x7f382403b100 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:51.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 --2- 172.21.15.152:0/2121795863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f383c107a90 0x7f383c197590 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:51.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 >> 172.21.15.152:0/2121795863 conn(0x7f383c0fd5a0 msgr2=0x7f383c0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:51.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 shutdown_connections 2024-03-04T07:30:51.999 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:51.998+0000 7f3841276700 1 -- 172.21.15.152:0/2121795863 wait complete. 2024-03-04T07:30:51.999 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-04T07:30:53.747 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T07:30:53.747 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T07:30:53.994 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:30:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:30:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: mon.a calling monitor election 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: mon.c calling monitor election 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T07:30:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: monmap e2: 2 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:30:57.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: fsmap 2024-03-04T07:30:57.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:30:57.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: mgrmap e13: x(active, since 59s) 2024-03-04T07:30:57.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: overall HEALTH_OK 2024-03-04T07:30:57.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:57.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:57.520 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:58.469 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.469+0000 7fda50a3e700 1 -- 172.21.15.152:0/764134891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda340035c0 msgr2=0x7fda34005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:58.469 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.469+0000 7fda50a3e700 1 --2- 172.21.15.152:0/764134891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda340035c0 0x7fda34005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda4c1009f0 tx=0x7fda4000feb0 comp rx=0 tx=0).stop 2024-03-04T07:30:58.470 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.469+0000 7fda50a3e700 1 -- 172.21.15.152:0/764134891 shutdown_connections 2024-03-04T07:30:58.470 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.469+0000 7fda50a3e700 1 --2- 172.21.15.152:0/764134891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda340035c0 0x7fda34005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:58.470 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.469+0000 7fda50a3e700 1 -- 172.21.15.152:0/764134891 >> 172.21.15.152:0/764134891 conn(0x7fda4c0fc5d0 msgr2=0x7fda4c0fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:58.470 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.469+0000 7fda50a3e700 1 -- 172.21.15.152:0/764134891 shutdown_connections 2024-03-04T07:30:58.470 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.469+0000 7fda50a3e700 1 -- 172.21.15.152:0/764134891 wait complete. 2024-03-04T07:30:58.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:58.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:30:58.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:30:58.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:58 smithi152 ceph-mon[37154]: Deploying daemon mon.b on smithi133 2024-03-04T07:30:58.471 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:58 smithi152 ceph-mon[37154]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:58.471 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:58.473 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.470+0000 7fda50a3e700 1 Processor -- start 2024-03-04T07:30:58.473 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.470+0000 7fda50a3e700 1 -- start start 2024-03-04T07:30:58.473 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.470+0000 7fda50a3e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 0x7fda4c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:58.473 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.470+0000 7fda50a3e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda4c19d950 0x7fda4c1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:58.473 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.470+0000 7fda50a3e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c108b60 con 0x7fda4c19d950 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.470+0000 7fda50a3e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c1089e0 con 0x7fda340035c0 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.470+0000 7fda4a59c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 0x7fda4c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda4a59c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 0x7fda4c19d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:56994/0 (socket says 172.21.15.152:56994) 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda4a59c700 1 -- 172.21.15.152:0/3729094590 learned_addr learned my addr 172.21.15.152:0/3729094590 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda4a59c700 1 -- 172.21.15.152:0/3729094590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 msgr2=0x7fda4c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda4a59c700 1 -- 172.21.15.152:0/3729094590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 msgr2=0x7fda4c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda4a59c700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 0x7fda4c19d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda4a59c700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 0x7fda4c19d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda49d9b700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda4c19d950 0x7fda4c1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda49d9b700 1 -- 172.21.15.152:0/3729094590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 msgr2=0x7fda4c19d410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda49d9b700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 0x7fda4c19d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:58.474 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.471+0000 7fda49d9b700 1 -- 172.21.15.152:0/3729094590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c19df90 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.472+0000 7fda49d9b700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda4c19d950 0x7fda4c1975d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00b4d0 tx=0x7fda3c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.472+0000 7fda3b7fe700 1 -- 172.21.15.152:0/3729094590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fda3c011630 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.472+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c197b10 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.472+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c1980a0 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.472+0000 7fda3b7fe700 1 -- 172.21.15.152:0/3729094590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda3c011cb0 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.472+0000 7fda3b7fe700 1 -- 172.21.15.152:0/3729094590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fda3c0193b0 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.473+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c102200 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.473+0000 7fda3b7fe700 1 -- 172.21.15.152:0/3729094590 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7fda3c018020 con 0x7fda4c19d950 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.473+0000 7fda3b7fe700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fda2c0389e0 0x7fda2c03aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:30:58.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.473+0000 7fda3b7fe700 1 -- 172.21.15.152:0/3729094590 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fda3c04b660 con 0x7fda4c19d950 2024-03-04T07:30:58.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.474+0000 7fda4a59c700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fda2c0389e0 0x7fda2c03aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:30:58.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.475+0000 7fda4a59c700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fda2c0389e0 0x7fda2c03aea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda4000fa60 tx=0x7fda40005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:30:58.488 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.488+0000 7fda3b7fe700 1 -- 172.21.15.152:0/3729094590 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fda3c01f020 con 0x7fda4c19d950 2024-03-04T07:30:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:30:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:30:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:30:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:58 smithi116 ceph-mon[32150]: Deploying daemon mon.b on smithi133 2024-03-04T07:30:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:58 smithi116 ceph-mon[32150]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:30:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:30:58.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.760+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda4c0636f0 con 0x7fda4c19d950 2024-03-04T07:30:58.761 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.761+0000 7fda3b7fe700 1 -- 172.21.15.152:0/3729094590 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7fda3c021560 con 0x7fda4c19d950 2024-03-04T07:30:58.761 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:30:58.761 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":2,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","modified":"2024-03-04T07:30:52.049301Z","created":"2024-03-04T07:28:44.925082Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T07:30:58.763 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fda2c0389e0 msgr2=0x7fda2c03aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fda2c0389e0 0x7fda2c03aea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda4000fa60 tx=0x7fda40005c20 comp rx=0 tx=0).stop 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda4c19d950 msgr2=0x7fda4c1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda4c19d950 0x7fda4c1975d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00b4d0 tx=0x7fda3c00b890 comp rx=0 tx=0).stop 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 shutdown_connections 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fda2c0389e0 0x7fda2c03aea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda340035c0 0x7fda4c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 --2- 172.21.15.152:0/3729094590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda4c19d950 0x7fda4c1975d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 >> 172.21.15.152:0/3729094590 conn(0x7fda4c0fc5d0 msgr2=0x7fda4c06a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 shutdown_connections 2024-03-04T07:30:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:30:58.763+0000 7fda50a3e700 1 -- 172.21.15.152:0/3729094590 wait complete. 2024-03-04T07:30:58.765 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 2 2024-03-04T07:30:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:30:59 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/3729094590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T07:30:59.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:30:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:30:59.049+0000 7f523ae00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-04T07:30:59.340 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:30:59 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/3729094590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T07:31:00.367 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T07:31:00.367 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T07:31:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:00 smithi116 ceph-mon[32150]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:00.520 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:00 smithi152 ceph-mon[37154]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:00.615 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:31:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:00 smithi133 systemd[1]: Started Ceph mon.b for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:31:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: mon.c calling monitor election 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: mon.a calling monitor election 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:06.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: fsmap 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: mgrmap e13: x(active, since 68s) 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: mon.b (rank 2) addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] is down (out of quorum) 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: mon.c calling monitor election 2024-03-04T07:31:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:31:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: mon.a calling monitor election 2024-03-04T07:31:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:31:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: fsmap 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: mgrmap e13: x(active, since 68s) 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T07:31:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T07:31:06.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: mon.b (rank 2) addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] is down (out of quorum) 2024-03-04T07:31:06.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:06.457 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.455+0000 7ff82ee56700 1 -- 172.21.15.152:0/1352465377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 msgr2=0x7ff818005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:06.457 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.455+0000 7ff82ee56700 1 --2- 172.21.15.152:0/1352465377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 0x7ff818005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff828101ac0 tx=0x7ff81c00fe60 comp rx=0 tx=0).stop 2024-03-04T07:31:06.457 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.455+0000 7ff82ee56700 1 -- 172.21.15.152:0/1352465377 shutdown_connections 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.455+0000 7ff82ee56700 1 --2- 172.21.15.152:0/1352465377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 0x7ff818005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.455+0000 7ff82ee56700 1 -- 172.21.15.152:0/1352465377 >> 172.21.15.152:0/1352465377 conn(0x7ff8280fd660 msgr2=0x7ff8280ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.455+0000 7ff82ee56700 1 -- 172.21.15.152:0/1352465377 shutdown_connections 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.455+0000 7ff82ee56700 1 -- 172.21.15.152:0/1352465377 wait complete. 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 Processor -- start 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 -- start start 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 0x7ff8280743c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 0x7ff828074d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 0x7ff8281a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff828109bd0 con 0x7ff8180035c0 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff828109a50 con 0x7ff8281a39d0 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82ee56700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff828109d50 con 0x7ff828074900 2024-03-04T07:31:06.458 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff827fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 0x7ff828074d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff827fff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 0x7ff828074d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:56214/0 (socket says 172.21.15.152:56214) 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff827fff700 1 -- 172.21.15.152:0/836786034 learned_addr learned my addr 172.21.15.152:0/836786034 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff827fff700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 msgr2=0x7ff828074d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff827fff700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 msgr2=0x7ff828074d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82cbf2700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 0x7ff8280743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82d3f3700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 0x7ff8281a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff827fff700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 0x7ff828074d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff827fff700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 0x7ff828074d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82d3f3700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 msgr2=0x7ff8281a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82d3f3700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 msgr2=0x7ff8281a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82d3f3700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 0x7ff8281a5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.456+0000 7ff82d3f3700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 0x7ff8281a5dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff82cbf2700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 msgr2=0x7ff828074d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T07:31:06.459 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff82cbf2700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 0x7ff828074d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff82cbf2700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 msgr2=0x7ff8281a5dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff82cbf2700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 0x7ff8281a5dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff82cbf2700 1 -- 172.21.15.152:0/836786034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8281a64f0 con 0x7ff8180035c0 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff82cbf2700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 0x7ff8280743c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff81c01f000 tx=0x7ff81c005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff825ffb700 1 -- 172.21.15.152:0/836786034 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c02b020 con 0x7ff8180035c0 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8281a6780 con 0x7ff8180035c0 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff825ffb700 1 -- 172.21.15.152:0/836786034 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff81c004110 con 0x7ff8180035c0 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.457+0000 7ff825ffb700 1 -- 172.21.15.152:0/836786034 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c022040 con 0x7ff8180035c0 2024-03-04T07:31:06.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.458+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8281a6cc0 con 0x7ff8180035c0 2024-03-04T07:31:06.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.459+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8280636f0 con 0x7ff8180035c0 2024-03-04T07:31:06.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.459+0000 7ff825ffb700 1 -- 172.21.15.152:0/836786034 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7ff81c02e020 con 0x7ff8180035c0 2024-03-04T07:31:06.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.459+0000 7ff825ffb700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff80c0390d0 0x7ff80c03b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:06.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.459+0000 7ff825ffb700 1 -- 172.21.15.152:0/836786034 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff81c04e5f0 con 0x7ff8180035c0 2024-03-04T07:31:06.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.459+0000 7ff827fff700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff80c0390d0 0x7ff80c03b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:06.461 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.460+0000 7ff827fff700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff80c0390d0 0x7ff80c03b590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff814009180 tx=0x7ff814008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:06.468 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.468+0000 7ff825ffb700 1 -- 172.21.15.152:0/836786034 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff81c035020 con 0x7ff8180035c0 2024-03-04T07:31:06.752 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.751+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff82802d9b0 con 0x7ff8180035c0 2024-03-04T07:31:06.752 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.753+0000 7ff825ffb700 1 -- 172.21.15.152:0/836786034 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1287 (secure 0 0 0) 0x7ff81c0146d0 con 0x7ff8180035c0 2024-03-04T07:31:06.753 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-04T07:31:06.753 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":3,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","modified":"2024-03-04T07:31:00.893857Z","created":"2024-03-04T07:28:44.925082Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff80c0390d0 msgr2=0x7ff80c03b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff80c0390d0 0x7ff80c03b590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff814009180 tx=0x7ff814008040 comp rx=0 tx=0).stop 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 msgr2=0x7ff8280743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 0x7ff8280743c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff81c01f000 tx=0x7ff81c005be0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 shutdown_connections 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8180035c0 0x7ff8280743c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff80c0390d0 0x7ff80c03b590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8281a39d0 0x7ff8281a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 --2- 172.21.15.152:0/836786034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff828074900 0x7ff828074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:06.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 >> 172.21.15.152:0/836786034 conn(0x7ff8280fd660 msgr2=0x7ff8280fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:06.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 shutdown_connections 2024-03-04T07:31:06.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:06.756+0000 7ff82ee56700 1 -- 172.21.15.152:0/836786034 wait complete. 2024-03-04T07:31:06.757 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 3 2024-03-04T07:31:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:06 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/836786034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T07:31:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:07.226 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-04T07:31:07.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-04T07:31:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:06 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/836786034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T07:31:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:31:08.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: mon.c calling monitor election 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: mon.a calling monitor election 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: fsmap 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: mgrmap e13: x(active, since 68s) 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: mon.b (rank 2) addr [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] is down (out of quorum) 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.339 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.339 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:08.339 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/836786034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T07:31:08.339 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: mon.b calling monitor election 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: mon.b calling monitor election 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: mon.c calling monitor election 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: mon.a calling monitor election 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: fsmap 2024-03-04T07:31:08.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: mgrmap e13: x(active, since 70s) 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: Cluster is now healthy 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: overall HEALTH_OK 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.824 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.839 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: mon.b calling monitor election 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: mon.b calling monitor election 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: mon.c calling monitor election 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: mon.a calling monitor election 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: fsmap 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: mgrmap e13: x(active, since 70s) 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T07:31:08.840 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: Cluster is now healthy 2024-03-04T07:31:08.841 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: overall HEALTH_OK 2024-03-04T07:31:08.841 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.841 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.841 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:08.841 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:08.841 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:08.841 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: mon.b calling monitor election 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: mon.b calling monitor election 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: mon.c calling monitor election 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: mon.a calling monitor election 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: monmap e3: 3 mons at {a=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],b=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: fsmap 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T07:31:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: mgrmap e13: x(active, since 70s) 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: Cluster is now healthy 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: overall HEALTH_OK 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.778+0000 7fde64200700 1 -- 172.21.15.116:0/745790205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 msgr2=0x7fde5c103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:09.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.778+0000 7fde64200700 1 --2- 172.21.15.116:0/745790205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c103060 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde50009910 tx=0x7fde50009c20 comp rx=0 tx=0).stop 2024-03-04T07:31:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.778+0000 7fde64200700 1 -- 172.21.15.116:0/745790205 shutdown_connections 2024-03-04T07:31:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.778+0000 7fde64200700 1 --2- 172.21.15.116:0/745790205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c103060 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.778+0000 7fde64200700 1 -- 172.21.15.116:0/745790205 >> 172.21.15.116:0/745790205 conn(0x7fde5c077760 msgr2=0x7fde5c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.778+0000 7fde64200700 1 -- 172.21.15.116:0/745790205 shutdown_connections 2024-03-04T07:31:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.778+0000 7fde64200700 1 -- 172.21.15.116:0/745790205 wait complete. 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.779+0000 7fde64200700 1 Processor -- start 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde64200700 1 -- start start 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde64200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde64200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c19fc60 0x7fde5c1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde64200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c199e20 0x7fde5c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde64200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c10c120 con 0x7fde5c102c80 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde64200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c10bfa0 con 0x7fde5c19fc60 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde64200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c10c2a0 con 0x7fde5c199e20 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde61800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde61800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58830/0 (socket says 172.21.15.116:58830) 2024-03-04T07:31:09.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde61800700 1 -- 172.21.15.116:0/3495489718 learned_addr learned my addr 172.21.15.116:0/3495489718 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:31:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.780+0000 7fde62200700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c199e20 0x7fde5c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.781+0000 7fde60e00700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c19fc60 0x7fde5c1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.781+0000 7fde61800700 1 -- 172.21.15.116:0/3495489718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c199e20 msgr2=0x7fde5c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.781+0000 7fde61800700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c199e20 0x7fde5c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.781+0000 7fde61800700 1 -- 172.21.15.116:0/3495489718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c19fc60 msgr2=0x7fde5c1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.781+0000 7fde61800700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c19fc60 0x7fde5c1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.781+0000 7fde61800700 1 -- 172.21.15.116:0/3495489718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c1a65f0 con 0x7fde5c102c80 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.781+0000 7fde61800700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c19f720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde500092e0 tx=0x7fde50004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.782+0000 7fde5a000700 1 -- 172.21.15.116:0/3495489718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50026070 con 0x7fde5c102c80 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.782+0000 7fde5a000700 1 -- 172.21.15.116:0/3495489718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde5000b750 con 0x7fde5c102c80 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.782+0000 7fde5a000700 1 -- 172.21.15.116:0/3495489718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50003a00 con 0x7fde5c102c80 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.782+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c1a67c0 con 0x7fde5c102c80 2024-03-04T07:31:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.782+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c1a6cf0 con 0x7fde5c102c80 2024-03-04T07:31:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.783+0000 7fde5a000700 1 -- 172.21.15.116:0/3495489718 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7fde50029070 con 0x7fde5c102c80 2024-03-04T07:31:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.783+0000 7fde5a000700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440390d0 0x7fde4403b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.783+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5c0636f0 con 0x7fde5c102c80 2024-03-04T07:31:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.783+0000 7fde5a000700 1 -- 172.21.15.116:0/3495489718 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fde5004c770 con 0x7fde5c102c80 2024-03-04T07:31:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.784+0000 7fde60e00700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440390d0 0x7fde4403b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.784+0000 7fde60e00700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440390d0 0x7fde4403b590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde48009ad0 tx=0x7fde48009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:09.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:09.789+0000 7fde5a000700 1 -- 172.21.15.116:0/3495489718 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fde50013070 con 0x7fde5c102c80 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: Updating smithi133:/etc/ceph/ceph.conf 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: Updating smithi133:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: Updating smithi152:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: Updating smithi116:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.898 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:09.899 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:09.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T07:31:09.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: Updating smithi133:/etc/ceph/ceph.conf 2024-03-04T07:31:09.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T07:31:09.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: Updating smithi133:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:09.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: Updating smithi152:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:09.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: Updating smithi116:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:09.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:10.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.017+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fde5c1a6fb0 con 0x7fde5c102c80 2024-03-04T07:31:10.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.018+0000 7fde5a000700 1 -- 172.21.15.116:0/3495489718 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7fde50030070 con 0x7fde5c102c80 2024-03-04T07:31:10.019 INFO:teuthology.orchestra.run.smithi116.stdout:# minimal ceph.conf for af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:31:10.019 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-03-04T07:31:10.019 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:31:10.019 INFO:teuthology.orchestra.run.smithi116.stdout: mon_host = [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-03-04T07:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.021+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440390d0 msgr2=0x7fde4403b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.021+0000 7fde64200700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440390d0 0x7fde4403b590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde48009ad0 tx=0x7fde48009210 comp rx=0 tx=0).stop 2024-03-04T07:31:10.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.021+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 msgr2=0x7fde5c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.021+0000 7fde64200700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c19f720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde500092e0 tx=0x7fde50004fa0 comp rx=0 tx=0).stop 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 shutdown_connections 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102c80 0x7fde5c19f720 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440390d0 0x7fde4403b590 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c199e20 0x7fde5c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 --2- 172.21.15.116:0/3495489718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c19fc60 0x7fde5c1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 >> 172.21.15.116:0/3495489718 conn(0x7fde5c077760 msgr2=0x7fde5c108080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 shutdown_connections 2024-03-04T07:31:10.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:10.022+0000 7fde64200700 1 -- 172.21.15.116:0/3495489718 wait complete. 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: Updating smithi133:/etc/ceph/ceph.conf 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: Updating smithi133:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: Updating smithi152:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: Updating smithi116:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:10.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:10.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:31:09 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:31:09.894+0000 7f523ae00700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-04T07:31:10.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:10 smithi116 ceph-mon[32150]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:10.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:10 smithi116 ceph-mon[32150]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T07:31:10.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:10 smithi116 ceph-mon[32150]: Reconfiguring daemon mon.a on smithi116 2024-03-04T07:31:10.669 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:10 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3495489718' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:10.855 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-04T07:31:10.856 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:31:10.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T07:31:10.921 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:10 smithi133 ceph-mon[36505]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:10.921 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:10 smithi133 ceph-mon[36505]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T07:31:10.921 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:10 smithi133 ceph-mon[36505]: Reconfiguring daemon mon.a on smithi116 2024-03-04T07:31:10.921 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:10 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3495489718' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:10.978 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:31:10.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:31:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:10 smithi152 ceph-mon[37154]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:10 smithi152 ceph-mon[37154]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T07:31:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:10 smithi152 ceph-mon[37154]: Reconfiguring daemon mon.a on smithi116 2024-03-04T07:31:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:10 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3495489718' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:11.103 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:31:11.103 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T07:31:11.138 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:31:11.138 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:31:11.208 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:31:11.208 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T07:31:11.238 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:31:11.238 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T07:31:11.304 INFO:tasks.cephadm:Adding mgr.x on smithi116 2024-03-04T07:31:11.305 INFO:tasks.cephadm:Adding mgr.y on smithi133 2024-03-04T07:31:11.305 INFO:tasks.cephadm:Adding mgr.z on smithi152 2024-03-04T07:31:11.305 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch apply mgr '3;smithi116=x;smithi133=y;smithi152=z' 2024-03-04T07:31:11.575 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:31:12.713 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:12 smithi152 ceph-mon[37154]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:12.714 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:12.714 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:12.714 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:12.714 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:12.714 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:12.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 -- 172.21.15.152:0/2319662311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 msgr2=0x7fd128100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:12.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 --2- 172.21.15.152:0/2319662311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 0x7fd128100f10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd118009910 tx=0x7fd118009c20 comp rx=0 tx=0).stop 2024-03-04T07:31:12.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 -- 172.21.15.152:0/2319662311 shutdown_connections 2024-03-04T07:31:12.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 --2- 172.21.15.152:0/2319662311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 0x7fd128100f10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:12.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 -- 172.21.15.152:0/2319662311 >> 172.21.15.152:0/2319662311 conn(0x7fd1280fc070 msgr2=0x7fd1280fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:12.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 -- 172.21.15.152:0/2319662311 shutdown_connections 2024-03-04T07:31:12.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 -- 172.21.15.152:0/2319662311 wait complete. 2024-03-04T07:31:12.911 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 Processor -- start 2024-03-04T07:31:12.911 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.910+0000 7fd12dc79700 1 -- start start 2024-03-04T07:31:12.913 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd12dc79700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 0x7fd12806ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:12.913 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd12dc79700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd12819a140 0x7fd12819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:12.913 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd12dc79700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd12819cac0 0x7fd12819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:12.913 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd12dc79700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd128106b30 con 0x7fd128100af0 2024-03-04T07:31:12.913 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd12dc79700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1281069b0 con 0x7fd12819cac0 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd12dc79700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd128106cb0 con 0x7fd12819a140 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd12819a140 0x7fd12819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd12819a140 0x7fd12819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:50924/0 (socket says 172.21.15.152:50924) 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 -- 172.21.15.152:0/1235172096 learned_addr learned my addr 172.21.15.152:0/1235172096 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd127fff700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd12819cac0 0x7fd12819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd1277fe700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 0x7fd12806ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 -- 172.21.15.152:0/1235172096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd12819cac0 msgr2=0x7fd12819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd12819cac0 0x7fd12819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 -- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 msgr2=0x7fd12806ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 0x7fd12806ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 -- 172.21.15.152:0/1235172096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12819f830 con 0x7fd12819a140 2024-03-04T07:31:12.914 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd127fff700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd12819cac0 0x7fd12819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd11ffff700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd12819a140 0x7fd12819c530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd11000b780 tx=0x7fd11000bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd1257fa700 1 -- 172.21.15.152:0/1235172096 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110004230 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12819fb20 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.911+0000 7fd1257fa700 1 -- 172.21.15.152:0/1235172096 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1100043d0 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.912+0000 7fd1257fa700 1 -- 172.21.15.152:0/1235172096 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110011c60 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.912+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1281a00b0 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.912+0000 7fd1257fa700 1 -- 172.21.15.152:0/1235172096 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7fd11001a400 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.912+0000 7fd1257fa700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1140390d0 0x7fd11403b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.913+0000 7fd1257fa700 1 -- 172.21.15.152:0/1235172096 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd11004b8b0 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.913+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1280636f0 con 0x7fd12819a140 2024-03-04T07:31:12.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.913+0000 7fd1277fe700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1140390d0 0x7fd11403b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:12.916 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.914+0000 7fd1277fe700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1140390d0 0x7fd11403b590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd118005330 tx=0x7fd118005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:12.917 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:12.918+0000 7fd1257fa700 1 -- 172.21.15.152:0/1235172096 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd110018750 con 0x7fd12819a140 2024-03-04T07:31:12.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:12 smithi133 ceph-mon[36505]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:12.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:12.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:12.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:12.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:12.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:12 smithi116 ceph-mon[32150]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:13.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:13.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:13.121 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.121+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi133=y;smithi152=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fd128062650 con 0x7fd1140390d0 2024-03-04T07:31:13.134 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.134+0000 7fd1257fa700 1 -- 172.21.15.152:0/1235172096 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd128062650 con 0x7fd1140390d0 2024-03-04T07:31:13.134 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mgr update... 2024-03-04T07:31:13.136 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.136+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1140390d0 msgr2=0x7fd11403b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.136+0000 7fd12dc79700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1140390d0 0x7fd11403b590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd118005330 tx=0x7fd118005fb0 comp rx=0 tx=0).stop 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd12819a140 msgr2=0x7fd12819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd12819a140 0x7fd12819c530 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd11000b780 tx=0x7fd11000bb40 comp rx=0 tx=0).stop 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 shutdown_connections 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd128100af0 0x7fd12806ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1140390d0 0x7fd11403b590 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd12819cac0 0x7fd12819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 --2- 172.21.15.152:0/1235172096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd12819a140 0x7fd12819c530 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 >> 172.21.15.152:0/1235172096 conn(0x7fd1280fc070 msgr2=0x7fd1280fccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 shutdown_connections 2024-03-04T07:31:13.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:31:13.137+0000 7fd12dc79700 1 -- 172.21.15.152:0/1235172096 wait complete. 2024-03-04T07:31:13.693 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:13 smithi152 ceph-mon[37154]: Reconfiguring mon.b (monmap changed)... 2024-03-04T07:31:13.693 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:13 smithi152 ceph-mon[37154]: Reconfiguring daemon mon.b on smithi133 2024-03-04T07:31:13.693 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:13 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:13.722 DEBUG:teuthology.orchestra.run.smithi133:mgr.y> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.y.service 2024-03-04T07:31:13.724 DEBUG:teuthology.orchestra.run.smithi152:mgr.z> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.z.service 2024-03-04T07:31:13.727 INFO:tasks.cephadm:Deploying OSDs... 2024-03-04T07:31:13.727 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:31:13.727 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T07:31:13.786 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T07:31:13.786 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-04T07:31:13.835 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 07:30:10.652676066 +0000 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 07:24:38.419225290 +0000 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 07:24:38.419225290 +0000 2024-03-04T07:31:13.836 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 07:24:38.419225290 +0000 2024-03-04T07:31:13.836 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T07:31:13.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:13 smithi116 ceph-mon[32150]: Reconfiguring mon.b (monmap changed)... 2024-03-04T07:31:13.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:13 smithi116 ceph-mon[32150]: Reconfiguring daemon mon.b on smithi133 2024-03-04T07:31:13.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T07:31:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T07:31:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000346113 s, 1.5 MB/s 2024-03-04T07:31:13.919 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T07:31:13.973 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-04T07:31:14.022 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T07:31:14.022 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:14.022 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-04T07:31:14.022 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:14.022 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:14.022 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 07:30:10.747674525 +0000 2024-03-04T07:31:14.022 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 07:24:38.993216174 +0000 2024-03-04T07:31:14.023 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 07:24:38.993216174 +0000 2024-03-04T07:31:14.023 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 07:24:38.993216174 +0000 2024-03-04T07:31:14.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T07:31:14.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:13 smithi133 ceph-mon[36505]: Reconfiguring mon.b (monmap changed)... 2024-03-04T07:31:14.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:13 smithi133 ceph-mon[36505]: Reconfiguring daemon mon.b on smithi133 2024-03-04T07:31:14.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:13 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:14.108 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T07:31:14.108 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T07:31:14.108 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000414254 s, 1.2 MB/s 2024-03-04T07:31:14.111 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T07:31:14.166 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 07:30:10.810673504 +0000 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 07:24:39.565207090 +0000 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 07:24:39.565207090 +0000 2024-03-04T07:31:14.214 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 07:24:39.565207090 +0000 2024-03-04T07:31:14.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T07:31:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T07:31:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T07:31:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000363809 s, 1.4 MB/s 2024-03-04T07:31:14.305 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T07:31:14.358 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-04T07:31:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T07:31:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:14.409 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-03-04T07:31:14.409 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:14.409 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:14.409 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 07:30:10.872672498 +0000 2024-03-04T07:31:14.409 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 07:24:40.137198006 +0000 2024-03-04T07:31:14.409 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 07:24:40.137198006 +0000 2024-03-04T07:31:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 07:24:40.137198006 +0000 2024-03-04T07:31:14.409 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T07:31:14.496 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-04T07:31:14.496 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-04T07:31:14.496 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000354174 s, 1.4 MB/s 2024-03-04T07:31:14.501 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T07:31:14.556 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-03-04T07:31:14.556 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T07:31:14.575 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T07:31:14.575 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-04 07:31:00.335149390 +0000 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-04 07:23:34.521561718 +0000 2024-03-04T07:31:14.633 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-04 07:23:34.521561718 +0000 2024-03-04T07:31:14.634 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-04 07:23:34.521561718 +0000 2024-03-04T07:31:14.634 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T07:31:14.701 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-04T07:31:14.701 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-04T07:31:14.701 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000285892 s, 1.8 MB/s 2024-03-04T07:31:14.703 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T07:31:14.760 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-04 07:31:00.336149371 +0000 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-04 07:23:34.894554651 +0000 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-04 07:23:34.894554651 +0000 2024-03-04T07:31:14.817 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-04 07:23:34.894554651 +0000 2024-03-04T07:31:14.818 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi133=y;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: Saving service mgr spec with placement smithi116=x;smithi133=y;smithi152=z;count:3 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:14.882 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:14.884 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-04T07:31:14.884 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-04T07:31:14.884 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.0002413 s, 2.1 MB/s 2024-03-04T07:31:14.885 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T07:31:14.943 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 2024-03-04T07:31:14.998 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T07:31:14.998 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:14.998 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-04T07:31:14.998 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:14.998 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:14.998 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-04 07:31:00.336149371 +0000 2024-03-04T07:31:14.999 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-04 07:23:35.434544421 +0000 2024-03-04T07:31:14.999 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-04 07:23:35.434544421 +0000 2024-03-04T07:31:14.999 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-04 07:23:35.434544421 +0000 2024-03-04T07:31:14.999 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi133=y;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: Saving service mgr spec with placement smithi116=x;smithi133=y;smithi152=z;count:3 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:15.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi133=y;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: Saving service mgr spec with placement smithi116=x;smithi133=y;smithi152=z;count:3 2024-03-04T07:31:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T07:31:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T07:31:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:15.064 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-04T07:31:15.064 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-04T07:31:15.064 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.0002568 s, 2.0 MB/s 2024-03-04T07:31:15.065 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T07:31:15.120 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 2024-03-04T07:31:15.174 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-03-04 07:31:00.336149371 +0000 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-03-04 07:23:35.782537828 +0000 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-03-04 07:23:35.782537828 +0000 2024-03-04T07:31:15.175 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-03-04 07:23:35.782537828 +0000 2024-03-04T07:31:15.175 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T07:31:15.240 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-03-04T07:31:15.240 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-03-04T07:31:15.240 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000214325 s, 2.4 MB/s 2024-03-04T07:31:15.241 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T07:31:15.297 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-04T07:31:15.297 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T07:31:15.313 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T07:31:15.313 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 07:30:50.995660966 +0000 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 07:23:33.731682793 +0000 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 07:23:33.731682793 +0000 2024-03-04T07:31:15.368 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 07:23:33.731682793 +0000 2024-03-04T07:31:15.369 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T07:31:15.434 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T07:31:15.434 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T07:31:15.434 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000197981 s, 2.6 MB/s 2024-03-04T07:31:15.436 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T07:31:15.491 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-03-04T07:31:15.545 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T07:31:15.545 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:15.545 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-04T07:31:15.546 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:15.546 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:15.546 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 07:30:50.995660966 +0000 2024-03-04T07:31:15.546 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 07:23:34.093678479 +0000 2024-03-04T07:31:15.546 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 07:23:34.093678479 +0000 2024-03-04T07:31:15.546 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 07:23:34.093678479 +0000 2024-03-04T07:31:15.546 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T07:31:15.611 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T07:31:15.611 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T07:31:15.611 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000183455 s, 2.8 MB/s 2024-03-04T07:31:15.612 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T07:31:15.668 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-03-04T07:31:15.722 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T07:31:15.722 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:15.722 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-04T07:31:15.722 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:15.722 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:15.723 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 07:30:50.996660954 +0000 2024-03-04T07:31:15.723 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 07:23:34.436674392 +0000 2024-03-04T07:31:15.723 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 07:23:34.436674392 +0000 2024-03-04T07:31:15.723 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 07:23:34.436674392 +0000 2024-03-04T07:31:15.723 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: Reconfiguring mon.c (monmap changed)... 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: Reconfiguring daemon mon.c on smithi152 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T07:31:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:15.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:15.789 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T07:31:15.789 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T07:31:15.789 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000198527 s, 2.6 MB/s 2024-03-04T07:31:15.791 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T07:31:15.845 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-03-04T07:31:15.900 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T07:31:15.900 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T07:31:15.900 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-04T07:31:15.901 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:31:15.901 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T07:31:15.901 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-04 07:30:50.996660954 +0000 2024-03-04T07:31:15.901 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-04 07:23:34.785670234 +0000 2024-03-04T07:31:15.901 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-04 07:23:34.785670234 +0000 2024-03-04T07:31:15.901 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-04 07:23:34.785670234 +0000 2024-03-04T07:31:15.901 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T07:31:15.966 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-04T07:31:15.966 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-04T07:31:15.966 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000201758 s, 2.5 MB/s 2024-03-04T07:31:15.967 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T07:31:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: Reconfiguring mon.c (monmap changed)... 2024-03-04T07:31:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: Reconfiguring daemon mon.c on smithi152 2024-03-04T07:31:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:16.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:16.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:16.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T07:31:16.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:16.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:16.023 INFO:tasks.cephadm:Deploying osd.0 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-04T07:31:16.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T07:31:16.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: Reconfiguring mon.c (monmap changed)... 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: Reconfiguring daemon mon.c on smithi152 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:16.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:16.452 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:31:16.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:16 smithi116 ceph-mon[32150]: Deploying daemon mgr.z on smithi152 2024-03-04T07:31:16.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:16 smithi116 ceph-mon[32150]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:16.873 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:16 smithi152 ceph-mon[37154]: Deploying daemon mgr.z on smithi152 2024-03-04T07:31:16.873 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:16 smithi152 ceph-mon[37154]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:17.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:16 smithi133 ceph-mon[36505]: Deploying daemon mgr.z on smithi152 2024-03-04T07:31:17.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:16 smithi133 ceph-mon[36505]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:17.521 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 systemd[1]: Starting Ceph mgr.z for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:31:17.796 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 podman[39014]: 2024-03-04 07:31:17.447346463 +0000 UTC m=+0.017710796 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:31:17.796 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 podman[39014]: 2024-03-04 07:31:17.560979326 +0000 UTC m=+0.131343726 container create b76a315aee7387c9a6de4a9d838414854cbe773ea8b116a94bc6acc7cba0af1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T07:31:17.796 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 podman[39014]: 2024-03-04 07:31:17.77000579 +0000 UTC m=+0.340370130 container init b76a315aee7387c9a6de4a9d838414854cbe773ea8b116a94bc6acc7cba0af1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:31:17.797 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 podman[39014]: 2024-03-04 07:31:17.772608357 +0000 UTC m=+0.342972684 container start b76a315aee7387c9a6de4a9d838414854cbe773ea8b116a94bc6acc7cba0af1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-04T07:31:17.797 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 ceph-mgr[39033]: -- 172.21.15.152:0/953267673 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fe2bb7e5a0 con 0x55fe2bb52c00 2024-03-04T07:31:18.271 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 bash[39014]: b76a315aee7387c9a6de4a9d838414854cbe773ea8b116a94bc6acc7cba0af1b 2024-03-04T07:31:18.271 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:17 smithi152 systemd[1]: Started Ceph mgr.z for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:31:18.622 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:18.316+0000 7efe37523200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T07:31:18.622 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:18.407+0000 7efe37523200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:18.853 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:18.853 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:18.853 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:18.853 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:18.853 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:18.853 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T07:31:18.854 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:18.854 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:19.369 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.042+0000 7efe37523200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T07:31:19.369 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.134+0000 7efe37523200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T07:31:19.369 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.275+0000 7efe37523200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T07:31:19.382 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T07:31:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T07:31:19.383 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-04T07:31:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-04T07:31:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0324208 s, 323 MB/s 2024-03-04T07:31:19.383 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-04T07:31:19.632 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.369+0000 7efe37523200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T07:31:19.632 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.451+0000 7efe37523200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T07:31:19.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:19 smithi133 ceph-mon[36505]: Deploying daemon mgr.y on smithi133 2024-03-04T07:31:19.907 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-mon[37154]: Deploying daemon mgr.y on smithi133 2024-03-04T07:31:19.907 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.714+0000 7efe37523200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T07:31:19.907 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.821+0000 7efe37523200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T07:31:19.907 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:19.906+0000 7efe37523200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T07:31:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:19 smithi116 ceph-mon[32150]: Deploying daemon mgr.y on smithi133 2024-03-04T07:31:20.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-04T07:31:20.179 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:20.001+0000 7efe37523200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T07:31:20.180 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:20.178+0000 7efe37523200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T07:31:20.616 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:31:20.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:20 smithi133 ceph-mon[36505]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:20.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:20 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:20.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:20 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:20.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:20 smithi116 ceph-mon[32150]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:20.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:20 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:20.931 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:20 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:20 smithi152 ceph-mon[37154]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:20 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:20 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.336 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:21.138+0000 7fc519159200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T07:31:21.521 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:21.160+0000 7efe37523200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T07:31:21.521 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:21.296+0000 7efe37523200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T07:31:21.909 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:21.816+0000 7fc519159200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T07:31:21.909 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:21 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.909 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:21 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.909 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:21 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:21.958 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.958 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.958 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:21.958 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:21.665+0000 7efe37523200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T07:31:21.958 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:21.756+0000 7efe37523200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T07:31:21.958 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:21.957+0000 7efe37523200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:31:21.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:21 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:21 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:21.963 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:21 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:22.238 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:21.908+0000 7fc519159200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T07:31:22.238 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.054+0000 7fc519159200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T07:31:22.238 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.155+0000 7fc519159200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T07:31:22.238 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.237+0000 7fc519159200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T07:31:22.270 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:22.150+0000 7efe37523200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T07:31:22.695 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:22.433+0000 7efe37523200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:31:22.696 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:22.695+0000 7efe37523200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T07:31:22.696 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:22.696 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.696 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.696 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.696 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.696 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:22.697 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:22.697 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.697 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:22.697 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:22.697 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:22.697 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 -- 172.21.15.116:0/4092532466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120102140 msgr2=0x7f8120102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 --2- 172.21.15.116:0/4092532466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120102140 0x7f8120102560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8114009990 tx=0x7f8114009ca0 comp rx=0 tx=0).stop 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 -- 172.21.15.116:0/4092532466 shutdown_connections 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 --2- 172.21.15.116:0/4092532466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8120103ce0 0x7f812010a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 --2- 172.21.15.116:0/4092532466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120102140 0x7f8120102560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 --2- 172.21.15.116:0/4092532466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8120103340 0x7f81201037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 -- 172.21.15.116:0/4092532466 >> 172.21.15.116:0/4092532466 conn(0x7f81200fd6e0 msgr2=0x7f81200ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 -- 172.21.15.116:0/4092532466 shutdown_connections 2024-03-04T07:31:22.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.765+0000 7f8128000700 1 -- 172.21.15.116:0/4092532466 wait complete. 2024-03-04T07:31:22.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.767+0000 7f8128000700 1 Processor -- start 2024-03-04T07:31:22.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.767+0000 7f8128000700 1 -- start start 2024-03-04T07:31:22.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8128000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8120102140 0x7f8120075e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:22.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8128000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120103340 0x7f81200744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:22.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8128000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8120103ce0 0x7f81200749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:22.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8128000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812010dd70 con 0x7f8120103ce0 2024-03-04T07:31:22.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8128000700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812010dbf0 con 0x7f8120102140 2024-03-04T07:31:22.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8128000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812010def0 con 0x7f8120103340 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8125600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8120102140 0x7f8120075e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8124c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120103340 0x7f81200744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8125600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8120102140 0x7f8120075e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49332/0 (socket says 172.21.15.116:49332) 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8125600700 1 -- 172.21.15.116:0/3265724102 learned_addr learned my addr 172.21.15.116:0/3265724102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8126000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8120103ce0 0x7f81200749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8124c00700 1 -- 172.21.15.116:0/3265724102 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8120102140 msgr2=0x7f8120075e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8124c00700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8120102140 0x7f8120075e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:22.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8124c00700 1 -- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8120103ce0 msgr2=0x7f81200749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8124c00700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8120103ce0 0x7f81200749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.768+0000 7f8124c00700 1 -- 172.21.15.116:0/3265724102 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81200752a0 con 0x7f8120103340 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.769+0000 7f8124c00700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120103340 0x7f81200744b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f810c00e9f0 tx=0x7f810c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.769+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c004220 con 0x7f8120103340 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.769+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f810c004d10 con 0x7f8120103340 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.769+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c005020 con 0x7f8120103340 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.769+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81201a2540 con 0x7f8120103340 2024-03-04T07:31:22.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.769+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81201a2a50 con 0x7f8120103340 2024-03-04T07:31:22.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.770+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8120068d80 con 0x7f8120103340 2024-03-04T07:31:22.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.770+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 13) v1 ==== 45985+0+0 (secure 0 0 0) 0x7f810c004750 con 0x7f8120103340 2024-03-04T07:31:22.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.771+0000 7f811e000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8108038e80 0x7f810803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:22.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.771+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f810c04bc70 con 0x7f8120103340 2024-03-04T07:31:22.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.771+0000 7f8125600700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8108038e80 0x7f810803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:22.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.772+0000 7f8125600700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8108038e80 0x7f810803b340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f811400b2e0 tx=0x7f8114005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:22.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:22.776+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f810c00c760 con 0x7f8120103340 2024-03-04T07:31:22.788 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.501+0000 7fc519159200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T07:31:22.788 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.610+0000 7fc519159200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.695+0000 7fc519159200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.788+0000 7fc519159200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:22.789 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:22.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:22.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T07:31:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T07:31:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:23.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:23.009+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8120062500 con 0x7f8108038e80 2024-03-04T07:31:23.020 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:22.929+0000 7efe37523200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T07:31:23.086 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:22.963+0000 7fc519159200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T07:31:23.927 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:23 smithi133 ceph-mon[36505]: Reconfiguring mgr.x (unknown last config time)... 2024-03-04T07:31:23.927 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:23 smithi133 ceph-mon[36505]: Reconfiguring daemon mgr.x on smithi116 2024-03-04T07:31:23.927 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:23 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:31:23.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:23 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:31:23.928 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:23 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:23 smithi116 ceph-mon[32150]: Reconfiguring mgr.x (unknown last config time)... 2024-03-04T07:31:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:23 smithi116 ceph-mon[32150]: Reconfiguring daemon mgr.x on smithi116 2024-03-04T07:31:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:23 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:31:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:23 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:31:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:23 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:24.021 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:23 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:23.717+0000 7efe37523200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T07:31:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:23 smithi152 ceph-mon[37154]: Reconfiguring mgr.x (unknown last config time)... 2024-03-04T07:31:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:23 smithi152 ceph-mon[37154]: Reconfiguring daemon mgr.x on smithi116 2024-03-04T07:31:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:23 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:31:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:23 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:31:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:23 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:24.336 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:23 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:23.926+0000 7fc519159200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T07:31:24.336 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:24.056+0000 7fc519159200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T07:31:24.590 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:24.283+0000 7efe37523200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T07:31:24.590 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:24.379+0000 7efe37523200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T07:31:24.590 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z[39029]: 2024-03-04T07:31:24.590+0000 7efe37523200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T07:31:24.669 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:24.386+0000 7fc519159200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T07:31:24.669 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:24.475+0000 7fc519159200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T07:31:24.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:24.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:24.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:24.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: Standby manager daemon z started 2024-03-04T07:31:24.722 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:24.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: Standby manager daemon z started 2024-03-04T07:31:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T07:31:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T07:31:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T07:31:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T07:31:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:25.086 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:24.669+0000 7fc519159200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:24.854+0000 7fc519159200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: Standby manager daemon z started 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.152:0/4142853374' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T07:31:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:25.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:25.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:25.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:25.393 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:25.133+0000 7fc519159200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T07:31:25.394 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:25.393+0000 7fc519159200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T07:31:25.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:25.611+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mgrmap(e 14) v1 ==== 91343+0+0 (secure 0 0 0) 0x7f810c005580 con 0x7f8120103340 2024-03-04T07:31:25.836 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:25.625+0000 7fc519159200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T07:31:26.836 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:26.416+0000 7fc519159200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T07:31:26.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:26 smithi133 ceph-mon[36505]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:26.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:26 smithi133 ceph-mon[36505]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-04T07:31:26.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:26 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T07:31:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:26 smithi116 ceph-mon[32150]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:26 smithi116 ceph-mon[32150]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-04T07:31:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:26 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T07:31:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:26 smithi152 ceph-mon[37154]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:26 smithi152 ceph-mon[37154]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-04T07:31:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:26 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T07:31:27.290 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:26.983+0000 7fc519159200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T07:31:27.290 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:27.079+0000 7fc519159200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T07:31:27.290 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:31:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y[37905]: 2024-03-04T07:31:27.289+0000 7fc519159200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T07:31:27.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:27.632+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mon.1 v2:172.21.15.152:3300/0 8 ==== mgrmap(e 15) v1 ==== 136701+0+0 (secure 0 0 0) 0x7f810c00aad0 con 0x7f8120103340 2024-03-04T07:31:27.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:27 smithi116 ceph-mon[32150]: Standby manager daemon y started 2024-03-04T07:31:27.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:27 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T07:31:27.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:27 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T07:31:27.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:27 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T07:31:27.634 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:27 smithi116 ceph-mon[32150]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T07:31:28.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:27 smithi152 ceph-mon[37154]: Standby manager daemon y started 2024-03-04T07:31:28.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:27 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T07:31:28.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:27 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T07:31:28.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:27 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T07:31:28.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:27 smithi152 ceph-mon[37154]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T07:31:28.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:27 smithi133 ceph-mon[36505]: Standby manager daemon y started 2024-03-04T07:31:28.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:27 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T07:31:28.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:27 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T07:31:28.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:27 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T07:31:28.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:27 smithi133 ceph-mon[36505]: from='mgr.? 172.21.15.133:0/2726657095' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T07:31:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:29.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-04T07:31:29.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T07:31:29.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/61764762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "914e00c8-058b-4b8a-8432-b88bb60d516b"}]: dispatch 2024-03-04T07:31:29.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/61764762' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "914e00c8-058b-4b8a-8432-b88bb60d516b"}]': finished 2024-03-04T07:31:29.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T07:31:29.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:29.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:28 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/529130163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/61764762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "914e00c8-058b-4b8a-8432-b88bb60d516b"}]: dispatch 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/61764762' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "914e00c8-058b-4b8a-8432-b88bb60d516b"}]': finished 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:28 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/529130163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:31:29.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-04T07:31:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T07:31:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/61764762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "914e00c8-058b-4b8a-8432-b88bb60d516b"}]: dispatch 2024-03-04T07:31:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/61764762' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "914e00c8-058b-4b8a-8432-b88bb60d516b"}]': finished 2024-03-04T07:31:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T07:31:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:28 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/529130163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:31:31.007 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:30 smithi116 ceph-mon[32150]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:30 smithi152 ceph-mon[37154]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:30 smithi133 ceph-mon[36505]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:32 smithi116 ceph-mon[32150]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:33.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:32 smithi152 ceph-mon[37154]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:33.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:32 smithi133 ceph-mon[36505]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:34.850 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:34 smithi116 ceph-mon[32150]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:34 smithi152 ceph-mon[37154]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:35.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:34 smithi133 ceph-mon[36505]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:36 smithi116 ceph-mon[32150]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:37.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:36 smithi152 ceph-mon[37154]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:37.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:36 smithi133 ceph-mon[36505]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:37.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T07:31:37.943 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T07:31:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:38.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T07:31:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:38.979 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:38 smithi116 ceph-mon[32150]: Deploying daemon osd.0 on smithi116 2024-03-04T07:31:38.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:38 smithi116 ceph-mon[32150]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:38 smithi152 ceph-mon[37154]: Deploying daemon osd.0 on smithi116 2024-03-04T07:31:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:38 smithi152 ceph-mon[37154]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:38 smithi133 ceph-mon[36505]: Deploying daemon osd.0 on smithi116 2024-03-04T07:31:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:38 smithi133 ceph-mon[36505]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:40 smithi116 ceph-mon[32150]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:41.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:40 smithi152 ceph-mon[37154]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:40 smithi133 ceph-mon[36505]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:42 smithi116 ceph-mon[32150]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:43.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:42 smithi152 ceph-mon[37154]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:42 smithi133 ceph-mon[36505]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:44 smithi116 ceph-mon[32150]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:45.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:44 smithi152 ceph-mon[37154]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:45.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:44 smithi133 ceph-mon[36505]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:45 smithi116 ceph-mon[32150]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:45 smithi133 ceph-mon[36505]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:45 smithi152 ceph-mon[37154]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:48.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:48 smithi116 ceph-mon[32150]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:48.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:48.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:48.728 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:48 smithi152 ceph-mon[37154]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:48 smithi133 ceph-mon[36505]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:48.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:48.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:48.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:48.790 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:49.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.559+0000 7f811e000700 1 -- 172.21.15.116:0/3265724102 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8120062500 con 0x7f8108038e80 2024-03-04T07:31:49.560 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 0 on host 'smithi116' 2024-03-04T07:31:49.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8108038e80 msgr2=0x7f810803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:49.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8108038e80 0x7f810803b340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f811400b2e0 tx=0x7f8114005fb0 comp rx=0 tx=0).stop 2024-03-04T07:31:49.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120103340 msgr2=0x7f81200744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:49.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120103340 0x7f81200744b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f810c00e9f0 tx=0x7f810c00ed00 comp rx=0 tx=0).stop 2024-03-04T07:31:49.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 shutdown_connections 2024-03-04T07:31:49.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8120103ce0 0x7f81200749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:49.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8108038e80 0x7f810803b340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:49.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8120102140 0x7f8120075e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:49.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 --2- 172.21.15.116:0/3265724102 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8120103340 0x7f81200744b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:49.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 >> 172.21.15.116:0/3265724102 conn(0x7f81200fd6e0 msgr2=0x7f8120106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:49.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.563+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 shutdown_connections 2024-03-04T07:31:49.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:31:49.564+0000 7f8128000700 1 -- 172.21.15.116:0/3265724102 wait complete. 2024-03-04T07:31:49.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:49 smithi116 ceph-mon[32150]: from='osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T07:31:49.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:49 smithi116 ceph-mon[32150]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T07:31:49.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:49 smithi152 ceph-mon[37154]: from='osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T07:31:49.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:49 smithi152 ceph-mon[37154]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T07:31:49.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:49 smithi133 ceph-mon[36505]: from='osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T07:31:49.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:49 smithi133 ceph-mon[36505]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T07:31:50.456 DEBUG:teuthology.orchestra.run.smithi116:osd.0> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.0.service 2024-03-04T07:31:50.461 INFO:tasks.cephadm:Deploying osd.1 on smithi133 with /dev/vg_nvme/lv_4... 2024-03-04T07:31:50.462 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:50.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:50.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:31:50.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T07:31:50.698 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:50.698 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:31:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T07:31:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:31:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T07:31:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:31:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:50.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:50.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:50.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:31:50.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T07:31:50.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:50.819 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:31:50.820 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:31:50.820 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T07:31:50.820 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:51.434 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:31:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0[44688]: 2024-03-04T07:31:51.024+0000 7fb30e800700 -1 osd.0 0 waiting for initial osdmap 2024-03-04T07:31:51.434 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:31:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0[44688]: 2024-03-04T07:31:51.040+0000 7fb303c00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:31:51.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:51.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:51 smithi116 ceph-mon[32150]: from='osd.0 ' entity='osd.0' 2024-03-04T07:31:51.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:51.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:51 smithi152 ceph-mon[37154]: from='osd.0 ' entity='osd.0' 2024-03-04T07:31:51.773 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:51.774 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:51 smithi133 ceph-mon[36505]: from='osd.0 ' entity='osd.0' 2024-03-04T07:31:52.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:52 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:31:52.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:52 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:31:52.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:52 smithi116 ceph-mon[32150]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:52.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:52 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:52.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:52 smithi116 ceph-mon[32150]: osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] boot 2024-03-04T07:31:52.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:52 smithi116 ceph-mon[32150]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T07:31:52.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:52 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:52.619 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T07:31:52.620 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T07:31:52.620 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2024-03-04T07:31:52.620 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2024-03-04T07:31:52.620 INFO:teuthology.orchestra.run.smithi133.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110608 s, 948 MB/s 2024-03-04T07:31:52.620 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2024-03-04T07:31:52.620 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:52 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:31:52.620 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:52 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:31:52.620 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:52 smithi133 ceph-mon[36505]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:52.620 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:52 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:52.620 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:52 smithi133 ceph-mon[36505]: osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] boot 2024-03-04T07:31:52.620 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:52 smithi133 ceph-mon[36505]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T07:31:52.621 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:52 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:52.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:52 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:31:52.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:52 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:31:52.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:52 smithi152 ceph-mon[37154]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T07:31:52.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:52 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:52.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:52 smithi152 ceph-mon[37154]: osd.0 [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] boot 2024-03-04T07:31:52.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:52 smithi152 ceph-mon[37154]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T07:31:52.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:52 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T07:31:53.198 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi133:vg_nvme/lv_4 2024-03-04T07:31:53.430 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:31:54.528 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:54 smithi133 ceph-mon[36505]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:54.528 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:54 smithi133 ceph-mon[36505]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T07:31:54.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:54 smithi116 ceph-mon[32150]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:54.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:54 smithi116 ceph-mon[32150]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T07:31:54.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:54 smithi152 ceph-mon[37154]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:54 smithi152 ceph-mon[37154]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T07:31:55.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 -- 172.21.15.133:0/2983675835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 msgr2=0x7f899c103b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:55.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 --2- 172.21.15.133:0/2983675835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c103b50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8990009990 tx=0x7f8990009ca0 comp rx=0 tx=0).stop 2024-03-04T07:31:55.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 -- 172.21.15.133:0/2983675835 shutdown_connections 2024-03-04T07:31:55.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 --2- 172.21.15.133:0/2983675835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f899c100b50 0x7f899c10a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:55.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 --2- 172.21.15.133:0/2983675835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c103b50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:55.019 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 --2- 172.21.15.133:0/2983675835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f899c1001b0 0x7f899c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 -- 172.21.15.133:0/2983675835 >> 172.21.15.133:0/2983675835 conn(0x7f899c0fbd10 msgr2=0x7f899c0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 -- 172.21.15.133:0/2983675835 shutdown_connections 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.018+0000 7f89a1945700 1 -- 172.21.15.133:0/2983675835 wait complete. 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 Processor -- start 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 -- start start 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f899c1001b0 0x7f899c102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f899c100b50 0x7f899c101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c1019c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c10ddc0 con 0x7f899c100b50 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c10dc40 con 0x7f899c103730 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f89a1945700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c10df40 con 0x7f899c1001b0 2024-03-04T07:31:55.020 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f899b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c1019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f899b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c1019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:60596/0 (socket says 172.21.15.133:60596) 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f899b7fe700 1 -- 172.21.15.133:0/1610169968 learned_addr learned my addr 172.21.15.133:0/1610169968 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f899affd700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f899c1001b0 0x7f899c102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f899b7fe700 1 -- 172.21.15.133:0/1610169968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f899c1001b0 msgr2=0x7f899c102e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.019+0000 7f899b7fe700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f899c1001b0 0x7f899c102e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f899b7fe700 1 -- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f899c100b50 msgr2=0x7f899c101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f899b7fe700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f899c100b50 0x7f899c101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f899b7fe700 1 -- 172.21.15.133:0/1610169968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c102270 con 0x7f899c103730 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f899affd700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f899c1001b0 0x7f899c102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f899b7fe700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c1019c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f899400b970 tx=0x7f899400bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f89a0943700 1 -- 172.21.15.133:0/1610169968 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994019070 con 0x7f899c103730 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c1a25f0 con 0x7f899c103730 2024-03-04T07:31:55.021 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f89a0943700 1 -- 172.21.15.133:0/1610169968 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8994014440 con 0x7f899c103730 2024-03-04T07:31:55.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f89a0943700 1 -- 172.21.15.133:0/1610169968 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89940075d0 con 0x7f899c103730 2024-03-04T07:31:55.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.020+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899c1a2b80 con 0x7f899c103730 2024-03-04T07:31:55.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.021+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899c068d80 con 0x7f899c103730 2024-03-04T07:31:55.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.022+0000 7f89a0943700 1 -- 172.21.15.133:0/1610169968 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 15) v1 ==== 136701+0+0 (secure 0 0 0) 0x7f899401c070 con 0x7f899c103730 2024-03-04T07:31:55.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.022+0000 7f89a0943700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f89840a1d80 0x7f89840a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:31:55.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.023+0000 7f89a0943700 1 -- 172.21.15.133:0/1610169968 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f89940cbbd0 con 0x7f899c103730 2024-03-04T07:31:55.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.023+0000 7f899affd700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f89840a1d80 0x7f89840a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:31:55.023 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.024+0000 7f899affd700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f89840a1d80 0x7f89840a4240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89900039c0 tx=0x7f8990005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:31:55.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.026+0000 7f89a0943700 1 -- 172.21.15.133:0/1610169968 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f899409aa70 con 0x7f899c103730 2024-03-04T07:31:55.229 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:31:55.229+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f899c062500 con 0x7f89840a1d80 2024-03-04T07:31:55.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:55.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:31:55.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:31:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:56 smithi116 ceph-mon[32150]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:56 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi116 2024-03-04T07:31:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:56 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi116 to 17091M 2024-03-04T07:31:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:56 smithi116 ceph-mon[32150]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:56.761 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:56 smithi133 ceph-mon[36505]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:56.761 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:56 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi116 2024-03-04T07:31:56.761 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:56 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi116 to 17091M 2024-03-04T07:31:56.761 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:56 smithi133 ceph-mon[36505]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:56.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:56 smithi152 ceph-mon[37154]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:31:56.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:56 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi116 2024-03-04T07:31:56.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:56 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi116 to 17091M 2024-03-04T07:31:56.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:56 smithi152 ceph-mon[37154]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:58.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:58 smithi133 ceph-mon[36505]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:58 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/3367338578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]: dispatch 2024-03-04T07:31:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:58 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]: dispatch 2024-03-04T07:31:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:58 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]': finished 2024-03-04T07:31:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:58 smithi133 ceph-mon[36505]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T07:31:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:31:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:58 smithi116 ceph-mon[32150]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:58 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/3367338578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]: dispatch 2024-03-04T07:31:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:58 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]: dispatch 2024-03-04T07:31:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:58 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]': finished 2024-03-04T07:31:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:58 smithi116 ceph-mon[32150]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T07:31:58.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:31:58.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:58 smithi152 ceph-mon[37154]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:31:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:58 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/3367338578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]: dispatch 2024-03-04T07:31:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:58 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]: dispatch 2024-03-04T07:31:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:58 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4809c734-3113-4aa9-b228-80c2e0c2dd5b"}]': finished 2024-03-04T07:31:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:58 smithi152 ceph-mon[37154]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T07:31:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:31:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:31:59 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/2718128894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:31:59.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:31:59 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/2718128894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:31:59.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:31:59 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/2718128894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:32:00.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:00 smithi116 ceph-mon[32150]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:00.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:00 smithi152 ceph-mon[37154]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:00.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:00 smithi133 ceph-mon[36505]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:02.753 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:02 smithi133 ceph-mon[36505]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:02.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:02 smithi116 ceph-mon[32150]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:02.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:02 smithi152 ceph-mon[37154]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:04.647 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:04 smithi133 ceph-mon[36505]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:04.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:04 smithi116 ceph-mon[32150]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:04.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:04 smithi152 ceph-mon[37154]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T07:32:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:05.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T07:32:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:05.775 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T07:32:05.775 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:06.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:06 smithi116 ceph-mon[32150]: Deploying daemon osd.1 on smithi133 2024-03-04T07:32:06.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:06 smithi116 ceph-mon[32150]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:06.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:06 smithi152 ceph-mon[37154]: Deploying daemon osd.1 on smithi133 2024-03-04T07:32:06.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:06 smithi152 ceph-mon[37154]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:06.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:06 smithi133 ceph-mon[36505]: Deploying daemon osd.1 on smithi133 2024-03-04T07:32:06.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:06 smithi133 ceph-mon[36505]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:08.540 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:08 smithi133 ceph-mon[36505]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:08 smithi116 ceph-mon[32150]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:08.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:08 smithi152 ceph-mon[37154]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:10.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:10 smithi116 ceph-mon[32150]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:10.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:10.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:10.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:10 smithi152 ceph-mon[37154]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:10.779 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:10 smithi133 ceph-mon[36505]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:10.779 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:10.779 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:10.779 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:12 smithi116 ceph-mon[32150]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:12.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:12 smithi152 ceph-mon[37154]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:12.835 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:12 smithi133 ceph-mon[36505]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:14.164 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.163+0000 7f89a0943700 1 -- 172.21.15.133:0/1610169968 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f899c062500 con 0x7f89840a1d80 2024-03-04T07:32:14.164 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 1 on host 'smithi133' 2024-03-04T07:32:14.165 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f89840a1d80 msgr2=0x7f89840a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:14.165 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f89840a1d80 0x7f89840a4240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89900039c0 tx=0x7f8990005fb0 comp rx=0 tx=0).stop 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 msgr2=0x7f899c1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c1019c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f899400b970 tx=0x7f899400bd30 comp rx=0 tx=0).stop 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 shutdown_connections 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f89840a1d80 0x7f89840a4240 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f899c103730 0x7f899c1019c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f899c1001b0 0x7f899c102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 --2- 172.21.15.133:0/1610169968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f899c100b50 0x7f899c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 >> 172.21.15.133:0/1610169968 conn(0x7f899c0fbd10 msgr2=0x7f899c1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 shutdown_connections 2024-03-04T07:32:14.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:32:14.165+0000 7f89a1945700 1 -- 172.21.15.133:0/1610169968 wait complete. 2024-03-04T07:32:14.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:14.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:14.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:14.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:14.790 DEBUG:teuthology.orchestra.run.smithi133:osd.1> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.1.service 2024-03-04T07:32:14.792 INFO:tasks.cephadm:Deploying osd.2 on smithi152 with /dev/vg_nvme/lv_4... 2024-03-04T07:32:14.793 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T07:32:15.039 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:32:15.463 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:15 smithi152 ceph-mon[37154]: from='osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T07:32:15.463 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:15 smithi152 ceph-mon[37154]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T07:32:15.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:15 smithi133 ceph-mon[36505]: from='osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T07:32:15.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:15 smithi133 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T07:32:15.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:15 smithi116 ceph-mon[32150]: from='osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T07:32:15.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:15 smithi116 ceph-mon[32150]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T07:32:16.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:16 smithi152 ceph-mon[37154]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T07:32:16.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:16 smithi152 ceph-mon[37154]: from='osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:32:16.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:16 smithi152 ceph-mon[37154]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T07:32:16.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:16 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:16.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:16 smithi152 ceph-mon[37154]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:32:16.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:16 smithi152 ceph-mon[37154]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:16.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T07:32:16.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-mon[36505]: from='osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:32:16.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-mon[36505]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T07:32:16.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:16.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:32:16.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-mon[36505]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:16 smithi116 ceph-mon[32150]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T07:32:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:16 smithi116 ceph-mon[32150]: from='osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:32:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:16 smithi116 ceph-mon[32150]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T07:32:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:16 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:16 smithi116 ceph-mon[32150]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:32:16.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:16 smithi116 ceph-mon[32150]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:16.808 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T07:32:16.808 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T07:32:16.808 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-04T07:32:16.808 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-04T07:32:16.808 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113924 s, 920 MB/s 2024-03-04T07:32:16.808 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-04T07:32:17.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1[41131]: 2024-03-04T07:32:16.737+0000 7f7d8b798700 -1 osd.1 0 waiting for initial osdmap 2024-03-04T07:32:17.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:16 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1[41131]: 2024-03-04T07:32:16.741+0000 7f7d83583700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:32:17.411 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_4 2024-03-04T07:32:17.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:17 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:32:17.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:17 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:32:17.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:17 smithi152 ceph-mon[37154]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:32:17.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:17 smithi152 ceph-mon[37154]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T07:32:17.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:17 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:17.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:17 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:17.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:17 smithi152 ceph-mon[37154]: from='osd.1 ' entity='osd.1' 2024-03-04T07:32:17.485 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:17 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:32:17.485 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:17 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:32:17.486 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:17 smithi133 ceph-mon[36505]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:32:17.486 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:17 smithi133 ceph-mon[36505]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T07:32:17.486 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:17 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:17.486 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:17 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:17.486 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:17 smithi133 ceph-mon[36505]: from='osd.1 ' entity='osd.1' 2024-03-04T07:32:17.658 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:32:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:17 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:32:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:17 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:32:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:17 smithi116 ceph-mon[32150]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:32:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:17 smithi116 ceph-mon[32150]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T07:32:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:17 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:17 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:17.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:17 smithi116 ceph-mon[32150]: from='osd.1 ' entity='osd.1' 2024-03-04T07:32:18.404 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] boot 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:18.405 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:18.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] boot 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:18.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: osd.1 [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] boot 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:18.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:18.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:18.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:19.135 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 -- 172.21.15.152:0/2472889174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 msgr2=0x7fa3ac102530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:19.135 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 --2- 172.21.15.152:0/2472889174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac102530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009990 tx=0x7fa3a0009ca0 comp rx=0 tx=0).stop 2024-03-04T07:32:19.135 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 -- 172.21.15.152:0/2472889174 shutdown_connections 2024-03-04T07:32:19.135 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 --2- 172.21.15.152:0/2472889174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3ac103cb0 0x7fa3ac10a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:19.136 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 --2- 172.21.15.152:0/2472889174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac102530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:19.136 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 --2- 172.21.15.152:0/2472889174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3ac103310 0x7fa3ac103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:19.136 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 -- 172.21.15.152:0/2472889174 >> 172.21.15.152:0/2472889174 conn(0x7fa3ac0fd670 msgr2=0x7fa3ac0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:32:19.136 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 -- 172.21.15.152:0/2472889174 shutdown_connections 2024-03-04T07:32:19.136 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.135+0000 7fa3b1081700 1 -- 172.21.15.152:0/2472889174 wait complete. 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 Processor -- start 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 -- start start 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3ac103310 0x7fa3ac1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3ac103cb0 0x7fa3ac19f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac10dd80 con 0x7fa3ac103cb0 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac10dc00 con 0x7fa3ac103310 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3b1081700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac10df00 con 0x7fa3ac102110 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3aad9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3aad9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac1983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:35060/0 (socket says 172.21.15.152:35060) 2024-03-04T07:32:19.138 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3aad9d700 1 -- 172.21.15.152:0/1158025519 learned_addr learned my addr 172.21.15.152:0/1158025519 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.136+0000 7fa3aa59c700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3ac103310 0x7fa3ac1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3aad9d700 1 -- 172.21.15.152:0/1158025519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3ac103310 msgr2=0x7fa3ac1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3aad9d700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3ac103310 0x7fa3ac1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3aad9d700 1 -- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3ac103cb0 msgr2=0x7fa3ac19f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3aad9d700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3ac103cb0 0x7fa3ac19f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3aad9d700 1 -- 172.21.15.152:0/1158025519 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac1a0000 con 0x7fa3ac102110 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3aad9d700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac1983a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000c00 tx=0x7fa3a00047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa393fff700 1 -- 172.21.15.152:0/1158025519 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0003c70 con 0x7fa3ac102110 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa393fff700 1 -- 172.21.15.152:0/1158025519 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3a0004830 con 0x7fa3ac102110 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac1a0290 con 0x7fa3ac102110 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa393fff700 1 -- 172.21.15.152:0/1158025519 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0005330 con 0x7fa3ac102110 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.137+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac1a0700 con 0x7fa3ac102110 2024-03-04T07:32:19.139 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.138+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac068d80 con 0x7fa3ac102110 2024-03-04T07:32:19.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.139+0000 7fa393fff700 1 -- 172.21.15.152:0/1158025519 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 136701+0+0 (secure 0 0 0) 0x7fa3a0005550 con 0x7fa3ac102110 2024-03-04T07:32:19.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.139+0000 7fa393fff700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3940a1d80 0x7fa3940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:19.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.140+0000 7fa393fff700 1 -- 172.21.15.152:0/1158025519 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fa3a00cd570 con 0x7fa3ac102110 2024-03-04T07:32:19.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.140+0000 7fa3aa59c700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3940a1d80 0x7fa3940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:32:19.141 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.141+0000 7fa3aa59c700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3940a1d80 0x7fa3940a4240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa39c009a10 tx=0x7fa39c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:32:19.144 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.144+0000 7fa393fff700 1 -- 172.21.15.152:0/1158025519 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa3a0013070 con 0x7fa3ac102110 2024-03-04T07:32:19.348 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:19.348+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3ac062500 con 0x7fa3940a1d80 2024-03-04T07:32:19.359 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:19 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi133 2024-03-04T07:32:19.359 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:19 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi133 to 17080M 2024-03-04T07:32:19.360 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:19 smithi133 ceph-mon[36505]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T07:32:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:19 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi133 2024-03-04T07:32:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:19 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi133 to 17080M 2024-03-04T07:32:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:19 smithi116 ceph-mon[32150]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T07:32:19.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:19 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi133 2024-03-04T07:32:19.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:19 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi133 to 17080M 2024-03-04T07:32:19.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:19 smithi152 ceph-mon[37154]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T07:32:20.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:20 smithi152 ceph-mon[37154]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:32:20.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:20 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:32:20.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:20 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:32:20.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:20 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:20.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:20 smithi152 ceph-mon[37154]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:20.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:20 smithi133 ceph-mon[36505]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:32:20.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:20 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:32:20.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:20 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:32:20.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:20 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:20.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:20 smithi133 ceph-mon[36505]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:20 smithi116 ceph-mon[32150]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:32:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:20 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:32:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:20 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:32:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:20 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:20 smithi116 ceph-mon[32150]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:22.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:22 smithi152 ceph-mon[37154]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:22.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:22 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/1726612851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]: dispatch 2024-03-04T07:32:22.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:22 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]: dispatch 2024-03-04T07:32:22.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:22 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]': finished 2024-03-04T07:32:22.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:22 smithi152 ceph-mon[37154]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T07:32:22.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:22.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:22 smithi116 ceph-mon[32150]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:22.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:22 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/1726612851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]: dispatch 2024-03-04T07:32:22.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:22 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]: dispatch 2024-03-04T07:32:22.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:22 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]': finished 2024-03-04T07:32:22.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:22 smithi116 ceph-mon[32150]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T07:32:22.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:22.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:22 smithi133 ceph-mon[36505]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:22.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:22 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/1726612851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]: dispatch 2024-03-04T07:32:22.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:22 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]: dispatch 2024-03-04T07:32:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:22 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e50d8f2-dbf0-4066-b2cb-df0821866acd"}]': finished 2024-03-04T07:32:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:22 smithi133 ceph-mon[36505]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T07:32:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:23.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:23 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/4229406102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:32:23.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:23 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/4229406102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:32:23.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:23 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/4229406102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:32:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:24 smithi116 ceph-mon[32150]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:24.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:24 smithi152 ceph-mon[37154]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:24.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:24 smithi133 ceph-mon[36505]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:26.702 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:26 smithi152 ceph-mon[37154]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:26.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:26 smithi116 ceph-mon[32150]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:26.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:26 smithi133 ceph-mon[36505]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:28.715 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:28 smithi152 ceph-mon[37154]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:28.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:28 smithi116 ceph-mon[32150]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:28.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:28 smithi133 ceph-mon[36505]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:29.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T07:32:29.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:29.764 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T07:32:29.764 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:29.784 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T07:32:29.784 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:30.737 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:30 smithi152 ceph-mon[37154]: Deploying daemon osd.2 on smithi152 2024-03-04T07:32:30.737 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:30 smithi152 ceph-mon[37154]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:30 smithi116 ceph-mon[32150]: Deploying daemon osd.2 on smithi152 2024-03-04T07:32:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:30 smithi116 ceph-mon[32150]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:30.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:30 smithi133 ceph-mon[36505]: Deploying daemon osd.2 on smithi152 2024-03-04T07:32:30.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:30 smithi133 ceph-mon[36505]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:32.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:32 smithi116 ceph-mon[32150]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:32 smithi152 ceph-mon[37154]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:32.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:32 smithi133 ceph-mon[36505]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:34.700 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:34 smithi152 ceph-mon[37154]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:34.700 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:34.700 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:34.700 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:34.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:34 smithi116 ceph-mon[32150]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:34.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:34.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:34.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:34.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:34 smithi133 ceph-mon[36505]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:34.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:34.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:34.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:36.704 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:36 smithi152 ceph-mon[37154]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:36.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:36 smithi116 ceph-mon[32150]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:36.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:36 smithi133 ceph-mon[36505]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:37.478 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.478+0000 7fa393fff700 1 -- 172.21.15.152:0/1158025519 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa3ac062500 con 0x7fa3940a1d80 2024-03-04T07:32:37.479 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 2 on host 'smithi152' 2024-03-04T07:32:37.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3940a1d80 msgr2=0x7fa3940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:37.481 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3940a1d80 0x7fa3940a4240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa39c009a10 tx=0x7fa39c009190 comp rx=0 tx=0).stop 2024-03-04T07:32:37.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 msgr2=0x7fa3ac1983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:37.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac1983a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0000c00 tx=0x7fa3a00047b0 comp rx=0 tx=0).stop 2024-03-04T07:32:37.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 shutdown_connections 2024-03-04T07:32:37.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3ac103cb0 0x7fa3ac19f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:37.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3940a1d80 0x7fa3940a4240 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:37.482 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3ac102110 0x7fa3ac1983a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:37.483 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 --2- 172.21.15.152:0/1158025519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3ac103310 0x7fa3ac1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:37.483 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 >> 172.21.15.152:0/1158025519 conn(0x7fa3ac0fd670 msgr2=0x7fa3ac106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:32:37.483 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 shutdown_connections 2024-03-04T07:32:37.483 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:32:37.481+0000 7fa3b1081700 1 -- 172.21.15.152:0/1158025519 wait complete. 2024-03-04T07:32:38.088 DEBUG:teuthology.orchestra.run.smithi152:osd.2> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.2.service 2024-03-04T07:32:38.132 INFO:tasks.cephadm:Deploying osd.3 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-04T07:32:38.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T07:32:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:38 smithi116 ceph-mon[32150]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:38 smithi116 ceph-mon[32150]: from='osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T07:32:38.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:38 smithi116 ceph-mon[32150]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T07:32:38.582 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:32:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:38 smithi152 ceph-mon[37154]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:38 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:38 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:38 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:38 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:38 smithi152 ceph-mon[37154]: from='osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T07:32:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:38 smithi152 ceph-mon[37154]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T07:32:38.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:38 smithi133 ceph-mon[36505]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:38.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:38 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:38 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:38 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:38 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:38 smithi133 ceph-mon[36505]: from='osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T07:32:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:38 smithi133 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T07:32:39.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:39 smithi152 ceph-mon[37154]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T07:32:39.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:39 smithi152 ceph-mon[37154]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T07:32:39.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:39 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:39.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:39 smithi152 ceph-mon[37154]: from='osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:32:39.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:39 smithi152 ceph-mon[37154]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:32:39.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:39 smithi133 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T07:32:39.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:39 smithi133 ceph-mon[36505]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T07:32:39.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:39 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:39.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:39 smithi133 ceph-mon[36505]: from='osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:32:39.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:39 smithi133 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:32:39.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:39 smithi116 ceph-mon[32150]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T07:32:39.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:39 smithi116 ceph-mon[32150]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T07:32:39.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:39 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:39.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:39 smithi116 ceph-mon[32150]: from='osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:32:39.946 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:39 smithi116 ceph-mon[32150]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:32:40.271 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2[42291]: 2024-03-04T07:32:39.963+0000 7f02db124700 -1 osd.2 0 waiting for initial osdmap 2024-03-04T07:32:40.271 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2[42291]: 2024-03-04T07:32:39.967+0000 7f02d270e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:32:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:40 smithi116 ceph-mon[32150]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:40 smithi116 ceph-mon[32150]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:32:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:40 smithi116 ceph-mon[32150]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T07:32:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:40.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:40 smithi116 ceph-mon[32150]: from='osd.2 ' entity='osd.2' 2024-03-04T07:32:40.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:40 smithi133 ceph-mon[36505]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:40 smithi133 ceph-mon[36505]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:32:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:40 smithi133 ceph-mon[36505]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T07:32:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:40 smithi133 ceph-mon[36505]: from='osd.2 ' entity='osd.2' 2024-03-04T07:32:40.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:40 smithi152 ceph-mon[37154]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:40.939 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:40 smithi152 ceph-mon[37154]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:32:40.939 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:40 smithi152 ceph-mon[37154]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T07:32:40.939 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:40.939 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:40.939 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:40 smithi152 ceph-mon[37154]: from='osd.2 ' entity='osd.2' 2024-03-04T07:32:41.465 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T07:32:41.466 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T07:32:41.466 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-04T07:32:41.466 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-04T07:32:41.466 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0361391 s, 290 MB/s 2024-03-04T07:32:41.466 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-04T07:32:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:32:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:32:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:42.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] boot 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:42.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:32:42.322 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-04T07:32:42.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] boot 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:42.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:42.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:42.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:42.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: osd.2 [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] boot 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:42.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:32:42.733 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:32:43.253 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:43 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi152 2024-03-04T07:32:43.253 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:43 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi152 to 17080M 2024-03-04T07:32:43.253 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:43 smithi116 ceph-mon[32150]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:43.253 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T07:32:43.253 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:43 smithi116 ceph-mon[32150]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T07:32:43.253 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:32:43.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:43 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi152 2024-03-04T07:32:43.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:43 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi152 to 17080M 2024-03-04T07:32:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:43 smithi133 ceph-mon[36505]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T07:32:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:43 smithi133 ceph-mon[36505]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T07:32:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:32:43.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:43 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi152 2024-03-04T07:32:43.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:43 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi152 to 17080M 2024-03-04T07:32:43.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:43 smithi152 ceph-mon[37154]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T07:32:43.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T07:32:43.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:43 smithi152 ceph-mon[37154]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T07:32:43.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:32:44.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T07:32:44.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:44 smithi116 ceph-mon[32150]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T07:32:44.418 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:44 smithi116 ceph-mon[32150]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:44.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:44 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T07:32:44.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:44 smithi133 ceph-mon[36505]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T07:32:44.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:44 smithi133 ceph-mon[36505]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:44.758 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:44 smithi116 sudo[49368]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T07:32:44.758 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:44 smithi116 sudo[49368]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:44.758 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:44 smithi116 sudo[49368]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:44.758 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:44 smithi116 sudo[49368]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:44.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:44 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T07:32:44.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:44 smithi152 ceph-mon[37154]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T07:32:44.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:44 smithi152 ceph-mon[37154]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.936+0000 7f4dabe00700 1 -- 172.21.15.116:0/126969439 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac074b20 msgr2=0x7f4dac074fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.936+0000 7f4dabe00700 1 --2- 172.21.15.116:0/126969439 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac074b20 0x7f4dac074fa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009990 tx=0x7f4d98009ca0 comp rx=0 tx=0).stop 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.936+0000 7f4dabe00700 1 -- 172.21.15.116:0/126969439 shutdown_connections 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.936+0000 7f4dabe00700 1 --2- 172.21.15.116:0/126969439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac109f00 0x7f4dac10c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.936+0000 7f4dabe00700 1 --2- 172.21.15.116:0/126969439 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4dac0760f0 0x7f4dac074550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.936+0000 7f4dabe00700 1 --2- 172.21.15.116:0/126969439 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac074b20 0x7f4dac074fa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.936+0000 7f4dabe00700 1 -- 172.21.15.116:0/126969439 >> 172.21.15.116:0/126969439 conn(0x7f4dac0fd5d0 msgr2=0x7f4dac0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.937+0000 7f4dabe00700 1 -- 172.21.15.116:0/126969439 shutdown_connections 2024-03-04T07:32:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.937+0000 7f4dabe00700 1 -- 172.21.15.116:0/126969439 wait complete. 2024-03-04T07:32:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 Processor -- start 2024-03-04T07:32:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 -- start start 2024-03-04T07:32:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4dac074b20 0x7f4dac19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac0760f0 0x7f4dac19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac109f00 0x7f4dac1a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac10fe60 con 0x7f4dac0760f0 2024-03-04T07:32:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac10fce0 con 0x7f4dac109f00 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.938+0000 7f4dabe00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac10ffe0 con 0x7f4dac074b20 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac0760f0 0x7f4dac19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac0760f0 0x7f4dac19a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51398/0 (socket says 172.21.15.116:51398) 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 -- 172.21.15.116:0/710571603 learned_addr learned my addr 172.21.15.116:0/710571603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4dab400700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac109f00 0x7f4dac1a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daaa00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4dac074b20 0x7f4dac19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 -- 172.21.15.116:0/710571603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4dac074b20 msgr2=0x7f4dac19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4dac074b20 0x7f4dac19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 -- 172.21.15.116:0/710571603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac109f00 msgr2=0x7f4dac1a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac109f00 0x7f4dac1a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.939+0000 7f4daa000700 1 -- 172.21.15.116:0/710571603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac1a20e0 con 0x7f4dac0760f0 2024-03-04T07:32:44.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.940+0000 7f4daa000700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac0760f0 0x7f4dac19a9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d98005f50 tx=0x7f4d9800bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:32:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.940+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98026070 con 0x7f4dac0760f0 2024-03-04T07:32:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.940+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac1a2370 con 0x7f4dac0760f0 2024-03-04T07:32:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.940+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d98003d70 con 0x7f4dac0760f0 2024-03-04T07:32:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.940+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98021410 con 0x7f4dac0760f0 2024-03-04T07:32:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.940+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac1a2820 con 0x7f4dac0760f0 2024-03-04T07:32:44.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.941+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac068d80 con 0x7f4dac0760f0 2024-03-04T07:32:44.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.943+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) v1 ==== 136701+0+0 (secure 0 0 0) 0x7f4d98029070 con 0x7f4dac0760f0 2024-03-04T07:32:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.945+0000 7f4da3400700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4d900a1d80 0x7f4d900a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:32:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.945+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f4d980cdc10 con 0x7f4dac0760f0 2024-03-04T07:32:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.946+0000 7f4daaa00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4d900a1d80 0x7f4d900a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:32:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.946+0000 7f4daaa00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4d900a1d80 0x7f4d900a4240 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0764d0 tx=0x7f4d9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:32:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:44.948+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4d98030020 con 0x7f4dac0760f0 2024-03-04T07:32:45.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:45.176+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dac062500 con 0x7f4d900a1d80 2024-03-04T07:32:45.176 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:44 smithi116 sudo[49398]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T07:32:45.176 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:45 smithi116 sudo[49398]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:45.176 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:45 smithi116 sudo[49398]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:45.177 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:32:45 smithi116 sudo[49398]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:45 smithi116 ceph-mon[32150]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T07:32:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:32:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:32:45.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:45.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45014]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T07:32:45.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:45.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45014]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:45.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45014]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:45.537 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:45 smithi133 ceph-mon[36505]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T07:32:45.537 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:45.537 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:32:45.538 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:32:45.538 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:45.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:45 smithi152 ceph-mon[37154]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T07:32:45.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:45.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:32:45.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:32:45.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:45.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T07:32:45.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:45.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:45.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:32:45 smithi133 sudo[45017]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:46.077 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:45 smithi152 sudo[45939]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T07:32:46.077 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:45 smithi152 sudo[45939]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:46.077 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:45 smithi152 sudo[45939]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:46.077 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:45 smithi152 sudo[45939]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:46.521 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:46 smithi152 sudo[45942]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T07:32:46.521 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:46 smithi152 sudo[45942]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:46.521 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:46 smithi152 sudo[45942]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:46.521 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:32:46 smithi152 sudo[45942]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:46 smithi116 sudo[49539]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T07:32:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:46 smithi116 sudo[49539]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:46 smithi116 sudo[49539]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:46 smithi116 sudo[49539]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:32:47.291+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4d98009d10 con 0x7f4dac0760f0 2024-03-04T07:32:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 sudo[49545]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T07:32:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 sudo[49545]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 sudo[49545]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:47.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 sudo[49545]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:47.777 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45945]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T07:32:47.777 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45945]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:47.777 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45945]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:47.777 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45945]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:47.918 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 ceph-mon[32150]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:32:47.918 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 ceph-mon[32150]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:47.918 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:47.918 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:47.918 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:47 smithi116 ceph-mon[32150]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:48.036 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:47 smithi133 ceph-mon[36505]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:47 smithi133 ceph-mon[36505]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:47 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:47 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:47 smithi133 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45020]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:48.037 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45020]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45948]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45948]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45948]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 sudo[45948]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 ceph-mon[37154]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 ceph-mon[37154]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:32:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:47 smithi152 ceph-mon[37154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:48.307 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45024]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T07:32:48.307 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45024]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T07:32:48.307 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45024]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T07:32:48.307 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 sudo[45024]: pam_unix(sudo:session): session closed for user root 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:49.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:48 smithi133 ceph-mon[36505]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-04T07:32:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:48 smithi116 ceph-mon[32150]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T07:32:49.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:48 smithi152 ceph-mon[37154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T07:32:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:49 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1730635395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]: dispatch 2024-03-04T07:32:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:49 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]: dispatch 2024-03-04T07:32:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:49 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]': finished 2024-03-04T07:32:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:49 smithi116 ceph-mon[32150]: osdmap e22: 4 total, 3 up, 4 in 2024-03-04T07:32:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:49 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:32:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:49 smithi116 ceph-mon[32150]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:50.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:49 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1730635395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]: dispatch 2024-03-04T07:32:50.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:49 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]: dispatch 2024-03-04T07:32:50.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:49 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]': finished 2024-03-04T07:32:50.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:49 smithi133 ceph-mon[36505]: osdmap e22: 4 total, 3 up, 4 in 2024-03-04T07:32:50.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:49 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:32:50.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:49 smithi133 ceph-mon[36505]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:50.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:49 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1730635395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]: dispatch 2024-03-04T07:32:50.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:49 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]: dispatch 2024-03-04T07:32:50.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:49 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3acba8d9-b204-41eb-8f35-936f0acb0f87"}]': finished 2024-03-04T07:32:50.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:49 smithi152 ceph-mon[37154]: osdmap e22: 4 total, 3 up, 4 in 2024-03-04T07:32:50.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:49 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:32:50.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:49 smithi152 ceph-mon[37154]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:51.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:50 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1748252069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:32:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:50 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1748252069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:32:51.270 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:50 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1748252069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:32:52.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:51 smithi152 ceph-mon[37154]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:52.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:51 smithi133 ceph-mon[36505]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:51 smithi116 ceph-mon[32150]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:32:54.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:54 smithi116 ceph-mon[32150]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 60 KiB/s wr, 4 op/s 2024-03-04T07:32:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:54 smithi152 ceph-mon[37154]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 60 KiB/s wr, 4 op/s 2024-03-04T07:32:54.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:54 smithi133 ceph-mon[36505]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 60 KiB/s wr, 4 op/s 2024-03-04T07:32:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:56 smithi116 ceph-mon[32150]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 41 KiB/s wr, 3 op/s 2024-03-04T07:32:56.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:56 smithi152 ceph-mon[37154]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 41 KiB/s wr, 3 op/s 2024-03-04T07:32:56.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:56 smithi133 ceph-mon[36505]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 41 KiB/s wr, 3 op/s 2024-03-04T07:32:58.728 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:58 smithi133 ceph-mon[36505]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 976 B/s rd, 31 KiB/s wr, 2 op/s 2024-03-04T07:32:58.728 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T07:32:58.728 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:58 smithi116 ceph-mon[32150]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 976 B/s rd, 31 KiB/s wr, 2 op/s 2024-03-04T07:32:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T07:32:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:58 smithi152 ceph-mon[37154]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 976 B/s rd, 31 KiB/s wr, 2 op/s 2024-03-04T07:32:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T07:32:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:32:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:32:59 smithi116 ceph-mon[32150]: Deploying daemon osd.3 on smithi116 2024-03-04T07:32:59.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:32:59 smithi133 ceph-mon[36505]: Deploying daemon osd.3 on smithi116 2024-03-04T07:32:59.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:32:59 smithi152 ceph-mon[37154]: Deploying daemon osd.3 on smithi116 2024-03-04T07:33:00.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:00 smithi116 ceph-mon[32150]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 788 B/s rd, 25 KiB/s wr, 1 op/s 2024-03-04T07:33:00.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:00 smithi152 ceph-mon[37154]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 788 B/s rd, 25 KiB/s wr, 1 op/s 2024-03-04T07:33:00.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:00 smithi133 ceph-mon[36505]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 788 B/s rd, 25 KiB/s wr, 1 op/s 2024-03-04T07:33:02.756 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:02 smithi116 ceph-mon[32150]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T07:33:02.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:02 smithi152 ceph-mon[37154]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T07:33:02.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:02 smithi133 ceph-mon[36505]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T07:33:04.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:04 smithi116 ceph-mon[32150]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T07:33:04.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:04 smithi152 ceph-mon[37154]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T07:33:04.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:04 smithi133 ceph-mon[36505]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 682 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T07:33:06.757 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:06 smithi116 ceph-mon[32150]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:06.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:06 smithi152 ceph-mon[37154]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:06.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:06 smithi133 ceph-mon[36505]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:08.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:07 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:07 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:08.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:07 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:08.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:07 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:08.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:07 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:08.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:07 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:08.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:08.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:08.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:08.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:08 smithi116 ceph-mon[32150]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:09.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:08 smithi152 ceph-mon[37154]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:09.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:08 smithi133 ceph-mon[36505]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:10 smithi116 ceph-mon[32150]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:11.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:11.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:10 smithi116 ceph-mon[32150]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T07:33:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:10 smithi152 ceph-mon[37154]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:10 smithi152 ceph-mon[37154]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T07:33:11.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:10 smithi133 ceph-mon[36505]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:11.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:10 smithi133 ceph-mon[36505]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T07:33:12.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.023+0000 7f4da3400700 1 -- 172.21.15.116:0/710571603 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4dac062500 con 0x7f4d900a1d80 2024-03-04T07:33:12.024 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 3 on host 'smithi116' 2024-03-04T07:33:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4d900a1d80 msgr2=0x7f4d900a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4d900a1d80 0x7f4d900a4240 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0764d0 tx=0x7f4d9c009260 comp rx=0 tx=0).stop 2024-03-04T07:33:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac0760f0 msgr2=0x7f4dac19a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac0760f0 0x7f4dac19a9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d98005f50 tx=0x7f4d9800bbb0 comp rx=0 tx=0).stop 2024-03-04T07:33:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 shutdown_connections 2024-03-04T07:33:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4d900a1d80 0x7f4d900a4240 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4dac109f00 0x7f4dac1a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4dac074b20 0x7f4dac19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 --2- 172.21.15.116:0/710571603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dac0760f0 0x7f4dac19a9c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 >> 172.21.15.116:0/710571603 conn(0x7f4dac0fd5d0 msgr2=0x7f4dac105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:33:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 shutdown_connections 2024-03-04T07:33:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:33:12.027+0000 7f4dabe00700 1 -- 172.21.15.116:0/710571603 wait complete. 2024-03-04T07:33:12.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T07:33:12.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T07:33:12.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:33:12.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:12.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:12.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:12.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:12.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:12.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T07:33:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T07:33:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:33:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T07:33:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:12.900 DEBUG:teuthology.orchestra.run.smithi116:osd.3> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.3.service 2024-03-04T07:33:12.907 INFO:tasks.cephadm:Deploying osd.4 on smithi133 with /dev/vg_nvme/lv_3... 2024-03-04T07:33:12.907 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T07:33:13.149 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:33:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:13 smithi116 ceph-mon[32150]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:33:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:13 smithi116 ceph-mon[32150]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T07:33:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:13.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:13 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:13.543 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:13 smithi133 ceph-mon[36505]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:33:13.543 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:13 smithi133 ceph-mon[36505]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T07:33:13.544 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:13 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:13.544 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:13 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:13.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:13 smithi152 ceph-mon[37154]: from='osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:33:13.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:13 smithi152 ceph-mon[37154]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T07:33:13.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:13 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:13.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:13 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:14.452 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:14 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:33:14.452 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:14 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:33:14.452 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:14 smithi116 ceph-mon[32150]: osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] boot 2024-03-04T07:33:14.452 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:14 smithi116 ceph-mon[32150]: osdmap e25: 4 total, 4 up, 4 in 2024-03-04T07:33:14.453 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:14.453 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:14 smithi116 ceph-mon[32150]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:14.453 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:14 smithi116 ceph-mon[32150]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T07:33:14.509 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:14 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:33:14.509 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:14 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:33:14.509 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:14 smithi133 ceph-mon[36505]: osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] boot 2024-03-04T07:33:14.510 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:14 smithi133 ceph-mon[36505]: osdmap e25: 4 total, 4 up, 4 in 2024-03-04T07:33:14.510 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:14.510 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:14 smithi133 ceph-mon[36505]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:14.510 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:14 smithi133 ceph-mon[36505]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T07:33:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:14 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:33:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:14 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:33:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:14 smithi152 ceph-mon[37154]: osd.3 [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] boot 2024-03-04T07:33:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:14 smithi152 ceph-mon[37154]: osdmap e25: 4 total, 4 up, 4 in 2024-03-04T07:33:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T07:33:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:14 smithi152 ceph-mon[37154]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:14 smithi152 ceph-mon[37154]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T07:33:15.086 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T07:33:15.086 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T07:33:15.086 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2024-03-04T07:33:15.086 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2024-03-04T07:33:15.087 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111501 s, 940 MB/s 2024-03-04T07:33:15.087 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2024-03-04T07:33:15.697 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi133:vg_nvme/lv_3 2024-03-04T07:33:15.941 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:33:16.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:16 smithi116 ceph-mon[32150]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T07:33:16.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:16 smithi116 ceph-mon[32150]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:16.759 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:16 smithi133 ceph-mon[36505]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T07:33:16.759 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:16 smithi133 ceph-mon[36505]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:16.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:16 smithi152 ceph-mon[37154]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T07:33:16.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:16 smithi152 ceph-mon[37154]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 -- 172.21.15.133:0/81085645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0101c00 msgr2=0x7fa2f01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/81085645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0101c00 0x7fa2f01094a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800b0a0 tx=0x7fa2e800b3b0 comp rx=0 tx=0).stop 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 -- 172.21.15.133:0/81085645 shutdown_connections 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/81085645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0102670 0x7fa2f0102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/81085645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0101c00 0x7fa2f01094a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/81085645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2f0101240 0x7fa2f01016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 -- 172.21.15.133:0/81085645 >> 172.21.15.133:0/81085645 conn(0x7fa2f00fbd30 msgr2=0x7fa2f00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 -- 172.21.15.133:0/81085645 shutdown_connections 2024-03-04T07:33:17.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.608+0000 7fa2f7d75700 1 -- 172.21.15.133:0/81085645 wait complete. 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 Processor -- start 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 -- start start 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101240 0x7fa2f0198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2f0101c00 0x7fa2f0198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0102670 0x7fa2f019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f010cc10 con 0x7fa2f0101240 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f010ca90 con 0x7fa2f0102670 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f7d75700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f010cd90 con 0x7fa2f0101c00 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f6312700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0102670 0x7fa2f019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f6312700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0102670 0x7fa2f019f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:58630/0 (socket says 172.21.15.133:58630) 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f6312700 1 -- 172.21.15.133:0/31996213 learned_addr learned my addr 172.21.15.133:0/31996213 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.609+0000 7fa2f5310700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2f0101c00 0x7fa2f0198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f5b11700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101240 0x7fa2f0198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:33:17.612 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f6312700 1 -- 172.21.15.133:0/31996213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2f0101c00 msgr2=0x7fa2f0198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f6312700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2f0101c00 0x7fa2f0198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f6312700 1 -- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101240 msgr2=0x7fa2f0198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f6312700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101240 0x7fa2f0198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f6312700 1 -- 172.21.15.133:0/31996213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f019fe80 con 0x7fa2f0102670 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f6312700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0102670 0x7fa2f019f7e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8007f00 tx=0x7fa2e8004dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2e6ffd700 1 -- 172.21.15.133:0/31996213 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8021070 con 0x7fa2f0102670 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2e6ffd700 1 -- 172.21.15.133:0/31996213 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2e8007c00 con 0x7fa2f0102670 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f01a0110 con 0x7fa2f0102670 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2e6ffd700 1 -- 172.21.15.133:0/31996213 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e801d450 con 0x7fa2f0102670 2024-03-04T07:33:17.613 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.610+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f01a05c0 con 0x7fa2f0102670 2024-03-04T07:33:17.614 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:17 smithi116 ceph-mon[32150]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T07:33:17.614 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.611+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f0068d80 con 0x7fa2f0102670 2024-03-04T07:33:17.614 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:17 smithi133 ceph-mon[36505]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T07:33:17.615 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.613+0000 7fa2e6ffd700 1 -- 172.21.15.133:0/31996213 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa2e8004120 con 0x7fa2f0102670 2024-03-04T07:33:17.615 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.613+0000 7fa2e6ffd700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2dc0a1e10 0x7fa2dc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:17.615 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.614+0000 7fa2e6ffd700 1 -- 172.21.15.133:0/31996213 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7fa2e800f020 con 0x7fa2f0102670 2024-03-04T07:33:17.615 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.614+0000 7fa2f5b11700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2dc0a1e10 0x7fa2dc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:33:17.615 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.615+0000 7fa2f5b11700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2dc0a1e10 0x7fa2dc0a42d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0005950 tx=0x7fa2e0005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:33:17.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.617+0000 7fa2e6ffd700 1 -- 172.21.15.133:0/31996213 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa2e802a020 con 0x7fa2f0102670 2024-03-04T07:33:17.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:17 smithi152 ceph-mon[37154]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T07:33:17.817 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:17.817+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2f00fde00 con 0x7fa2dc0a1e10 2024-03-04T07:33:18.725 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:18 smithi133 ceph-mon[36505]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T07:33:18.725 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:18 smithi133 ceph-mon[36505]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:18.726 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:33:18.726 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:33:18.726 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:18 smithi116 ceph-mon[32150]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T07:33:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:18 smithi116 ceph-mon[32150]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:33:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:33:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:18 smithi152 ceph-mon[37154]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T07:33:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:18 smithi152 ceph-mon[37154]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:33:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:33:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:19.808 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:33:19.808 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi116 2024-03-04T07:33:19.808 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi116 to 8545M 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:19.809 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:33:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi116 2024-03-04T07:33:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi116 to 8545M 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:20.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi116 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi116 to 8545M 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:20.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:20.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:20.974 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:20 smithi133 ceph-mon[36505]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-04T07:33:20.974 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:20 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/1916453630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]: dispatch 2024-03-04T07:33:20.974 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:20 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]: dispatch 2024-03-04T07:33:20.974 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:20 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]': finished 2024-03-04T07:33:20.974 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:20 smithi133 ceph-mon[36505]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T07:33:20.974 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:20 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:21.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:20 smithi116 ceph-mon[32150]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-04T07:33:21.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:20 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/1916453630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]: dispatch 2024-03-04T07:33:21.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:20 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]: dispatch 2024-03-04T07:33:21.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:20 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]': finished 2024-03-04T07:33:21.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:20 smithi116 ceph-mon[32150]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T07:33:21.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:20 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:20 smithi152 ceph-mon[37154]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-04T07:33:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:20 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/1916453630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]: dispatch 2024-03-04T07:33:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:20 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]: dispatch 2024-03-04T07:33:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:20 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60bcd05b-b11c-4cca-948b-7dd432c6cd2c"}]': finished 2024-03-04T07:33:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:20 smithi152 ceph-mon[37154]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T07:33:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:20 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:22.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:21 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/1083957532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:33:22.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:21 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/1083957532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:33:22.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:21 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/1083957532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:33:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:22 smithi116 ceph-mon[32150]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:33:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:22 smithi152 ceph-mon[37154]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:33:23.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:22 smithi133 ceph-mon[36505]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:33:24.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:24 smithi133 ceph-mon[36505]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-04T07:33:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:24 smithi116 ceph-mon[32150]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-04T07:33:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:24 smithi152 ceph-mon[37154]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-04T07:33:26.940 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:26 smithi133 ceph-mon[36505]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T07:33:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:26 smithi116 ceph-mon[32150]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T07:33:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:26 smithi152 ceph-mon[37154]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T07:33:28.962 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:28 smithi133 ceph-mon[36505]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T07:33:28.962 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T07:33:28.962 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:28 smithi116 ceph-mon[32150]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T07:33:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T07:33:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:28 smithi152 ceph-mon[37154]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T07:33:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T07:33:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:29.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:29 smithi133 ceph-mon[36505]: Deploying daemon osd.4 on smithi133 2024-03-04T07:33:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:29 smithi116 ceph-mon[32150]: Deploying daemon osd.4 on smithi133 2024-03-04T07:33:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:29 smithi152 ceph-mon[37154]: Deploying daemon osd.4 on smithi133 2024-03-04T07:33:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:30 smithi116 ceph-mon[32150]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:31.013 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:30 smithi133 ceph-mon[36505]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:30 smithi152 ceph-mon[37154]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:32.940 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:32 smithi133 ceph-mon[36505]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:32 smithi116 ceph-mon[32150]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:33.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:32 smithi152 ceph-mon[37154]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:34.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:34 smithi133 ceph-mon[36505]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:34.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:34.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:34.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:34 smithi116 ceph-mon[32150]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:34 smithi152 ceph-mon[37154]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:36.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:36 smithi133 ceph-mon[36505]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:36.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:36.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:36 smithi116 ceph-mon[32150]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:36 smithi152 ceph-mon[37154]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.301 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.300+0000 7fa2e6ffd700 1 -- 172.21.15.133:0/31996213 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa2f00fde00 con 0x7fa2dc0a1e10 2024-03-04T07:33:37.301 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 4 on host 'smithi133' 2024-03-04T07:33:37.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2dc0a1e10 msgr2=0x7fa2dc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:37.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2dc0a1e10 0x7fa2dc0a42d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0005950 tx=0x7fa2e0005cf0 comp rx=0 tx=0).stop 2024-03-04T07:33:37.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0102670 msgr2=0x7fa2f019f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:37.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0102670 0x7fa2f019f7e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8007f00 tx=0x7fa2e8004dd0 comp rx=0 tx=0).stop 2024-03-04T07:33:37.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 shutdown_connections 2024-03-04T07:33:37.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2f0101240 0x7fa2f0198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:37.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2dc0a1e10 0x7fa2dc0a42d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:37.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2f0102670 0x7fa2f019f7e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:37.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 --2- 172.21.15.133:0/31996213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2f0101c00 0x7fa2f0198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:37.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 >> 172.21.15.133:0/31996213 conn(0x7fa2f00fbd30 msgr2=0x7fa2f00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:33:37.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 shutdown_connections 2024-03-04T07:33:37.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:33:37.303+0000 7fa2f7d75700 1 -- 172.21.15.133:0/31996213 wait complete. 2024-03-04T07:33:37.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:37.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:37.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:37.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:37.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T07:33:37.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:37 smithi133 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T07:33:37.981 DEBUG:teuthology.orchestra.run.smithi133:osd.4> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.4.service 2024-03-04T07:33:37.983 INFO:tasks.cephadm:Deploying osd.5 on smithi152 with /dev/vg_nvme/lv_3... 2024-03-04T07:33:37.983 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T07:33:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:37 smithi116 ceph-mon[32150]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T07:33:38.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:37 smithi152 ceph-mon[37154]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T07:33:38.232 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:33:38.630 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:38 smithi152 ceph-mon[37154]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:38.630 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:38 smithi152 ceph-mon[37154]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T07:33:38.630 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:38 smithi152 ceph-mon[37154]: from='osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:33:38.630 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:38 smithi152 ceph-mon[37154]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T07:33:38.630 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:38 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:38.630 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:38 smithi152 ceph-mon[37154]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:33:38.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:38 smithi133 ceph-mon[36505]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:38.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:38 smithi133 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T07:33:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:38 smithi133 ceph-mon[36505]: from='osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:33:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:38 smithi133 ceph-mon[36505]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T07:33:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:38 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:38 smithi133 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:33:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:38 smithi116 ceph-mon[32150]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:38 smithi116 ceph-mon[32150]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T07:33:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:38 smithi116 ceph-mon[32150]: from='osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:33:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:38 smithi116 ceph-mon[32150]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T07:33:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:39.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:38 smithi116 ceph-mon[32150]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:33:40.087 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:33:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:33:39.809+0000 7fb330a4e700 -1 osd.4 0 waiting for initial osdmap 2024-03-04T07:33:40.087 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:33:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:33:39.815+0000 7fb328839700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:33:40.193 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T07:33:40.193 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T07:33:40.193 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-04T07:33:40.193 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-04T07:33:40.193 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011385 s, 921 MB/s 2024-03-04T07:33:40.193 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-04T07:33:40.306 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:40 smithi152 ceph-mon[37154]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:33:40.306 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:40 smithi152 ceph-mon[37154]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T07:33:40.306 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:40.307 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:40.307 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:40 smithi152 ceph-mon[37154]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:40.307 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:40 smithi152 ceph-mon[37154]: from='osd.4 ' entity='osd.4' 2024-03-04T07:33:40.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:40 smithi133 ceph-mon[36505]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:33:40.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:40 smithi133 ceph-mon[36505]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T07:33:40.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:40.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:40.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:40 smithi133 ceph-mon[36505]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:40.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:40 smithi133 ceph-mon[36505]: from='osd.4 ' entity='osd.4' 2024-03-04T07:33:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:40 smithi116 ceph-mon[32150]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:33:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:40 smithi116 ceph-mon[32150]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T07:33:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:40.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:40 smithi116 ceph-mon[32150]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-04T07:33:40.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:40 smithi116 ceph-mon[32150]: from='osd.4 ' entity='osd.4' 2024-03-04T07:33:40.813 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_3 2024-03-04T07:33:41.093 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:33:41.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:41 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:33:41.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:41 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:33:41.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:41.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:41 smithi152 ceph-mon[37154]: osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] boot 2024-03-04T07:33:41.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:41 smithi152 ceph-mon[37154]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T07:33:41.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:41.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:41 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:33:41.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:41 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:33:41.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:41.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:41 smithi133 ceph-mon[36505]: osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] boot 2024-03-04T07:33:41.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:41 smithi133 ceph-mon[36505]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T07:33:41.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:41.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:41 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:33:41.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:41 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:33:41.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:41.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:41 smithi116 ceph-mon[32150]: osd.4 [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] boot 2024-03-04T07:33:41.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:41 smithi116 ceph-mon[32150]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T07:33:41.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T07:33:42.494 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:42.494 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi133 2024-03-04T07:33:42.494 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.494 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.494 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:42.495 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:42.495 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi133 to 8540M 2024-03-04T07:33:42.495 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.495 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:42.495 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:42.495 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.495 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:42 smithi152 ceph-mon[37154]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T07:33:42.697 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 -- 172.21.15.152:0/14999511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed0101c80 msgr2=0x7f8ed0109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:42.697 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 --2- 172.21.15.152:0/14999511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed0101c80 0x7f8ed0109540 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800b0d0 tx=0x7f8ec800b3e0 comp rx=0 tx=0).stop 2024-03-04T07:33:42.697 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 -- 172.21.15.152:0/14999511 shutdown_connections 2024-03-04T07:33:42.697 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 --2- 172.21.15.152:0/14999511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed0101c80 0x7f8ed0109540 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:42.697 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 --2- 172.21.15.152:0/14999511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ed01025a0 0x7f8ed01029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:42.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 --2- 172.21.15.152:0/14999511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01012e0 0x7f8ed0101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:42.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 -- 172.21.15.152:0/14999511 >> 172.21.15.152:0/14999511 conn(0x7f8ed00fbd10 msgr2=0x7f8ed00fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:33:42.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 -- 172.21.15.152:0/14999511 shutdown_connections 2024-03-04T07:33:42.698 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 -- 172.21.15.152:0/14999511 wait complete. 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 Processor -- start 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.697+0000 7f8ed5294700 1 -- start start 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ed5294700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed01012e0 0x7f8ed0198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ed5294700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ed0101c80 0x7f8ed0198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ed5294700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01025a0 0x7f8ed019f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ed5294700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed010cc70 con 0x7f8ed01025a0 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ed5294700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed010caf0 con 0x7f8ed0101c80 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ed5294700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed010cdf0 con 0x7f8ed01012e0 2024-03-04T07:33:42.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed01012e0 0x7f8ed0198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed01012e0 0x7f8ed0198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:50128/0 (socket says 172.21.15.152:50128) 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 -- 172.21.15.152:0/1084986505 learned_addr learned my addr 172.21.15.152:0/1084986505 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ec67fc700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ed0101c80 0x7f8ed0198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 -- 172.21.15.152:0/1084986505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ed0101c80 msgr2=0x7f8ed0198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ed0101c80 0x7f8ed0198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 -- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01025a0 msgr2=0x7f8ed019f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01025a0 0x7f8ed019f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 -- 172.21.15.152:0/1084986505 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed019ff80 con 0x7f8ed01012e0 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8ec67fc700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ed0101c80 0x7f8ed0198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eceffd700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed01012e0 0x7f8ed0198320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000c920 tx=0x7f8ec000cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.698+0000 7f8eccff9700 1 -- 172.21.15.152:0/1084986505 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0004220 con 0x7f8ed01012e0 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.699+0000 7f8eccff9700 1 -- 172.21.15.152:0/1084986505 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec0007da0 con 0x7f8ed01012e0 2024-03-04T07:33:42.701 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.699+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed01a0270 con 0x7f8ed01012e0 2024-03-04T07:33:42.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.699+0000 7f8eccff9700 1 -- 172.21.15.152:0/1084986505 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0003e10 con 0x7f8ed01012e0 2024-03-04T07:33:42.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.699+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed01a07e0 con 0x7f8ed01012e0 2024-03-04T07:33:42.702 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.700+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0068d80 con 0x7f8ed01012e0 2024-03-04T07:33:42.703 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.700+0000 7f8eccff9700 1 -- 172.21.15.152:0/1084986505 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8ec00078c0 con 0x7f8ed01012e0 2024-03-04T07:33:42.703 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.701+0000 7f8eccff9700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8ebc0a1e10 0x7f8ebc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:33:42.703 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.701+0000 7f8eccff9700 1 -- 172.21.15.152:0/1084986505 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f8ec00cc9b0 con 0x7f8ed01012e0 2024-03-04T07:33:42.703 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.702+0000 7f8ec67fc700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8ebc0a1e10 0x7f8ebc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:33:42.703 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.702+0000 7f8ec67fc700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8ebc0a1e10 0x7f8ebc0a42d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8007900 tx=0x7f8eb8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:33:42.706 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.706+0000 7f8eccff9700 1 -- 172.21.15.152:0/1084986505 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8ec0014030 con 0x7f8ed01012e0 2024-03-04T07:33:42.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:42.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi133 2024-03-04T07:33:42.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:42.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:42.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi133 to 8540M 2024-03-04T07:33:42.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:42.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:42.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:42 smithi116 ceph-mon[32150]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T07:33:42.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi133 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi133 to 8540M 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:42 smithi133 ceph-mon[36505]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T07:33:42.910 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:33:42.910+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ed00fde80 con 0x7f8ebc0a1e10 2024-03-04T07:33:43.722 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:33:43.722 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:33:43.722 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:43.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:33:43.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:33:43.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:43.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:33:43.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:33:43.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:44.754 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:44 smithi152 ceph-mon[37154]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:33:44.754 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:44 smithi152 ceph-mon[37154]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:44.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:44 smithi116 ceph-mon[32150]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:33:44.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:44 smithi116 ceph-mon[32150]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:44.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:44 smithi133 ceph-mon[36505]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:33:44.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:44 smithi133 ceph-mon[36505]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:46.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:46 smithi116 ceph-mon[32150]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:46.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:46 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/3233478522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]: dispatch 2024-03-04T07:33:46.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:46 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]: dispatch 2024-03-04T07:33:46.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:46 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]': finished 2024-03-04T07:33:46.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:46 smithi116 ceph-mon[32150]: osdmap e35: 6 total, 5 up, 6 in 2024-03-04T07:33:46.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:46 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:33:46.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:46 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/2599372517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:33:46.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:46 smithi152 ceph-mon[37154]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:46.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:46 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/3233478522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]: dispatch 2024-03-04T07:33:46.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:46 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]: dispatch 2024-03-04T07:33:46.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:46 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]': finished 2024-03-04T07:33:46.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:46 smithi152 ceph-mon[37154]: osdmap e35: 6 total, 5 up, 6 in 2024-03-04T07:33:46.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:46 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:33:46.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:46 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/2599372517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:33:46.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:46 smithi133 ceph-mon[36505]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:46.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:46 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/3233478522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]: dispatch 2024-03-04T07:33:46.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:46 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]: dispatch 2024-03-04T07:33:46.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:46 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0fa4743-f949-4748-9772-3bb1292ebb01"}]': finished 2024-03-04T07:33:46.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:46 smithi133 ceph-mon[36505]: osdmap e35: 6 total, 5 up, 6 in 2024-03-04T07:33:46.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:46 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:33:46.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:46 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/2599372517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:33:48.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:48 smithi116 ceph-mon[32150]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:48 smithi152 ceph-mon[37154]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:48 smithi133 ceph-mon[36505]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:50.715 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:50 smithi152 ceph-mon[37154]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:50 smithi116 ceph-mon[32150]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:50.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:50 smithi133 ceph-mon[36505]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:52.718 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:52 smithi152 ceph-mon[37154]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:52.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:52 smithi116 ceph-mon[32150]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:52.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:52 smithi133 ceph-mon[36505]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:53.687 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T07:33:53.688 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:53.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T07:33:53.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:53.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T07:33:53.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:33:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:54 smithi152 ceph-mon[37154]: Deploying daemon osd.5 on smithi152 2024-03-04T07:33:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:54 smithi152 ceph-mon[37154]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:54.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:54 smithi116 ceph-mon[32150]: Deploying daemon osd.5 on smithi152 2024-03-04T07:33:54.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:54 smithi116 ceph-mon[32150]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:54.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:54 smithi133 ceph-mon[36505]: Deploying daemon osd.5 on smithi152 2024-03-04T07:33:54.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:54 smithi133 ceph-mon[36505]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:56.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:56 smithi116 ceph-mon[32150]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:56.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:56 smithi152 ceph-mon[37154]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:56.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:56 smithi133 ceph-mon[36505]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:58 smithi116 ceph-mon[32150]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:58.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:33:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:58 smithi152 ceph-mon[37154]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:58.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:33:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:58.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:58 smithi133 ceph-mon[36505]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:33:58.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:33:58.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:33:58.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:33:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:00.698 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:00 smithi152 ceph-mon[37154]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:00.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:00 smithi116 ceph-mon[32150]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:00.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:00 smithi133 ceph-mon[36505]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:01.395 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 5 on host 'smithi152' 2024-03-04T07:34:01.395 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.394+0000 7f8eccff9700 1 -- 172.21.15.152:0/1084986505 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8ed00fde80 con 0x7f8ebc0a1e10 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8ebc0a1e10 msgr2=0x7f8ebc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8ebc0a1e10 0x7f8ebc0a42d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8007900 tx=0x7f8eb8008040 comp rx=0 tx=0).stop 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed01012e0 msgr2=0x7f8ed0198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed01012e0 0x7f8ed0198320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000c920 tx=0x7f8ec000cce0 comp rx=0 tx=0).stop 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 shutdown_connections 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ed01025a0 0x7f8ed019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8ebc0a1e10 0x7f8ebc0a42d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ed01012e0 0x7f8ed0198320 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 --2- 172.21.15.152:0/1084986505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ed0101c80 0x7f8ed0198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:01.397 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 >> 172.21.15.152:0/1084986505 conn(0x7f8ed00fbd10 msgr2=0x7f8ed00fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:34:01.398 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 shutdown_connections 2024-03-04T07:34:01.398 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:34:01.397+0000 7f8ed5294700 1 -- 172.21.15.152:0/1084986505 wait complete. 2024-03-04T07:34:02.019 DEBUG:teuthology.orchestra.run.smithi152:osd.5> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.5.service 2024-03-04T07:34:02.021 INFO:tasks.cephadm:Deploying osd.6 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-04T07:34:02.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T07:34:02.459 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:34:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:02 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:02 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:02 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:02 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:02 smithi116 ceph-mon[32150]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:02 smithi116 ceph-mon[32150]: from='osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T07:34:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:02 smithi116 ceph-mon[32150]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T07:34:02.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:02 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:02 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:02 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:02 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:02 smithi133 ceph-mon[36505]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:02 smithi133 ceph-mon[36505]: from='osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T07:34:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:02 smithi133 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T07:34:02.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:02 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:02 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:02 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:02 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:02.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:02 smithi152 ceph-mon[37154]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:02.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:02 smithi152 ceph-mon[37154]: from='osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T07:34:02.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:02 smithi152 ceph-mon[37154]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T07:34:03.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:03 smithi152 ceph-mon[37154]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T07:34:03.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:03 smithi152 ceph-mon[37154]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T07:34:03.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:03.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:03 smithi152 ceph-mon[37154]: from='osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:34:03.470 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:03 smithi152 ceph-mon[37154]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:34:03.470 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:34:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:34:03.409+0000 7f343288f700 -1 osd.5 0 waiting for initial osdmap 2024-03-04T07:34:03.471 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:34:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:34:03.415+0000 7f342e682700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:34:03.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:03 smithi116 ceph-mon[32150]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T07:34:03.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:03 smithi116 ceph-mon[32150]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T07:34:03.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:03.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:03 smithi116 ceph-mon[32150]: from='osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:34:03.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:03 smithi116 ceph-mon[32150]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:34:03.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:03 smithi133 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T07:34:03.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:03 smithi133 ceph-mon[36505]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T07:34:03.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:03.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:03 smithi133 ceph-mon[36505]: from='osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:34:03.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:03 smithi133 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:34:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:04 smithi152 ceph-mon[37154]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:34:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:04 smithi152 ceph-mon[37154]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:04 smithi152 ceph-mon[37154]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T07:34:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:04 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:04 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:04.558 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:04 smithi116 ceph-mon[32150]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:34:04.558 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:04 smithi116 ceph-mon[32150]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:04.559 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:04 smithi116 ceph-mon[32150]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T07:34:04.559 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:04 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:04.559 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:04 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:04.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:04 smithi133 ceph-mon[36505]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:34:04.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:04 smithi133 ceph-mon[36505]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:04.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:04 smithi133 ceph-mon[36505]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T07:34:04.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:04 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:04.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:04 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:05.519 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T07:34:05.520 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T07:34:05.520 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-04T07:34:05.520 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-04T07:34:05.520 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0325646 s, 322 MB/s 2024-03-04T07:34:05.520 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] boot 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: osdmap e38: 6 total, 6 up, 6 in 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:05.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] boot 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: osdmap e38: 6 total, 6 up, 6 in 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:05.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:05.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:05.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:05.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:05.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: osd.5 [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] boot 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: osdmap e38: 6 total, 6 up, 6 in 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:05.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:05.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:05.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:05.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:06.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-04T07:34:06.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:06 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi152 2024-03-04T07:34:06.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:06 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi152 to 8540M 2024-03-04T07:34:06.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:06 smithi116 ceph-mon[32150]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:06.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:06 smithi116 ceph-mon[32150]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T07:34:06.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:06 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi152 2024-03-04T07:34:06.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:06 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi152 to 8540M 2024-03-04T07:34:06.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:06 smithi152 ceph-mon[37154]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:06.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:06 smithi152 ceph-mon[37154]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T07:34:06.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:06 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi152 2024-03-04T07:34:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:06 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi152 to 8540M 2024-03-04T07:34:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:06 smithi133 ceph-mon[36505]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-04T07:34:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:06 smithi133 ceph-mon[36505]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T07:34:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:34:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:07 smithi116 ceph-mon[32150]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T07:34:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:07 smithi116 ceph-mon[32150]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T07:34:07.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:07 smithi152 ceph-mon[37154]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T07:34:07.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:07 smithi152 ceph-mon[37154]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T07:34:07.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:07 smithi133 ceph-mon[36505]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T07:34:07.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:07 smithi133 ceph-mon[36505]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T07:34:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:08 smithi116 ceph-mon[32150]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:08 smithi116 ceph-mon[32150]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T07:34:08.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:08 smithi152 ceph-mon[37154]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:08.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:08 smithi152 ceph-mon[37154]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T07:34:08.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:08 smithi133 ceph-mon[36505]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:08 smithi133 ceph-mon[36505]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T07:34:09.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 -- 172.21.15.116:0/4026822584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0105530 msgr2=0x7fb4c0105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:09.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 --2- 172.21.15.116:0/4026822584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0105530 0x7fb4c0105990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009990 tx=0x7fb4ac009ca0 comp rx=0 tx=0).stop 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 -- 172.21.15.116:0/4026822584 shutdown_connections 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 --2- 172.21.15.116:0/4026822584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0104330 0x7fb4c0104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 --2- 172.21.15.116:0/4026822584 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4c0105ed0 0x7fb4c010c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 --2- 172.21.15.116:0/4026822584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0105530 0x7fb4c0105990 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 -- 172.21.15.116:0/4026822584 >> 172.21.15.116:0/4026822584 conn(0x7fb4c00ff8d0 msgr2=0x7fb4c0101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 -- 172.21.15.116:0/4026822584 shutdown_connections 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.281+0000 7fb4c8600700 1 -- 172.21.15.116:0/4026822584 wait complete. 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.283+0000 7fb4c8600700 1 Processor -- start 2024-03-04T07:34:09.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.283+0000 7fb4c8600700 1 -- start start 2024-03-04T07:34:09.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.283+0000 7fb4c8600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0104330 0x7fb4c019a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:09.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.283+0000 7fb4c8600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0105530 0x7fb4c019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:09.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c8600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4c0105ed0 0x7fb4c01a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c8600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c010ff40 con 0x7fb4c0105530 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c8600700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c010fdc0 con 0x7fb4c0105ed0 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c8600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01100c0 con 0x7fb4c0104330 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0105530 0x7fb4c019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0105530 0x7fb4c019aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33030/0 (socket says 172.21.15.116:33030) 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5200700 1 -- 172.21.15.116:0/3733916336 learned_addr learned my addr 172.21.15.116:0/3733916336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5c00700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0104330 0x7fb4c019a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:34:09.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c6600700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4c0105ed0 0x7fb4c01a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:34:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5c00700 1 -- 172.21.15.116:0/3733916336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4c0105ed0 msgr2=0x7fb4c01a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5c00700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4c0105ed0 0x7fb4c01a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5c00700 1 -- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0105530 msgr2=0x7fb4c019aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5c00700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0105530 0x7fb4c019aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.284+0000 7fb4c5c00700 1 -- 172.21.15.116:0/3733916336 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c01a21b0 con 0x7fb4c0104330 2024-03-04T07:34:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.285+0000 7fb4c5c00700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0104330 0x7fb4c019a550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e9f0 tx=0x7fb4b400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:34:09.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.285+0000 7fb4be000700 1 -- 172.21.15.116:0/3733916336 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4004220 con 0x7fb4c0104330 2024-03-04T07:34:09.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.285+0000 7fb4be000700 1 -- 172.21.15.116:0/3733916336 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4b40043c0 con 0x7fb4c0104330 2024-03-04T07:34:09.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.285+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c01a24a0 con 0x7fb4c0104330 2024-03-04T07:34:09.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.285+0000 7fb4be000700 1 -- 172.21.15.116:0/3733916336 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4003cb0 con 0x7fb4c0104330 2024-03-04T07:34:09.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.285+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c01a29e0 con 0x7fb4c0104330 2024-03-04T07:34:09.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.286+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c0068d80 con 0x7fb4c0104330 2024-03-04T07:34:09.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.288+0000 7fb4be000700 1 -- 172.21.15.116:0/3733916336 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb4b400c7c0 con 0x7fb4c0104330 2024-03-04T07:34:09.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.289+0000 7fb4be000700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4a80a1e10 0x7fb4a80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:09.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.289+0000 7fb4be000700 1 -- 172.21.15.116:0/3733916336 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4533+0+0 (secure 0 0 0) 0x7fb4b40cd1a0 con 0x7fb4c0104330 2024-03-04T07:34:09.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.290+0000 7fb4c5200700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4a80a1e10 0x7fb4a80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:34:09.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.290+0000 7fb4c5200700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4a80a1e10 0x7fb4a80a42d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00b2e0 tx=0x7fb4ac00b8f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:34:09.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.293+0000 7fb4be000700 1 -- 172.21.15.116:0/3733916336 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb4b409b3f0 con 0x7fb4c0104330 2024-03-04T07:34:09.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:09.525+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4c0062500 con 0x7fb4a80a1e10 2024-03-04T07:34:10.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:10 smithi116 ceph-mon[32150]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:34:10.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:10 smithi116 ceph-mon[32150]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:34:10.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:34:10.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:34:10.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:10 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:10 smithi152 ceph-mon[37154]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:34:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:10 smithi152 ceph-mon[37154]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:34:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:34:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:34:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:10 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:10.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:10 smithi133 ceph-mon[36505]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:34:10.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:10 smithi133 ceph-mon[36505]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:34:10.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:34:10.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:34:10.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:10 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:12 smithi116 ceph-mon[32150]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:34:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:12 smithi152 ceph-mon[37154]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:34:12.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:12 smithi133 ceph-mon[36505]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:34:14.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:14 smithi116 ceph-mon[32150]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-04T07:34:14.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/317506943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2442f5fc-a522-4f08-8a68-c0bc903427b4"}]: dispatch 2024-03-04T07:34:14.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/317506943' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2442f5fc-a522-4f08-8a68-c0bc903427b4"}]': finished 2024-03-04T07:34:14.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:14 smithi116 ceph-mon[32150]: osdmap e43: 7 total, 6 up, 7 in 2024-03-04T07:34:14.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:14 smithi152 ceph-mon[37154]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-04T07:34:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:14 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/317506943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2442f5fc-a522-4f08-8a68-c0bc903427b4"}]: dispatch 2024-03-04T07:34:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:14 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/317506943' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2442f5fc-a522-4f08-8a68-c0bc903427b4"}]': finished 2024-03-04T07:34:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:14 smithi152 ceph-mon[37154]: osdmap e43: 7 total, 6 up, 7 in 2024-03-04T07:34:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:14.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:14 smithi133 ceph-mon[36505]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-04T07:34:14.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:14 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/317506943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2442f5fc-a522-4f08-8a68-c0bc903427b4"}]: dispatch 2024-03-04T07:34:14.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:14 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/317506943' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2442f5fc-a522-4f08-8a68-c0bc903427b4"}]': finished 2024-03-04T07:34:14.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:14 smithi133 ceph-mon[36505]: osdmap e43: 7 total, 6 up, 7 in 2024-03-04T07:34:14.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:15.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2834868418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:34:15.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2834868418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:34:15.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2834868418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:34:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:16 smithi116 ceph-mon[32150]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T07:34:16.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:16 smithi152 ceph-mon[37154]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T07:34:16.834 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:16 smithi133 ceph-mon[36505]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T07:34:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:18 smithi116 ceph-mon[32150]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:18 smithi152 ceph-mon[37154]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:18.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:18 smithi133 ceph-mon[36505]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:20 smithi116 ceph-mon[32150]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:20.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:20 smithi152 ceph-mon[37154]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:20.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:20 smithi133 ceph-mon[36505]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:22.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:22 smithi116 ceph-mon[32150]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:22.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:22 smithi152 ceph-mon[37154]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:22.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:22 smithi133 ceph-mon[36505]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:23.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:23 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T07:34:23.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:23 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:23.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:23 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T07:34:23.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:23 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:23.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:23 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T07:34:23.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:23 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:24 smithi116 ceph-mon[32150]: Deploying daemon osd.6 on smithi116 2024-03-04T07:34:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:24 smithi116 ceph-mon[32150]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:24.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:24 smithi152 ceph-mon[37154]: Deploying daemon osd.6 on smithi116 2024-03-04T07:34:24.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:24 smithi152 ceph-mon[37154]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:24.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:24 smithi133 ceph-mon[36505]: Deploying daemon osd.6 on smithi116 2024-03-04T07:34:24.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:24 smithi133 ceph-mon[36505]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:26.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:26 smithi116 ceph-mon[32150]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:26.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:26 smithi152 ceph-mon[37154]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:26.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:26 smithi133 ceph-mon[36505]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:28.713 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:28 smithi116 ceph-mon[32150]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:28 smithi152 ceph-mon[37154]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:28.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:28 smithi133 ceph-mon[36505]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:30.708 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:30 smithi116 ceph-mon[32150]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:30.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:30 smithi152 ceph-mon[37154]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:30.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:30 smithi133 ceph-mon[36505]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:32 smithi152 ceph-mon[37154]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:32.801 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:32 smithi116 ceph-mon[32150]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:32.801 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:32.802 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:32.802 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:32.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:32 smithi133 ceph-mon[36505]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:32.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:32.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:32.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:34.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:34 smithi116 ceph-mon[32150]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:34.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:34 smithi152 ceph-mon[37154]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:34.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:34 smithi133 ceph-mon[36505]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:36.979 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 6 on host 'smithi116' 2024-03-04T07:34:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.977+0000 7fb4be000700 1 -- 172.21.15.116:0/3733916336 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb4c0062500 con 0x7fb4a80a1e10 2024-03-04T07:34:36.979 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:36 smithi116 ceph-mon[32150]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:36.979 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:36.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:36.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:36 smithi116 ceph-mon[32150]: from='osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T07:34:36.980 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:36 smithi116 ceph-mon[32150]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T07:34:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.982+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4a80a1e10 msgr2=0x7fb4a80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.982+0000 7fb4c8600700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4a80a1e10 0x7fb4a80a42d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00b2e0 tx=0x7fb4ac00b8f0 comp rx=0 tx=0).stop 2024-03-04T07:34:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0104330 msgr2=0x7fb4c019a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0104330 0x7fb4c019a550 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e9f0 tx=0x7fb4b400ed00 comp rx=0 tx=0).stop 2024-03-04T07:34:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 shutdown_connections 2024-03-04T07:34:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4a80a1e10 0x7fb4a80a42d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb4c0105ed0 0x7fb4c01a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4c0104330 0x7fb4c019a550 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 --2- 172.21.15.116:0/3733916336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4c0105530 0x7fb4c019aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 >> 172.21.15.116:0/3733916336 conn(0x7fb4c00ff8d0 msgr2=0x7fb4c0108900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:34:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 shutdown_connections 2024-03-04T07:34:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:34:36.983+0000 7fb4c8600700 1 -- 172.21.15.116:0/3733916336 wait complete. 2024-03-04T07:34:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:36 smithi152 ceph-mon[37154]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:36 smithi152 ceph-mon[37154]: from='osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T07:34:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:36 smithi152 ceph-mon[37154]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T07:34:37.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:36 smithi133 ceph-mon[36505]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:36 smithi133 ceph-mon[36505]: from='osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T07:34:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:36 smithi133 ceph-mon[36505]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T07:34:37.837 DEBUG:teuthology.orchestra.run.smithi116:osd.6> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.6.service 2024-03-04T07:34:37.841 INFO:tasks.cephadm:Deploying osd.7 on smithi133 with /dev/vg_nvme/lv_2... 2024-03-04T07:34:37.841 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T07:34:37.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:37 smithi116 ceph-mon[32150]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T07:34:37.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:37 smithi116 ceph-mon[32150]: from='osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:34:37.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:37 smithi116 ceph-mon[32150]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T07:34:37.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:37.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:37 smithi116 ceph-mon[32150]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:34:37.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:37.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:37 smithi152 ceph-mon[37154]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T07:34:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:37 smithi152 ceph-mon[37154]: from='osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:34:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:37 smithi152 ceph-mon[37154]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T07:34:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:37 smithi152 ceph-mon[37154]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:34:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:38.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:37 smithi133 ceph-mon[36505]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T07:34:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:37 smithi133 ceph-mon[36505]: from='osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:34:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:37 smithi133 ceph-mon[36505]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T07:34:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:37 smithi133 ceph-mon[36505]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:34:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:38.087 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:34:38.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:34:38.285+0000 7f7731400700 -1 osd.6 0 waiting for initial osdmap 2024-03-04T07:34:38.509 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:34:38.306+0000 7f7727200700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:34:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:38 smithi133 ceph-mon[36505]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:38 smithi133 ceph-mon[36505]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:34:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:38 smithi133 ceph-mon[36505]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T07:34:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:38 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:38 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:38.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:38 smithi133 ceph-mon[36505]: from='osd.6 ' entity='osd.6' 2024-03-04T07:34:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-mon[32150]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-mon[32150]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:34:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-mon[32150]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T07:34:39.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:38 smithi116 ceph-mon[32150]: from='osd.6 ' entity='osd.6' 2024-03-04T07:34:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:38 smithi152 ceph-mon[37154]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:38 smithi152 ceph-mon[37154]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:34:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:38 smithi152 ceph-mon[37154]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T07:34:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:38 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:38 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:38 smithi152 ceph-mon[37154]: from='osd.6 ' entity='osd.6' 2024-03-04T07:34:39.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:39 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:34:39.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:39 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:34:39.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:39 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:39 smithi116 ceph-mon[32150]: osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] boot 2024-03-04T07:34:39.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:39 smithi116 ceph-mon[32150]: osdmap e46: 7 total, 7 up, 7 in 2024-03-04T07:34:39.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:39 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:39 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:39 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:39 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:39 smithi133 ceph-mon[36505]: osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] boot 2024-03-04T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:39 smithi133 ceph-mon[36505]: osdmap e46: 7 total, 7 up, 7 in 2024-03-04T07:34:39.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:39 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:39.973 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T07:34:39.973 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T07:34:39.973 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2024-03-04T07:34:39.973 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2024-03-04T07:34:39.973 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113733 s, 922 MB/s 2024-03-04T07:34:39.973 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2024-03-04T07:34:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:39 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:34:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:39 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:34:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:39 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:39 smithi152 ceph-mon[37154]: osd.6 [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] boot 2024-03-04T07:34:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:39 smithi152 ceph-mon[37154]: osdmap e46: 7 total, 7 up, 7 in 2024-03-04T07:34:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:39 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T07:34:40.677 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi133:vg_nvme/lv_2 2024-03-04T07:34:40.919 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:34:40.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:40 smithi152 ceph-mon[37154]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:40 smithi116 ceph-mon[32150]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:41.040 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:40 smithi133 ceph-mon[36505]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-04T07:34:42.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:41 smithi116 ceph-mon[32150]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T07:34:42.020 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:41 smithi133 ceph-mon[36505]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T07:34:42.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:41 smithi152 ceph-mon[37154]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T07:34:42.616 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 -- 172.21.15.133:0/1821176079 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc103310 msgr2=0x7ff3dc103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/1821176079 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc103310 0x7ff3dc103770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80099c0 tx=0x7ff3d8009cd0 comp rx=0 tx=0).stop 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 -- 172.21.15.133:0/1821176079 shutdown_connections 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/1821176079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3dc102110 0x7ff3dc102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/1821176079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3dc103cb0 0x7ff3dc10a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/1821176079 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc103310 0x7ff3dc103770 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 -- 172.21.15.133:0/1821176079 >> 172.21.15.133:0/1821176079 conn(0x7ff3dc0fd6d0 msgr2=0x7ff3dc0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 -- 172.21.15.133:0/1821176079 shutdown_connections 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.615+0000 7ff3e3f97700 1 -- 172.21.15.133:0/1821176079 wait complete. 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 Processor -- start 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 -- start start 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc102110 0x7ff3dc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:42.617 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3dc103310 0x7ff3dc198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3dc103cb0 0x7ff3dc19fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc10ddf0 con 0x7ff3dc103cb0 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc10dc70 con 0x7ff3dc102110 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e3f97700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc10df70 con 0x7ff3dc103310 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e1d33700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc102110 0x7ff3dc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e1d33700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc102110 0x7ff3dc198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:36164/0 (socket says 172.21.15.133:36164) 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e1d33700 1 -- 172.21.15.133:0/3559336110 learned_addr learned my addr 172.21.15.133:0/3559336110 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.616+0000 7ff3e1532700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3dc103310 0x7ff3dc198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e1d33700 1 -- 172.21.15.133:0/3559336110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3dc103310 msgr2=0x7ff3dc198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e1d33700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3dc103310 0x7ff3dc198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e1d33700 1 -- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3dc103cb0 msgr2=0x7ff3dc19fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e1d33700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3dc103cb0 0x7ff3dc19fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e1d33700 1 -- 172.21.15.133:0/3559336110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc1a00a0 con 0x7ff3dc102110 2024-03-04T07:34:42.618 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e1d33700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc102110 0x7ff3dc198440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc00b500 tx=0x7ff3cc00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:34:42.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3d2ffd700 1 -- 172.21.15.133:0/3559336110 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc015020 con 0x7ff3dc102110 2024-03-04T07:34:42.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc1a0390 con 0x7ff3dc102110 2024-03-04T07:34:42.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3d2ffd700 1 -- 172.21.15.133:0/3559336110 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3cc011d10 con 0x7ff3dc102110 2024-03-04T07:34:42.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3d2ffd700 1 -- 172.21.15.133:0/3559336110 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc00f2e0 con 0x7ff3dc102110 2024-03-04T07:34:42.619 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.617+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc1a0950 con 0x7ff3dc102110 2024-03-04T07:34:42.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.618+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc068d80 con 0x7ff3dc102110 2024-03-04T07:34:42.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.619+0000 7ff3d2ffd700 1 -- 172.21.15.133:0/3559336110 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff3cc018020 con 0x7ff3dc102110 2024-03-04T07:34:42.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.620+0000 7ff3d2ffd700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff3c80a1e10 0x7ff3c80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:34:42.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.620+0000 7ff3d2ffd700 1 -- 172.21.15.133:0/3559336110 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7ff3cc0ccbd0 con 0x7ff3dc102110 2024-03-04T07:34:42.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.620+0000 7ff3e1532700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff3c80a1e10 0x7ff3c80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:34:42.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.621+0000 7ff3e1532700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff3c80a1e10 0x7ff3c80a42d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8000c00 tx=0x7ff3d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:34:42.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.623+0000 7ff3d2ffd700 1 -- 172.21.15.133:0/3559336110 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff3cc01f020 con 0x7ff3dc102110 2024-03-04T07:34:42.826 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:34:42.826+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3dc062500 con 0x7ff3c80a1e10 2024-03-04T07:34:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:42 smithi116 ceph-mon[32150]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:42 smithi152 ceph-mon[37154]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:43.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:42 smithi133 ceph-mon[36505]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:43.743 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:34:43.743 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:43.744 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:43.745 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:44.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:44.010 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:44.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:44.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:44.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:34:44.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:44.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:34:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:44 smithi116 ceph-mon[32150]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:34:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:44 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi116 2024-03-04T07:34:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:44 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi116 to 5697M 2024-03-04T07:34:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:44 smithi116 ceph-mon[32150]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:45.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:44 smithi152 ceph-mon[37154]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:34:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:44 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi116 2024-03-04T07:34:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:44 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi116 to 5697M 2024-03-04T07:34:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:44 smithi152 ceph-mon[37154]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:44 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:45.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:44 smithi133 ceph-mon[36505]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:34:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:44 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi116 2024-03-04T07:34:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:44 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi116 to 5697M 2024-03-04T07:34:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:44 smithi133 ceph-mon[36505]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:44 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:34:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:45 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/4063818403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]: dispatch 2024-03-04T07:34:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:45 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]: dispatch 2024-03-04T07:34:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:45 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]': finished 2024-03-04T07:34:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:45 smithi116 ceph-mon[32150]: osdmap e48: 8 total, 7 up, 8 in 2024-03-04T07:34:46.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:34:46.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:45 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/4063818403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]: dispatch 2024-03-04T07:34:46.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:45 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]: dispatch 2024-03-04T07:34:46.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:45 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]': finished 2024-03-04T07:34:46.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:45 smithi152 ceph-mon[37154]: osdmap e48: 8 total, 7 up, 8 in 2024-03-04T07:34:46.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:34:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:45 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/4063818403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]: dispatch 2024-03-04T07:34:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:45 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]: dispatch 2024-03-04T07:34:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:45 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f1dbc539-c51f-4ae3-9239-8c032ef60b4f"}]': finished 2024-03-04T07:34:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:45 smithi133 ceph-mon[36505]: osdmap e48: 8 total, 7 up, 8 in 2024-03-04T07:34:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:34:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:46 smithi116 ceph-mon[32150]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:46 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/3743409858' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:34:47.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:46 smithi152 ceph-mon[37154]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:47.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:46 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/3743409858' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:34:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:46 smithi133 ceph-mon[36505]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:46 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/3743409858' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:34:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:48 smithi116 ceph-mon[32150]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:49.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:48 smithi152 ceph-mon[37154]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:49.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:48 smithi133 ceph-mon[36505]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:50.909 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:50 smithi133 ceph-mon[36505]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:50 smithi116 ceph-mon[32150]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:50 smithi152 ceph-mon[37154]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:52.970 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:52 smithi133 ceph-mon[36505]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:52 smithi116 ceph-mon[32150]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:53.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:52 smithi152 ceph-mon[37154]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:53.990 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T07:34:53.990 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T07:34:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:54.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T07:34:54.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:34:54.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:54 smithi133 ceph-mon[36505]: Deploying daemon osd.7 on smithi133 2024-03-04T07:34:54.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:54 smithi133 ceph-mon[36505]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:54 smithi116 ceph-mon[32150]: Deploying daemon osd.7 on smithi133 2024-03-04T07:34:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:54 smithi116 ceph-mon[32150]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:55.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:54 smithi152 ceph-mon[37154]: Deploying daemon osd.7 on smithi133 2024-03-04T07:34:55.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:54 smithi152 ceph-mon[37154]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:56 smithi116 ceph-mon[32150]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:57.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:56 smithi152 ceph-mon[37154]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:57.039 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:56 smithi133 ceph-mon[36505]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:58.936 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:58 smithi133 ceph-mon[36505]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:58 smithi116 ceph-mon[32150]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:34:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:58 smithi152 ceph-mon[37154]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:59 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:59 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:00.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:34:59 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:00.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:59 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:00.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:59 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:00.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:34:59 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:00.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:59 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:00.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:59 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:00.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:34:59 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:01.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:00 smithi152 ceph-mon[37154]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:00 smithi133 ceph-mon[36505]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:00 smithi116 ceph-mon[32150]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:02.024 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:01 smithi133 ceph-mon[36505]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:01 smithi116 ceph-mon[32150]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:02.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:01 smithi152 ceph-mon[37154]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:02.412 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.411+0000 7ff3d2ffd700 1 -- 172.21.15.133:0/3559336110 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff3dc062500 con 0x7ff3c80a1e10 2024-03-04T07:35:02.412 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 7 on host 'smithi133' 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff3c80a1e10 msgr2=0x7ff3c80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff3c80a1e10 0x7ff3c80a42d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8000c00 tx=0x7ff3d8005fb0 comp rx=0 tx=0).stop 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc102110 msgr2=0x7ff3dc198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc102110 0x7ff3dc198440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc00b500 tx=0x7ff3cc00b810 comp rx=0 tx=0).stop 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 shutdown_connections 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3dc103cb0 0x7ff3dc19fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff3c80a1e10 0x7ff3c80a42d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff3dc102110 0x7ff3dc198440 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:02.416 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 --2- 172.21.15.133:0/3559336110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3dc103310 0x7ff3dc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:02.417 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 >> 172.21.15.133:0/3559336110 conn(0x7ff3dc0fd6d0 msgr2=0x7ff3dc106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:35:02.417 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 shutdown_connections 2024-03-04T07:35:02.417 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:35:02.415+0000 7ff3e3f97700 1 -- 172.21.15.133:0/3559336110 wait complete. 2024-03-04T07:35:03.081 DEBUG:teuthology.orchestra.run.smithi133:osd.7> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.7.service 2024-03-04T07:35:03.084 INFO:tasks.cephadm:Deploying osd.8 on smithi152 with /dev/vg_nvme/lv_2... 2024-03-04T07:35:03.084 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T07:35:03.336 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:35:03.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:03 smithi133 ceph-mon[36505]: from='osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T07:35:03.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:03 smithi133 ceph-mon[36505]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T07:35:03.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.673 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.673 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.673 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:03 smithi152 ceph-mon[37154]: from='osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T07:35:03.673 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:03 smithi152 ceph-mon[37154]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T07:35:03.674 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.674 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:03 smithi116 ceph-mon[32150]: from='osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T07:35:03.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:03 smithi116 ceph-mon[32150]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T07:35:03.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:03.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:04.656 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-mon[36505]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T07:35:04.657 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-mon[36505]: from='osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:35:04.657 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-mon[36505]: osdmap e49: 8 total, 7 up, 8 in 2024-03-04T07:35:04.657 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:04.657 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-mon[36505]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:35:04.657 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-mon[36505]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:04.657 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:35:04.392+0000 7fc5a07e2700 -1 osd.7 0 waiting for initial osdmap 2024-03-04T07:35:04.657 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:35:04 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:35:04.403+0000 7fc59c5d5700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:35:04.758 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:04 smithi152 ceph-mon[37154]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T07:35:04.758 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:04 smithi152 ceph-mon[37154]: from='osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:35:04.758 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:04 smithi152 ceph-mon[37154]: osdmap e49: 8 total, 7 up, 8 in 2024-03-04T07:35:04.758 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:04 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:04.758 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:04 smithi152 ceph-mon[37154]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:35:04.758 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:04 smithi152 ceph-mon[37154]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:04.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:04 smithi116 ceph-mon[32150]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T07:35:04.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:04 smithi116 ceph-mon[32150]: from='osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:35:04.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:04 smithi116 ceph-mon[32150]: osdmap e49: 8 total, 7 up, 8 in 2024-03-04T07:35:04.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:04 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:04.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:04 smithi116 ceph-mon[32150]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:35:04.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:04 smithi116 ceph-mon[32150]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-04T07:35:05.299 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T07:35:05.299 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T07:35:05.299 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-04T07:35:05.299 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-04T07:35:05.299 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114095 s, 919 MB/s 2024-03-04T07:35:05.300 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-04T07:35:05.639 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:05 smithi152 ceph-mon[37154]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:35:05.640 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:05 smithi152 ceph-mon[37154]: osdmap e50: 8 total, 7 up, 8 in 2024-03-04T07:35:05.640 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:05.640 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:05 smithi152 ceph-mon[37154]: osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] boot 2024-03-04T07:35:05.640 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:05 smithi152 ceph-mon[37154]: osdmap e51: 8 total, 8 up, 8 in 2024-03-04T07:35:05.640 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:05 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:05 smithi116 ceph-mon[32150]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:35:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:05 smithi116 ceph-mon[32150]: osdmap e50: 8 total, 7 up, 8 in 2024-03-04T07:35:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:05 smithi116 ceph-mon[32150]: osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] boot 2024-03-04T07:35:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:05 smithi116 ceph-mon[32150]: osdmap e51: 8 total, 8 up, 8 in 2024-03-04T07:35:05.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:05 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:05.801 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:05 smithi133 ceph-mon[36505]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:35:05.801 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:05 smithi133 ceph-mon[36505]: osdmap e50: 8 total, 7 up, 8 in 2024-03-04T07:35:05.801 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:05.801 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:05 smithi133 ceph-mon[36505]: osd.7 [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] boot 2024-03-04T07:35:05.801 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:05 smithi133 ceph-mon[36505]: osdmap e51: 8 total, 8 up, 8 in 2024-03-04T07:35:05.801 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:05 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T07:35:05.947 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_2 2024-03-04T07:35:06.185 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: pgmap v195: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:06.632 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:06 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: pgmap v195: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:06.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:06.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:06 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: pgmap v195: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:06.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:06.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:06.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:06.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:06 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:07.679 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi133 2024-03-04T07:35:07.679 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi133 to 5693M 2024-03-04T07:35:07.679 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: osdmap e52: 8 total, 8 up, 8 in 2024-03-04T07:35:07.680 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-04T07:35:07.680 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:07.680 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:07.680 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:07.680 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:07.680 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:07 smithi152 ceph-mon[37154]: osdmap e53: 8 total, 8 up, 8 in 2024-03-04T07:35:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi133 2024-03-04T07:35:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi133 to 5693M 2024-03-04T07:35:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: osdmap e52: 8 total, 8 up, 8 in 2024-03-04T07:35:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-04T07:35:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:07.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:07 smithi116 ceph-mon[32150]: osdmap e53: 8 total, 8 up, 8 in 2024-03-04T07:35:07.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi133 2024-03-04T07:35:07.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi133 to 5693M 2024-03-04T07:35:07.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: osdmap e52: 8 total, 8 up, 8 in 2024-03-04T07:35:07.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-04T07:35:07.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:07.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:07.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:07.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:07.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:07 smithi133 ceph-mon[36505]: osdmap e53: 8 total, 8 up, 8 in 2024-03-04T07:35:07.918 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 -- 172.21.15.152:0/2736066099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958102b90 msgr2=0x7f0958109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:07.918 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 --2- 172.21.15.152:0/2736066099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958102b90 0x7f0958109440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f095000b0a0 tx=0x7f095000b3b0 comp rx=0 tx=0).stop 2024-03-04T07:35:07.918 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 -- 172.21.15.152:0/2736066099 shutdown_connections 2024-03-04T07:35:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 --2- 172.21.15.152:0/2736066099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958102b90 0x7f0958109440 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 --2- 172.21.15.152:0/2736066099 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0958100ff0 0x7f0958101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 --2- 172.21.15.152:0/2736066099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09581021f0 0x7f0958102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 -- 172.21.15.152:0/2736066099 >> 172.21.15.152:0/2736066099 conn(0x7f09580fc590 msgr2=0x7f09580fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:35:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 -- 172.21.15.152:0/2736066099 shutdown_connections 2024-03-04T07:35:07.919 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.918+0000 7f095c94e700 1 -- 172.21.15.152:0/2736066099 wait complete. 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 Processor -- start 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 -- start start 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958100ff0 0x7f0958104f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09581021f0 0x7f09581035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0958102b90 0x7f0958103b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095810ccd0 con 0x7f0958102b90 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095810cb50 con 0x7f09581021f0 2024-03-04T07:35:07.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095c94e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095810ce50 con 0x7f0958100ff0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958100ff0 0x7f0958104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958100ff0 0x7f0958104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:54514/0 (socket says 172.21.15.152:54514) 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 -- 172.21.15.152:0/3706018742 learned_addr learned my addr 172.21.15.152:0/3706018742 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f0955d9b700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09581021f0 0x7f09581035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 -- 172.21.15.152:0/3706018742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09581021f0 msgr2=0x7f09581035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09581021f0 0x7f09581035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 -- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0958102b90 msgr2=0x7f0958103b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0958102b90 0x7f0958103b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.919+0000 7f095659c700 1 -- 172.21.15.152:0/3706018742 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09581043c0 con 0x7f0958100ff0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.920+0000 7f095659c700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958100ff0 0x7f0958104f80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f094c00b500 tx=0x7f094c00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.920+0000 7f09477fe700 1 -- 172.21.15.152:0/3706018742 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c015020 con 0x7f0958100ff0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.920+0000 7f09477fe700 1 -- 172.21.15.152:0/3706018742 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f094c011d10 con 0x7f0958100ff0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.920+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09581a2610 con 0x7f0958100ff0 2024-03-04T07:35:07.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.920+0000 7f09477fe700 1 -- 172.21.15.152:0/3706018742 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c010e50 con 0x7f0958100ff0 2024-03-04T07:35:07.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.920+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09581a2b50 con 0x7f0958100ff0 2024-03-04T07:35:07.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.921+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0958068d80 con 0x7f0958100ff0 2024-03-04T07:35:07.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.922+0000 7f09477fe700 1 -- 172.21.15.152:0/3706018742 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f094c018020 con 0x7f0958100ff0 2024-03-04T07:35:07.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.923+0000 7f09477fe700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1fb0 0x7f09400a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:07.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.923+0000 7f09477fe700 1 -- 172.21.15.152:0/3706018742 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f094c0ccf10 con 0x7f0958100ff0 2024-03-04T07:35:07.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.923+0000 7f0955d9b700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1fb0 0x7f09400a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:35:07.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.924+0000 7f0955d9b700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1fb0 0x7f09400a4470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0948009a10 tx=0x7f0948009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:35:07.926 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:07.926+0000 7f09477fe700 1 -- 172.21.15.152:0/3706018742 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f094c01f020 con 0x7f0958100ff0 2024-03-04T07:35:08.126 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:08.126+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f09580fe790 con 0x7f09400a1fb0 2024-03-04T07:35:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:08 smithi116 ceph-mon[32150]: pgmap v198: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:35:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:35:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:08.759 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:08 smithi152 ceph-mon[37154]: pgmap v198: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:08.760 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:35:08.760 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:35:08.760 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:08.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:08 smithi133 ceph-mon[36505]: pgmap v198: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:35:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:35:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:09.714 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:09 smithi152 ceph-mon[37154]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:35:09.714 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:09 smithi152 ceph-mon[37154]: osdmap e54: 8 total, 8 up, 8 in 2024-03-04T07:35:09.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:09 smithi116 ceph-mon[32150]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:35:09.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:09 smithi116 ceph-mon[32150]: osdmap e54: 8 total, 8 up, 8 in 2024-03-04T07:35:09.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:09 smithi133 ceph-mon[36505]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:35:09.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:09 smithi133 ceph-mon[36505]: osdmap e54: 8 total, 8 up, 8 in 2024-03-04T07:35:10.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:10 smithi116 ceph-mon[32150]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:10.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:10 smithi116 ceph-mon[32150]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T07:35:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:10 smithi152 ceph-mon[37154]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:10 smithi152 ceph-mon[37154]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T07:35:10.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:10 smithi133 ceph-mon[36505]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:10.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:10 smithi133 ceph-mon[36505]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T07:35:11.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:11 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/686630756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]: dispatch 2024-03-04T07:35:11.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:11 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]: dispatch 2024-03-04T07:35:11.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:11 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]': finished 2024-03-04T07:35:11.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:11 smithi152 ceph-mon[37154]: osdmap e56: 9 total, 8 up, 9 in 2024-03-04T07:35:11.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:11 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:11.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/686630756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]: dispatch 2024-03-04T07:35:11.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:11 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]: dispatch 2024-03-04T07:35:11.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:11 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]': finished 2024-03-04T07:35:11.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:11 smithi116 ceph-mon[32150]: osdmap e56: 9 total, 8 up, 9 in 2024-03-04T07:35:11.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:11 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:11.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:11 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/686630756' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]: dispatch 2024-03-04T07:35:11.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:11 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]: dispatch 2024-03-04T07:35:11.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:11 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9bc6819-8f22-4e6d-83e1-96690fc82524"}]': finished 2024-03-04T07:35:11.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:11 smithi133 ceph-mon[36505]: osdmap e56: 9 total, 8 up, 9 in 2024-03-04T07:35:11.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:11 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:12 smithi116 ceph-mon[32150]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:12 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/265934328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:35:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:12 smithi116 ceph-mon[32150]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-04T07:35:12.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:12 smithi116 ceph-mon[32150]: Cluster is now healthy 2024-03-04T07:35:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:12 smithi152 ceph-mon[37154]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:12 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/265934328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:35:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:12 smithi152 ceph-mon[37154]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-04T07:35:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:12 smithi152 ceph-mon[37154]: Cluster is now healthy 2024-03-04T07:35:12.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:12 smithi133 ceph-mon[36505]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:12.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:12 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/265934328' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:35:12.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:12 smithi133 ceph-mon[36505]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-04T07:35:12.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:12 smithi133 ceph-mon[36505]: Cluster is now healthy 2024-03-04T07:35:14.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:14 smithi116 ceph-mon[32150]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:14.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:14 smithi152 ceph-mon[37154]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:14.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:14 smithi133 ceph-mon[36505]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:16 smithi116 ceph-mon[32150]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:16.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:16 smithi152 ceph-mon[37154]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:16.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:16 smithi133 ceph-mon[36505]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:18 smithi116 ceph-mon[32150]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-04T07:35:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:18.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:18 smithi152 ceph-mon[37154]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-04T07:35:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:18.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:18 smithi133 ceph-mon[36505]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:18.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-04T07:35:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:19 smithi116 ceph-mon[32150]: Deploying daemon osd.8 on smithi152 2024-03-04T07:35:19.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:19 smithi152 ceph-mon[37154]: Deploying daemon osd.8 on smithi152 2024-03-04T07:35:19.835 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:19 smithi133 ceph-mon[36505]: Deploying daemon osd.8 on smithi152 2024-03-04T07:35:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:20 smithi116 ceph-mon[32150]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:20.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:20 smithi152 ceph-mon[37154]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:20.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:20 smithi133 ceph-mon[36505]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:22.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:22 smithi116 ceph-mon[32150]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:22.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:22 smithi152 ceph-mon[37154]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:22.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:22 smithi133 ceph-mon[36505]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:24.690 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:24 smithi152 ceph-mon[37154]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:24.690 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:24.690 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:24.690 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:24 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:24 smithi116 ceph-mon[32150]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:24 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:24.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:24 smithi133 ceph-mon[36505]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:24.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:24.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:24.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:24 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:26.747 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:26 smithi152 ceph-mon[37154]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:26.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:26 smithi116 ceph-mon[32150]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:26.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:26 smithi133 ceph-mon[36505]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:27.460 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 8 on host 'smithi152' 2024-03-04T07:35:27.460 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.459+0000 7f09477fe700 1 -- 172.21.15.152:0/3706018742 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f09580fe790 con 0x7f09400a1fb0 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1fb0 msgr2=0x7f09400a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1fb0 0x7f09400a4470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0948009a10 tx=0x7f0948009190 comp rx=0 tx=0).stop 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958100ff0 msgr2=0x7f0958104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958100ff0 0x7f0958104f80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f094c00b500 tx=0x7f094c00b810 comp rx=0 tx=0).stop 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 shutdown_connections 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0958102b90 0x7f0958103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1fb0 0x7f09400a4470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0958100ff0 0x7f0958104f80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 --2- 172.21.15.152:0/3706018742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09581021f0 0x7f09581035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:27.462 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 >> 172.21.15.152:0/3706018742 conn(0x7f09580fc590 msgr2=0x7f09580fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:35:27.463 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 shutdown_connections 2024-03-04T07:35:27.463 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:35:27.462+0000 7f095c94e700 1 -- 172.21.15.152:0/3706018742 wait complete. 2024-03-04T07:35:28.114 DEBUG:teuthology.orchestra.run.smithi152:osd.8> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.8.service 2024-03-04T07:35:28.117 INFO:tasks.cephadm:Deploying osd.9 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-04T07:35:28.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T07:35:28.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:28 smithi116 ceph-mon[32150]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:28.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:28 smithi116 ceph-mon[32150]: from='osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T07:35:28.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:28 smithi116 ceph-mon[32150]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T07:35:28.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.577 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:35:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:28 smithi152 ceph-mon[37154]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:28 smithi152 ceph-mon[37154]: from='osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T07:35:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:28 smithi152 ceph-mon[37154]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T07:35:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:28 smithi133 ceph-mon[36505]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:28 smithi133 ceph-mon[36505]: from='osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T07:35:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:28 smithi133 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T07:35:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:29.752 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:29 smithi116 ceph-mon[32150]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-04T07:35:29.752 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:29 smithi116 ceph-mon[32150]: from='osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:35:29.752 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:29 smithi116 ceph-mon[32150]: osdmap e57: 9 total, 8 up, 9 in 2024-03-04T07:35:29.753 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:29.753 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:29 smithi116 ceph-mon[32150]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:35:29.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:29 smithi152 ceph-mon[37154]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-04T07:35:29.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:29 smithi152 ceph-mon[37154]: from='osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:35:29.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:29 smithi152 ceph-mon[37154]: osdmap e57: 9 total, 8 up, 9 in 2024-03-04T07:35:29.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:29.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:29 smithi152 ceph-mon[37154]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:35:29.756 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:35:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:35:29.496+0000 7fb699d05700 -1 osd.8 0 waiting for initial osdmap 2024-03-04T07:35:29.756 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:35:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:35:29.507+0000 7fb695af8700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:35:29.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:29 smithi133 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-04T07:35:29.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:29 smithi133 ceph-mon[36505]: from='osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:35:29.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:29 smithi133 ceph-mon[36505]: osdmap e57: 9 total, 8 up, 9 in 2024-03-04T07:35:29.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:29.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:29 smithi133 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:35:30.671 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:30 smithi116 ceph-mon[32150]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:30.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:30 smithi116 ceph-mon[32150]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:35:30.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:30 smithi116 ceph-mon[32150]: osdmap e58: 9 total, 8 up, 9 in 2024-03-04T07:35:30.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:30.672 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:30.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:30 smithi152 ceph-mon[37154]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:30.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:30 smithi152 ceph-mon[37154]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:35:30.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:30 smithi152 ceph-mon[37154]: osdmap e58: 9 total, 8 up, 9 in 2024-03-04T07:35:30.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:30.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:30.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:30 smithi133 ceph-mon[36505]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-04T07:35:30.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:30 smithi133 ceph-mon[36505]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:35:30.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:30 smithi133 ceph-mon[36505]: osdmap e58: 9 total, 8 up, 9 in 2024-03-04T07:35:30.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:30.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:31.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:31 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:35:31.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:31 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:35:31.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:31 smithi152 ceph-mon[37154]: osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] boot 2024-03-04T07:35:31.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:31 smithi152 ceph-mon[37154]: osdmap e59: 9 total, 9 up, 9 in 2024-03-04T07:35:31.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:31 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:31.563 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T07:35:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T07:35:31.564 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-04T07:35:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-04T07:35:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0349304 s, 300 MB/s 2024-03-04T07:35:31.564 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-04T07:35:31.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:31 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:35:31.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:31 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:35:31.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:31 smithi116 ceph-mon[32150]: osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] boot 2024-03-04T07:35:31.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:31 smithi116 ceph-mon[32150]: osdmap e59: 9 total, 9 up, 9 in 2024-03-04T07:35:31.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:31.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:31 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:35:31.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:31 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:35:31.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:31 smithi133 ceph-mon[36505]: osd.8 [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] boot 2024-03-04T07:35:31.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:31 smithi133 ceph-mon[36505]: osdmap e59: 9 total, 9 up, 9 in 2024-03-04T07:35:31.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:31 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T07:35:32.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-04T07:35:32.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:32.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: osdmap e60: 9 total, 9 up, 9 in 2024-03-04T07:35:32.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi152 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi152 to 5693M 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:32.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:32.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: osdmap e60: 9 total, 9 up, 9 in 2024-03-04T07:35:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi152 2024-03-04T07:35:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi152 to 5693M 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:32.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:32.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: osdmap e60: 9 total, 9 up, 9 in 2024-03-04T07:35:32.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi152 2024-03-04T07:35:32.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi152 to 5693M 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:35:32.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:32.968 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:35:33.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:33 smithi116 ceph-mon[32150]: osdmap e61: 9 total, 9 up, 9 in 2024-03-04T07:35:33.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:33 smithi133 ceph-mon[36505]: osdmap e61: 9 total, 9 up, 9 in 2024-03-04T07:35:34.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:33 smithi152 ceph-mon[37154]: osdmap e61: 9 total, 9 up, 9 in 2024-03-04T07:35:34.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:34 smithi116 ceph-mon[32150]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:34.717 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:34 smithi116 ceph-mon[32150]: osdmap e62: 9 total, 9 up, 9 in 2024-03-04T07:35:34.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:34 smithi133 ceph-mon[36505]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:34.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:34 smithi133 ceph-mon[36505]: osdmap e62: 9 total, 9 up, 9 in 2024-03-04T07:35:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:34 smithi152 ceph-mon[37154]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:34 smithi152 ceph-mon[37154]: osdmap e62: 9 total, 9 up, 9 in 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.259+0000 7f7253a00700 1 -- 172.21.15.116:0/2631824957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 msgr2=0x7f724c10c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.259+0000 7f7253a00700 1 --2- 172.21.15.116:0/2631824957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 0x7f724c10c790 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f72440099c0 tx=0x7f7244009cd0 comp rx=0 tx=0).stop 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.259+0000 7f7253a00700 1 -- 172.21.15.116:0/2631824957 shutdown_connections 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.259+0000 7f7253a00700 1 --2- 172.21.15.116:0/2631824957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 0x7f724c10c790 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.259+0000 7f7253a00700 1 --2- 172.21.15.116:0/2631824957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c104340 0x7f724c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.259+0000 7f7253a00700 1 --2- 172.21.15.116:0/2631824957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f724c105540 0x7f724c1059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.259+0000 7f7253a00700 1 -- 172.21.15.116:0/2631824957 >> 172.21.15.116:0/2631824957 conn(0x7f724c0ff8e0 msgr2=0x7f724c101d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.260+0000 7f7253a00700 1 -- 172.21.15.116:0/2631824957 shutdown_connections 2024-03-04T07:35:35.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.260+0000 7f7253a00700 1 -- 172.21.15.116:0/2631824957 wait complete. 2024-03-04T07:35:35.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 Processor -- start 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 -- start start 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f724c104340 0x7f724c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c105540 0x7f724c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 0x7f724c1a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c10ff30 con 0x7f724c105ee0 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c10fdb0 con 0x7f724c105540 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.261+0000 7f7253a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c1100b0 con 0x7f724c104340 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7251a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 0x7f724c1a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c105540 0x7f724c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7251a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 0x7f724c1a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52082/0 (socket says 172.21.15.116:52082) 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7251a00700 1 -- 172.21.15.116:0/503691703 learned_addr learned my addr 172.21.15.116:0/503691703 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c105540 0x7f724c19a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:39522/0 (socket says 172.21.15.116:39522) 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7251000700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f724c104340 0x7f724c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:35:35.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 -- 172.21.15.116:0/503691703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f724c104340 msgr2=0x7f724c19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:35.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f724c104340 0x7f724c19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:35.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 -- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 msgr2=0x7f724c1a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:35:35.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 0x7f724c1a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:35:35.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 -- 172.21.15.116:0/503691703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c1a20e0 con 0x7f724c105540 2024-03-04T07:35:35.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.262+0000 7f7243e00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c105540 0x7f724c19a9c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f723800e9f0 tx=0x7f723800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:35:35.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.263+0000 7f724a000700 1 -- 172.21.15.116:0/503691703 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238004220 con 0x7f724c105540 2024-03-04T07:35:35.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.263+0000 7f724a000700 1 -- 172.21.15.116:0/503691703 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7238004d10 con 0x7f724c105540 2024-03-04T07:35:35.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.263+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724c1a23d0 con 0x7f724c105540 2024-03-04T07:35:35.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.263+0000 7f724a000700 1 -- 172.21.15.116:0/503691703 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238005020 con 0x7f724c105540 2024-03-04T07:35:35.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.263+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724c1a28c0 con 0x7f724c105540 2024-03-04T07:35:35.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.264+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c068d80 con 0x7f724c105540 2024-03-04T07:35:35.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.266+0000 7f724a000700 1 -- 172.21.15.116:0/503691703 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f7238004750 con 0x7f724c105540 2024-03-04T07:35:35.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.267+0000 7f724a000700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f72300a1e90 0x7f72300a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:35:35.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.267+0000 7f724a000700 1 -- 172.21.15.116:0/503691703 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f72380cd3f0 con 0x7f724c105540 2024-03-04T07:35:35.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.267+0000 7f7251000700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f72300a1e90 0x7f72300a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:35:35.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.268+0000 7f7251000700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f72300a1e90 0x7f72300a4350 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f724c1053a0 tx=0x7f723c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:35:35.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.270+0000 7f724a000700 1 -- 172.21.15.116:0/503691703 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f723809b1c0 con 0x7f724c105540 2024-03-04T07:35:35.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:35:35.503+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f724c062500 con 0x7f72300a1e90 2024-03-04T07:35:35.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:35 smithi116 ceph-mon[32150]: osdmap e63: 9 total, 9 up, 9 in 2024-03-04T07:35:35.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:35 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:35:35.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:35 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:35:35.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:35 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:35.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:35 smithi133 ceph-mon[36505]: osdmap e63: 9 total, 9 up, 9 in 2024-03-04T07:35:35.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:35 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:35:35.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:35 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:35:35.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:35 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:36.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:35 smithi152 ceph-mon[37154]: osdmap e63: 9 total, 9 up, 9 in 2024-03-04T07:35:36.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:35 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:35:36.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:35 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:35:36.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:35 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:36.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:36 smithi116 ceph-mon[32150]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:35:36.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:36 smithi116 ceph-mon[32150]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:35:36.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:36 smithi133 ceph-mon[36505]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:35:36.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:36 smithi133 ceph-mon[36505]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:35:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:36 smithi152 ceph-mon[37154]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:35:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:36 smithi152 ceph-mon[37154]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:35:38.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:38 smithi116 ceph-mon[32150]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-04T07:35:38.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:38 smithi133 ceph-mon[36505]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-04T07:35:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:38 smithi152 ceph-mon[37154]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-04T07:35:40.823 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:40 smithi152 ceph-mon[37154]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-04T07:35:40.824 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:40 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2892014552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2931b69e-19e6-4739-8f27-b1d0b83f96d5"}]: dispatch 2024-03-04T07:35:40.824 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:40 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2892014552' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2931b69e-19e6-4739-8f27-b1d0b83f96d5"}]': finished 2024-03-04T07:35:40.824 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:40 smithi152 ceph-mon[37154]: osdmap e64: 10 total, 9 up, 10 in 2024-03-04T07:35:40.824 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:35:40.824 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:40 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2817590977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:35:40.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:40 smithi133 ceph-mon[36505]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-04T07:35:40.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:40 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2892014552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2931b69e-19e6-4739-8f27-b1d0b83f96d5"}]: dispatch 2024-03-04T07:35:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:40 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2892014552' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2931b69e-19e6-4739-8f27-b1d0b83f96d5"}]': finished 2024-03-04T07:35:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:40 smithi133 ceph-mon[36505]: osdmap e64: 10 total, 9 up, 10 in 2024-03-04T07:35:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:35:40.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:40 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2817590977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:35:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:40 smithi116 ceph-mon[32150]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-04T07:35:41.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:40 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2892014552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2931b69e-19e6-4739-8f27-b1d0b83f96d5"}]: dispatch 2024-03-04T07:35:41.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:40 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2892014552' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2931b69e-19e6-4739-8f27-b1d0b83f96d5"}]': finished 2024-03-04T07:35:41.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:40 smithi116 ceph-mon[32150]: osdmap e64: 10 total, 9 up, 10 in 2024-03-04T07:35:41.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:35:41.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:40 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2817590977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:35:42.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:42 smithi133 ceph-mon[36505]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-04T07:35:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:42 smithi116 ceph-mon[32150]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-04T07:35:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:42 smithi152 ceph-mon[37154]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-04T07:35:44.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:44 smithi133 ceph-mon[36505]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T07:35:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:44 smithi116 ceph-mon[32150]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T07:35:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:44 smithi152 ceph-mon[37154]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T07:35:46.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:46 smithi133 ceph-mon[36505]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:46.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:46 smithi116 ceph-mon[32150]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:47.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:46 smithi152 ceph-mon[37154]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:48.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:48 smithi133 ceph-mon[36505]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:48 smithi116 ceph-mon[32150]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:49.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:48 smithi152 ceph-mon[37154]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:49.831 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:49 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-04T07:35:49.831 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:49 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:49 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-04T07:35:50.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:49 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:50.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:49 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-04T07:35:50.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:49 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:35:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:50 smithi116 ceph-mon[32150]: Deploying daemon osd.9 on smithi116 2024-03-04T07:35:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:50 smithi116 ceph-mon[32150]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:50.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:50 smithi133 ceph-mon[36505]: Deploying daemon osd.9 on smithi116 2024-03-04T07:35:50.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:50 smithi133 ceph-mon[36505]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:50 smithi152 ceph-mon[37154]: Deploying daemon osd.9 on smithi116 2024-03-04T07:35:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:50 smithi152 ceph-mon[37154]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:52 smithi116 ceph-mon[32150]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:53.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:52 smithi152 ceph-mon[37154]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:53.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:52 smithi133 ceph-mon[36505]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:54.741 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:54 smithi116 ceph-mon[32150]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:55.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:54 smithi152 ceph-mon[37154]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:55.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:54 smithi133 ceph-mon[36505]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:56.749 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:56 smithi116 ceph-mon[32150]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:57.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:56 smithi152 ceph-mon[37154]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:57.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:56 smithi133 ceph-mon[36505]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:58.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:58 smithi116 ceph-mon[32150]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:58.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:58.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:58.950 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:35:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:58 smithi152 ceph-mon[37154]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:35:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:35:59.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:58 smithi133 ceph-mon[36505]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:35:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:35:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:35:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:00.928 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:00 smithi116 ceph-mon[32150]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:01.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:00 smithi152 ceph-mon[37154]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:01.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:00 smithi133 ceph-mon[36505]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:02.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:01 smithi116 ceph-mon[32150]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-04T07:36:02.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:01 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:02.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:01 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:02.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:01 smithi152 ceph-mon[37154]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-04T07:36:02.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:01 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:02.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:01 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:02.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:01 smithi133 ceph-mon[36505]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-04T07:36:02.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:01 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:02.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:01 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:02.763 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 9 on host 'smithi116' 2024-03-04T07:36:02.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.761+0000 7f724a000700 1 -- 172.21.15.116:0/503691703 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f724c062500 con 0x7f72300a1e90 2024-03-04T07:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f72300a1e90 msgr2=0x7f72300a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f72300a1e90 0x7f72300a4350 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f724c1053a0 tx=0x7f723c00b040 comp rx=0 tx=0).stop 2024-03-04T07:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c105540 msgr2=0x7f724c19a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c105540 0x7f724c19a9c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f723800e9f0 tx=0x7f723800ed00 comp rx=0 tx=0).stop 2024-03-04T07:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 shutdown_connections 2024-03-04T07:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f724c105ee0 0x7f724c1a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:02.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f72300a1e90 0x7f72300a4350 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f724c104340 0x7f724c19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 --2- 172.21.15.116:0/503691703 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f724c105540 0x7f724c19a9c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 >> 172.21.15.116:0/503691703 conn(0x7f724c0ff8e0 msgr2=0x7f724c108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.766+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 shutdown_connections 2024-03-04T07:36:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:02.767+0000 7f7253a00700 1 -- 172.21.15.116:0/503691703 wait complete. 2024-03-04T07:36:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:02 smithi116 ceph-mon[32150]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:02 smithi116 ceph-mon[32150]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-04T07:36:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:02 smithi116 ceph-mon[32150]: osdmap e65: 10 total, 9 up, 10 in 2024-03-04T07:36:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:02 smithi116 ceph-mon[32150]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:36:03.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:02 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:02 smithi152 ceph-mon[37154]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:02 smithi152 ceph-mon[37154]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-04T07:36:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:02 smithi152 ceph-mon[37154]: osdmap e65: 10 total, 9 up, 10 in 2024-03-04T07:36:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:02 smithi152 ceph-mon[37154]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:36:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:02 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:02 smithi133 ceph-mon[36505]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:02 smithi133 ceph-mon[36505]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-04T07:36:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:02 smithi133 ceph-mon[36505]: osdmap e65: 10 total, 9 up, 10 in 2024-03-04T07:36:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:02 smithi133 ceph-mon[36505]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-04T07:36:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:02 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:03.665 DEBUG:teuthology.orchestra.run.smithi116:osd.9> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.9.service 2024-03-04T07:36:03.669 INFO:tasks.cephadm:Deploying osd.10 on smithi133 with /dev/vg_nvme/lv_1... 2024-03-04T07:36:03.670 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T07:36:03.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:03 smithi116 ceph-mon[32150]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:36:03.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:03 smithi116 ceph-mon[32150]: osdmap e66: 10 total, 9 up, 10 in 2024-03-04T07:36:03.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:03.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:03.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:03.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:03 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:03.916 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:36:04.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:03 smithi152 ceph-mon[37154]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:36:04.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:03 smithi152 ceph-mon[37154]: osdmap e66: 10 total, 9 up, 10 in 2024-03-04T07:36:04.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:04.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:04.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:04.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:03 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:04.038 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:03 smithi133 ceph-mon[36505]: from='osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-04T07:36:04.038 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:03 smithi133 ceph-mon[36505]: osdmap e66: 10 total, 9 up, 10 in 2024-03-04T07:36:04.038 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:04.039 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:04.039 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:04.039 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:03 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:04 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:04 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:04 smithi133 ceph-mon[36505]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:04 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:04 smithi133 ceph-mon[36505]: osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] boot 2024-03-04T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:04 smithi133 ceph-mon[36505]: osdmap e67: 10 total, 10 up, 10 in 2024-03-04T07:36:04.972 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:04 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:04 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:36:05.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:04 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:36:05.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:04 smithi116 ceph-mon[32150]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:05.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:04 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:05.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:04 smithi116 ceph-mon[32150]: osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] boot 2024-03-04T07:36:05.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:04 smithi116 ceph-mon[32150]: osdmap e67: 10 total, 10 up, 10 in 2024-03-04T07:36:05.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:04 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:04 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:36:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:04 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:36:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:04 smithi152 ceph-mon[37154]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-04T07:36:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:04 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:04 smithi152 ceph-mon[37154]: osd.9 [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] boot 2024-03-04T07:36:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:04 smithi152 ceph-mon[37154]: osdmap e67: 10 total, 10 up, 10 in 2024-03-04T07:36:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:04 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T07:36:05.804 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T07:36:05.805 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T07:36:05.805 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2024-03-04T07:36:05.805 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2024-03-04T07:36:05.805 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126609 s, 828 MB/s 2024-03-04T07:36:05.805 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2024-03-04T07:36:06.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:05 smithi152 ceph-mon[37154]: osdmap e68: 10 total, 10 up, 10 in 2024-03-04T07:36:06.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:05 smithi133 ceph-mon[36505]: osdmap e68: 10 total, 10 up, 10 in 2024-03-04T07:36:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:05 smithi116 ceph-mon[32150]: osdmap e68: 10 total, 10 up, 10 in 2024-03-04T07:36:06.515 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi133:vg_nvme/lv_1 2024-03-04T07:36:06.759 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T07:36:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:06 smithi152 ceph-mon[37154]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:07.066 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:06 smithi133 ceph-mon[36505]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:06 smithi116 ceph-mon[32150]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 -- 172.21.15.133:0/356747097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 msgr2=0x7efd30101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 --2- 172.21.15.133:0/356747097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd30101410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efd240099c0 tx=0x7efd24009cd0 comp rx=0 tx=0).stop 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 -- 172.21.15.133:0/356747097 shutdown_connections 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 --2- 172.21.15.133:0/356747097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd30102b90 0x7efd30109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 --2- 172.21.15.133:0/356747097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd30101410 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 --2- 172.21.15.133:0/356747097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd301021f0 0x7efd30102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 -- 172.21.15.133:0/356747097 >> 172.21.15.133:0/356747097 conn(0x7efd300fc590 msgr2=0x7efd300fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 -- 172.21.15.133:0/356747097 shutdown_connections 2024-03-04T07:36:08.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.333+0000 7efd353da700 1 -- 172.21.15.133:0/356747097 wait complete. 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 Processor -- start 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 -- start start 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd3006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd301021f0 0x7efd3019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd30102b90 0x7efd301a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3010ccb0 con 0x7efd301021f0 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3010cb30 con 0x7efd30100ff0 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd353da700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3010ce30 con 0x7efd30102b90 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd2effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd3006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd2effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd3006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53682/0 (socket says 172.21.15.133:53682) 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd2effd700 1 -- 172.21.15.133:0/3515851298 learned_addr learned my addr 172.21.15.133:0/3515851298 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-03-04T07:36:08.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.334+0000 7efd2f7fe700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd30102b90 0x7efd301a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd2effd700 1 -- 172.21.15.133:0/3515851298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd30102b90 msgr2=0x7efd301a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd2effd700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd30102b90 0x7efd301a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd2effd700 1 -- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd301021f0 msgr2=0x7efd3019f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd2effd700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd301021f0 0x7efd3019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd2effd700 1 -- 172.21.15.133:0/3515851298 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd301a2370 con 0x7efd30100ff0 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd2effd700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd3006a8d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efd24003890 tx=0x7efd24004630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd17fff700 1 -- 172.21.15.133:0/3515851298 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd24027070 con 0x7efd30100ff0 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd301a2600 con 0x7efd30100ff0 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd17fff700 1 -- 172.21.15.133:0/3515851298 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd24022470 con 0x7efd30100ff0 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd17fff700 1 -- 172.21.15.133:0/3515851298 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2400f6c0 con 0x7efd30100ff0 2024-03-04T07:36:08.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.335+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd301a2bb0 con 0x7efd30100ff0 2024-03-04T07:36:08.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.336+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd30068d80 con 0x7efd30100ff0 2024-03-04T07:36:08.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.337+0000 7efd17fff700 1 -- 172.21.15.133:0/3515851298 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7efd2402a070 con 0x7efd30100ff0 2024-03-04T07:36:08.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.338+0000 7efd17fff700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd180a1e10 0x7efd180a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:08.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.338+0000 7efd17fff700 1 -- 172.21.15.133:0/3515851298 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7efd240ce3b0 con 0x7efd30100ff0 2024-03-04T07:36:08.339 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.338+0000 7efd2e7fc700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd180a1e10 0x7efd180a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:08.340 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.339+0000 7efd2e7fc700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd180a1e10 0x7efd180a42d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd3006ae50 tx=0x7efd20006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:36:08.343 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.343+0000 7efd17fff700 1 -- 172.21.15.133:0/3515851298 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efd24013070 con 0x7efd30100ff0 2024-03-04T07:36:08.550 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:08.549+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7efd30062500 con 0x7efd180a1e10 2024-03-04T07:36:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:08.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:08.838 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:09.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:09.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:09.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:10.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:09 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi116 2024-03-04T07:36:10.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:09 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-04T07:36:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:09 smithi133 ceph-mon[36505]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:36:10.172 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:09 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi116 2024-03-04T07:36:10.172 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:09 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-04T07:36:10.172 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:09 smithi152 ceph-mon[37154]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:36:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:09 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi116 2024-03-04T07:36:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:09 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-04T07:36:10.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:09 smithi116 ceph-mon[32150]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:36:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:10 smithi133 ceph-mon[36505]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:10 smithi116 ceph-mon[32150]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:10 smithi152 ceph-mon[37154]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:12.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:11 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/2242911891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]: dispatch 2024-03-04T07:36:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:11 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]: dispatch 2024-03-04T07:36:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:11 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]': finished 2024-03-04T07:36:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:11 smithi133 ceph-mon[36505]: osdmap e69: 11 total, 10 up, 11 in 2024-03-04T07:36:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:11 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:11 smithi133 ceph-mon[36505]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/2242911891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]: dispatch 2024-03-04T07:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:11 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]: dispatch 2024-03-04T07:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:11 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]': finished 2024-03-04T07:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:11 smithi116 ceph-mon[32150]: osdmap e69: 11 total, 10 up, 11 in 2024-03-04T07:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:11 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:11 smithi116 ceph-mon[32150]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:11 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/2242911891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]: dispatch 2024-03-04T07:36:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:11 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]: dispatch 2024-03-04T07:36:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:11 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4111372f-edde-4015-beb6-fb5895afe8d7"}]': finished 2024-03-04T07:36:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:11 smithi152 ceph-mon[37154]: osdmap e69: 11 total, 10 up, 11 in 2024-03-04T07:36:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:11 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:11 smithi152 ceph-mon[37154]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:13.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:12 smithi133 ceph-mon[36505]: from='client.? 172.21.15.133:0/1531557418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:36:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:12 smithi116 ceph-mon[32150]: from='client.? 172.21.15.133:0/1531557418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:36:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:12 smithi152 ceph-mon[37154]: from='client.? 172.21.15.133:0/1531557418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:36:14.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:13 smithi133 ceph-mon[36505]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:13 smithi116 ceph-mon[32150]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:14.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:13 smithi152 ceph-mon[37154]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:16.710 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:16 smithi133 ceph-mon[36505]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:16 smithi116 ceph-mon[32150]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:16.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:16 smithi152 ceph-mon[37154]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:18 smithi116 ceph-mon[32150]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:18.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:18 smithi152 ceph-mon[37154]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:18.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:18 smithi133 ceph-mon[36505]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:19.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-04T07:36:19.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:19.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:19 smithi133 ceph-mon[36505]: Deploying daemon osd.10 on smithi133 2024-03-04T07:36:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-04T07:36:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:19 smithi116 ceph-mon[32150]: Deploying daemon osd.10 on smithi133 2024-03-04T07:36:19.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-04T07:36:19.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:19.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:19 smithi152 ceph-mon[37154]: Deploying daemon osd.10 on smithi133 2024-03-04T07:36:20.589 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:20 smithi133 ceph-mon[36505]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:20 smithi116 ceph-mon[32150]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:20.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:20 smithi152 ceph-mon[37154]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:22.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:22 smithi152 ceph-mon[37154]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:22.817 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:22 smithi133 ceph-mon[36505]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:22 smithi116 ceph-mon[32150]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:24.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:24 smithi152 ceph-mon[37154]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:24.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:24 smithi133 ceph-mon[36505]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:24 smithi116 ceph-mon[32150]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:25.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:25.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:25.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:25.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:26.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:26 smithi133 ceph-mon[36505]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:26 smithi116 ceph-mon[32150]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:26 smithi152 ceph-mon[37154]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:28.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.233+0000 7efd17fff700 1 -- 172.21.15.133:0/3515851298 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7efd30062500 con 0x7efd180a1e10 2024-03-04T07:36:28.233 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 10 on host 'smithi133' 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd180a1e10 msgr2=0x7efd180a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd180a1e10 0x7efd180a42d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efd3006ae50 tx=0x7efd20006d20 comp rx=0 tx=0).stop 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 msgr2=0x7efd3006a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd3006a8d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efd24003890 tx=0x7efd24004630 comp rx=0 tx=0).stop 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 shutdown_connections 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd180a1e10 0x7efd180a42d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd30102b90 0x7efd301a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:28.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd30100ff0 0x7efd3006a8d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:28.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 --2- 172.21.15.133:0/3515851298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd301021f0 0x7efd3019f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:28.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 >> 172.21.15.133:0/3515851298 conn(0x7efd300fc590 msgr2=0x7efd30105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:36:28.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 shutdown_connections 2024-03-04T07:36:28.236 INFO:teuthology.orchestra.run.smithi133.stderr:2024-03-04T07:36:28.235+0000 7efd353da700 1 -- 172.21.15.133:0/3515851298 wait complete. 2024-03-04T07:36:28.811 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:28 smithi116 ceph-mon[32150]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:28.811 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:28.811 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:28.812 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:28 smithi116 ceph-mon[32150]: from='osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T07:36:28.812 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:28 smithi116 ceph-mon[32150]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T07:36:28.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:28 smithi133 ceph-mon[36505]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:28 smithi133 ceph-mon[36505]: from='osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T07:36:28.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:28 smithi133 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T07:36:28.905 DEBUG:teuthology.orchestra.run.smithi133:osd.10> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.10.service 2024-03-04T07:36:28.908 INFO:tasks.cephadm:Deploying osd.11 on smithi152 with /dev/vg_nvme/lv_1... 2024-03-04T07:36:28.908 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T07:36:28.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:28 smithi152 ceph-mon[37154]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:28.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:28.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:28.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:28 smithi152 ceph-mon[37154]: from='osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T07:36:28.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:28 smithi152 ceph-mon[37154]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T07:36:29.156 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: from='osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: osdmap e70: 11 total, 10 up, 11 in 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:36:30.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:30 smithi116 ceph-mon[32150]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:30.224 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:30.224 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:30.224 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-04T07:36:30.225 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: from='osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:36:30.225 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: osdmap e70: 11 total, 10 up, 11 in 2024-03-04T07:36:30.225 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:30.225 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:36:30.225 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-mon[36505]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:30.370 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:30.370 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:30.371 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-04T07:36:30.371 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: from='osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:36:30.371 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: osdmap e70: 11 total, 10 up, 11 in 2024-03-04T07:36:30.371 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:30.371 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-03-04T07:36:30.371 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:30 smithi152 ceph-mon[37154]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:30.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:36:30.224+0000 7f8b3ebc5700 -1 osd.10 0 waiting for initial osdmap 2024-03-04T07:36:30.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:36:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:36:30.235+0000 7f8b3a9b8700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:36:31.273 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:31 smithi133 ceph-mon[36505]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:36:31.273 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:31 smithi133 ceph-mon[36505]: osdmap e71: 11 total, 10 up, 11 in 2024-03-04T07:36:31.273 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:31 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:31.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:31 smithi152 ceph-mon[37154]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:36:31.274 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:31 smithi152 ceph-mon[37154]: osdmap e71: 11 total, 10 up, 11 in 2024-03-04T07:36:31.274 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:31 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:31.274 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T07:36:31.274 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T07:36:31.274 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-04T07:36:31.274 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-04T07:36:31.274 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0191235 s, 548 MB/s 2024-03-04T07:36:31.274 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-04T07:36:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:31 smithi116 ceph-mon[32150]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-03-04T07:36:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:31 smithi116 ceph-mon[32150]: osdmap e71: 11 total, 10 up, 11 in 2024-03-04T07:36:31.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:31.897 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_1 2024-03-04T07:36:32.191 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:36:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:32 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:36:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:32 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:36:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:32 smithi116 ceph-mon[32150]: osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] boot 2024-03-04T07:36:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:32 smithi116 ceph-mon[32150]: osdmap e72: 11 total, 11 up, 11 in 2024-03-04T07:36:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:32 smithi116 ceph-mon[32150]: pgmap v259: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:32 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:36:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:32 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:36:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:32 smithi152 ceph-mon[37154]: osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] boot 2024-03-04T07:36:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:32 smithi152 ceph-mon[37154]: osdmap e72: 11 total, 11 up, 11 in 2024-03-04T07:36:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:32 smithi152 ceph-mon[37154]: pgmap v259: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:32.568 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:32 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:36:32.568 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:32 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:36:32.568 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:32.568 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:32 smithi133 ceph-mon[36505]: osd.10 [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] boot 2024-03-04T07:36:32.568 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:32 smithi133 ceph-mon[36505]: osdmap e72: 11 total, 11 up, 11 in 2024-03-04T07:36:32.568 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T07:36:32.568 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:32 smithi133 ceph-mon[36505]: pgmap v259: 1 pgs: 1 peering; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: osdmap e73: 11 total, 11 up, 11 in 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi133 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi133 to 4270M 2024-03-04T07:36:33.297 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.298 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:33.298 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:33.298 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.298 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:33.298 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:33.298 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:33.298 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: osdmap e73: 11 total, 11 up, 11 in 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi133 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi133 to 4270M 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:33.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:33.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:33.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:33.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:33.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: osdmap e73: 11 total, 11 up, 11 in 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi133 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:33.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi133 to 4270M 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:33.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:33.829 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.828+0000 7f3589c29700 1 -- 172.21.15.152:0/3465342842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 msgr2=0x7f3584101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.828+0000 7f3589c29700 1 --2- 172.21.15.152:0/3465342842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584101410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3574009990 tx=0x7f3574009ca0 comp rx=0 tx=0).stop 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- 172.21.15.152:0/3465342842 shutdown_connections 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 --2- 172.21.15.152:0/3465342842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584102b90 0x7f3584109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 --2- 172.21.15.152:0/3465342842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584101410 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 --2- 172.21.15.152:0/3465342842 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35841021f0 0x7f3584102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- 172.21.15.152:0/3465342842 >> 172.21.15.152:0/3465342842 conn(0x7f35840fc5b0 msgr2=0x7f35840fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- 172.21.15.152:0/3465342842 shutdown_connections 2024-03-04T07:36:34.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- 172.21.15.152:0/3465342842 wait complete. 2024-03-04T07:36:34.077 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 Processor -- start 2024-03-04T07:36:34.077 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- start start 2024-03-04T07:36:34.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:34.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35841021f0 0x7f35841989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:34.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584102b90 0x7f358419fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:34.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358410ccb0 con 0x7f3584102b90 2024-03-04T07:36:34.078 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358410cb30 con 0x7f35841021f0 2024-03-04T07:36:34.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f3589c29700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358410ce30 con 0x7f3584100ff0 2024-03-04T07:36:34.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f35837fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:34.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f35837fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:50688/0 (socket says 172.21.15.152:50688) 2024-03-04T07:36:34.079 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.829+0000 7f35837fe700 1 -- 172.21.15.152:0/2816601523 learned_addr learned my addr 172.21.15.152:0/2816601523 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:36:34.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f357bfff700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35841021f0 0x7f35841989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:34.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35837fe700 1 -- 172.21.15.152:0/2816601523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35841021f0 msgr2=0x7f35841989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:34.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35837fe700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35841021f0 0x7f35841989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:34.080 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35837fe700 1 -- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584102b90 msgr2=0x7f358419fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:34.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35837fe700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584102b90 0x7f358419fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:34.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35837fe700 1 -- 172.21.15.152:0/2816601523 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35841a00d0 con 0x7f3584100ff0 2024-03-04T07:36:34.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35837fe700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584198470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35740047b0 tx=0x7f3574009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:36:34.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35817fa700 1 -- 172.21.15.152:0/2816601523 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3574027070 con 0x7f3584100ff0 2024-03-04T07:36:34.081 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35817fa700 1 -- 172.21.15.152:0/2816601523 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3574022470 con 0x7f3584100ff0 2024-03-04T07:36:34.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f35817fa700 1 -- 172.21.15.152:0/2816601523 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357400f6c0 con 0x7f3584100ff0 2024-03-04T07:36:34.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35841a0360 con 0x7f3584100ff0 2024-03-04T07:36:34.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.830+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35841a0890 con 0x7f3584100ff0 2024-03-04T07:36:34.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.831+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3584068d80 con 0x7f3584100ff0 2024-03-04T07:36:34.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.832+0000 7f35817fa700 1 -- 172.21.15.152:0/2816601523 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f357402a070 con 0x7f3584100ff0 2024-03-04T07:36:34.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.833+0000 7f35817fa700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f35700a1e90 0x7f35700a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:34.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.833+0000 7f35817fa700 1 -- 172.21.15.152:0/2816601523 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f35740cefc0 con 0x7f3584100ff0 2024-03-04T07:36:34.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.833+0000 7f357bfff700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f35700a1e90 0x7f35700a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:34.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.834+0000 7f357bfff700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f35700a1e90 0x7f35700a4350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f356c006fd0 tx=0x7f356c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:36:34.084 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:33.837+0000 7f35817fa700 1 -- 172.21.15.152:0/2816601523 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3574031070 con 0x7f3584100ff0 2024-03-04T07:36:34.084 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:34.041+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f35840fe7e0 con 0x7f35700a1e90 2024-03-04T07:36:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:34 smithi152 ceph-mon[37154]: osdmap e74: 11 total, 11 up, 11 in 2024-03-04T07:36:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:34 smithi152 ceph-mon[37154]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:34.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:36:34.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:36:34.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:34 smithi133 ceph-mon[36505]: osdmap e74: 11 total, 11 up, 11 in 2024-03-04T07:36:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:34 smithi133 ceph-mon[36505]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:36:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:36:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:34.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:34 smithi116 ceph-mon[32150]: osdmap e74: 11 total, 11 up, 11 in 2024-03-04T07:36:34.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:34 smithi116 ceph-mon[32150]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:34.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T07:36:34.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T07:36:34.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:35.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:35 smithi133 ceph-mon[36505]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:36:35.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:35 smithi133 ceph-mon[36505]: osdmap e75: 11 total, 11 up, 11 in 2024-03-04T07:36:35.666 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:35 smithi152 ceph-mon[37154]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:36:35.667 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:35 smithi152 ceph-mon[37154]: osdmap e75: 11 total, 11 up, 11 in 2024-03-04T07:36:35.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:35 smithi116 ceph-mon[32150]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:36:35.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:35 smithi116 ceph-mon[32150]: osdmap e75: 11 total, 11 up, 11 in 2024-03-04T07:36:36.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:36 smithi133 ceph-mon[36505]: osdmap e76: 11 total, 11 up, 11 in 2024-03-04T07:36:36.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:36 smithi133 ceph-mon[36505]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:36:36.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:36 smithi116 ceph-mon[32150]: osdmap e76: 11 total, 11 up, 11 in 2024-03-04T07:36:36.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:36 smithi116 ceph-mon[32150]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:36:36.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:36 smithi152 ceph-mon[37154]: osdmap e76: 11 total, 11 up, 11 in 2024-03-04T07:36:36.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:36 smithi152 ceph-mon[37154]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T07:36:37.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:37 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/159757091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]: dispatch 2024-03-04T07:36:37.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:37 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]: dispatch 2024-03-04T07:36:37.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:37 smithi152 ceph-mon[37154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]': finished 2024-03-04T07:36:37.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:37 smithi152 ceph-mon[37154]: osdmap e77: 12 total, 11 up, 12 in 2024-03-04T07:36:37.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:37.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:37 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/159757091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]: dispatch 2024-03-04T07:36:37.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:37 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]: dispatch 2024-03-04T07:36:37.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:37 smithi133 ceph-mon[36505]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]': finished 2024-03-04T07:36:37.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:37 smithi133 ceph-mon[36505]: osdmap e77: 12 total, 11 up, 12 in 2024-03-04T07:36:37.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:37.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:37 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/159757091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]: dispatch 2024-03-04T07:36:37.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:37 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]: dispatch 2024-03-04T07:36:37.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:37 smithi116 ceph-mon[32150]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c4a6c93-8d7c-448c-99cb-a0ede94976a9"}]': finished 2024-03-04T07:36:37.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:37 smithi116 ceph-mon[32150]: osdmap e77: 12 total, 11 up, 12 in 2024-03-04T07:36:37.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:38.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:38 smithi133 ceph-mon[36505]: from='client.? 172.21.15.152:0/3188563179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:36:38.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:38 smithi133 ceph-mon[36505]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-04T07:36:38.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:38 smithi116 ceph-mon[32150]: from='client.? 172.21.15.152:0/3188563179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:36:38.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:38 smithi116 ceph-mon[32150]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-04T07:36:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:38 smithi152 ceph-mon[37154]: from='client.? 172.21.15.152:0/3188563179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T07:36:38.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:38 smithi152 ceph-mon[37154]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-04T07:36:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:40 smithi116 ceph-mon[32150]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:36:40.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:40 smithi152 ceph-mon[37154]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:36:40.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:40 smithi133 ceph-mon[36505]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T07:36:42.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:42 smithi116 ceph-mon[32150]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-04T07:36:42.770 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:42 smithi152 ceph-mon[37154]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-04T07:36:42.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:42 smithi133 ceph-mon[36505]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-04T07:36:44.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:44 smithi116 ceph-mon[32150]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-04T07:36:44.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-04T07:36:44.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:44.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:44 smithi152 ceph-mon[37154]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-04T07:36:44.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:44 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-04T07:36:44.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:44 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:44.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:44 smithi133 ceph-mon[36505]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-04T07:36:44.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:44 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-04T07:36:44.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:44 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:45.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:45 smithi152 ceph-mon[37154]: Deploying daemon osd.11 on smithi152 2024-03-04T07:36:45.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:45 smithi133 ceph-mon[36505]: Deploying daemon osd.11 on smithi152 2024-03-04T07:36:46.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:45 smithi116 ceph-mon[32150]: Deploying daemon osd.11 on smithi152 2024-03-04T07:36:46.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:46 smithi152 ceph-mon[37154]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:46.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:46 smithi133 ceph-mon[36505]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:46 smithi116 ceph-mon[32150]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:48.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:48 smithi133 ceph-mon[36505]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:48.946 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:48 smithi152 ceph-mon[37154]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:48 smithi116 ceph-mon[32150]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:50.545 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:50 smithi152 ceph-mon[37154]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:50.545 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:50.545 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:50.545 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:50 smithi133 ceph-mon[36505]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:50 smithi116 ceph-mon[32150]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:52.647 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:52 smithi152 ceph-mon[37154]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:52.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:52 smithi133 ceph-mon[36505]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:52 smithi116 ceph-mon[32150]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:53.376 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.376+0000 7f35817fa700 1 -- 172.21.15.152:0/2816601523 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f35840fe7e0 con 0x7f35700a1e90 2024-03-04T07:36:53.376 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 11 on host 'smithi152' 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f35700a1e90 msgr2=0x7f35700a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f35700a1e90 0x7f35700a4350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f356c006fd0 tx=0x7f356c008040 comp rx=0 tx=0).stop 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 msgr2=0x7f3584198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584198470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35740047b0 tx=0x7f3574009f90 comp rx=0 tx=0).stop 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 shutdown_connections 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584102b90 0x7f358419fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f35700a1e90 0x7f35700a4350 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:53.378 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3584100ff0 0x7f3584198470 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:53.379 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 --2- 172.21.15.152:0/2816601523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f35841021f0 0x7f35841989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:53.379 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 >> 172.21.15.152:0/2816601523 conn(0x7f35840fc5b0 msgr2=0x7f35840fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:36:53.379 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 shutdown_connections 2024-03-04T07:36:53.379 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:36:53.378+0000 7f3589c29700 1 -- 172.21.15.152:0/2816601523 wait complete. 2024-03-04T07:36:54.032 DEBUG:teuthology.orchestra.run.smithi152:osd.11> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.11.service 2024-03-04T07:36:54.035 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-04T07:36:54.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T07:36:54.496 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:36:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:54 smithi116 ceph-mon[32150]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:54 smithi116 ceph-mon[32150]: from='osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T07:36:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:54 smithi116 ceph-mon[32150]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T07:36:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:54 smithi133 ceph-mon[36505]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:54 smithi133 ceph-mon[36505]: from='osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T07:36:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:54 smithi133 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T07:36:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:54 smithi152 ceph-mon[37154]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:54 smithi152 ceph-mon[37154]: from='osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T07:36:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:54 smithi152 ceph-mon[37154]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T07:36:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:54.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:55.696 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:55 smithi116 ceph-mon[32150]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-04T07:36:55.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:55 smithi116 ceph-mon[32150]: osdmap e78: 12 total, 11 up, 12 in 2024-03-04T07:36:55.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:55.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:55 smithi116 ceph-mon[32150]: from='osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:36:55.697 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:55 smithi116 ceph-mon[32150]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:36:55.768 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:55 smithi152 ceph-mon[37154]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-04T07:36:55.768 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:55 smithi152 ceph-mon[37154]: osdmap e78: 12 total, 11 up, 12 in 2024-03-04T07:36:55.769 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:55.769 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:55 smithi152 ceph-mon[37154]: from='osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:36:55.769 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:55 smithi152 ceph-mon[37154]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:36:55.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:55 smithi133 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-04T07:36:55.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:55 smithi133 ceph-mon[36505]: osdmap e78: 12 total, 11 up, 12 in 2024-03-04T07:36:55.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:55.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:55 smithi133 ceph-mon[36505]: from='osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:36:55.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:55 smithi133 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-04T07:36:56.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:36:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:36:55.768+0000 7f326061b700 -1 osd.11 0 waiting for initial osdmap 2024-03-04T07:36:56.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:36:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:36:55.783+0000 7f3258406700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T07:36:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:56 smithi152 ceph-mon[37154]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:36:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:56 smithi152 ceph-mon[37154]: osdmap e79: 12 total, 11 up, 12 in 2024-03-04T07:36:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:56 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:56 smithi152 ceph-mon[37154]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:56 smithi152 ceph-mon[37154]: from='osd.11 ' entity='osd.11' 2024-03-04T07:36:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:56 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:56.688 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:56 smithi116 ceph-mon[32150]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:36:56.689 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:56 smithi116 ceph-mon[32150]: osdmap e79: 12 total, 11 up, 12 in 2024-03-04T07:36:56.689 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:56 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:56.689 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:56 smithi116 ceph-mon[32150]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:56.689 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:56 smithi116 ceph-mon[32150]: from='osd.11 ' entity='osd.11' 2024-03-04T07:36:56.689 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:56 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:56.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:56 smithi133 ceph-mon[36505]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-04T07:36:56.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:56 smithi133 ceph-mon[36505]: osdmap e79: 12 total, 11 up, 12 in 2024-03-04T07:36:56.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:56 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:56.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:56 smithi133 ceph-mon[36505]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-04T07:36:56.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:56 smithi133 ceph-mon[36505]: from='osd.11 ' entity='osd.11' 2024-03-04T07:36:56.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:56 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 -- 172.21.15.116:0/3367783761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 msgr2=0x7efc40110a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3367783761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 0x7efc40110a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc38009f90 tx=0x7efc3800b330 comp rx=0 tx=0).stop 2024-03-04T07:36:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 -- 172.21.15.116:0/3367783761 shutdown_connections 2024-03-04T07:36:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3367783761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 0x7efc40110a00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3367783761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc400760f0 0x7efc40074550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3367783761 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc40074b20 0x7efc40074fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 -- 172.21.15.116:0/3367783761 >> 172.21.15.116:0/3367783761 conn(0x7efc400fd5d0 msgr2=0x7efc400ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:36:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 -- 172.21.15.116:0/3367783761 shutdown_connections 2024-03-04T07:36:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.855+0000 7efc3fe00700 1 -- 172.21.15.116:0/3367783761 wait complete. 2024-03-04T07:36:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.856+0000 7efc3fe00700 1 Processor -- start 2024-03-04T07:36:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3fe00700 1 -- start start 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc40074b20 0x7efc4019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc400760f0 0x7efc4019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 0x7efc401a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3fe00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40114210 con 0x7efc4010e610 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3fe00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40114090 con 0x7efc400760f0 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3fe00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40114390 con 0x7efc40074b20 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc40074b20 0x7efc4019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc40074b20 0x7efc4019e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:34390/0 (socket says 172.21.15.116:34390) 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.857+0000 7efc3ea00700 1 -- 172.21.15.116:0/3225407138 learned_addr learned my addr 172.21.15.116:0/3225407138 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3f400700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 0x7efc401a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3e000700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc400760f0 0x7efc4019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3ea00700 1 -- 172.21.15.116:0/3225407138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc400760f0 msgr2=0x7efc4019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3ea00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc400760f0 0x7efc4019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3ea00700 1 -- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 msgr2=0x7efc401a5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3ea00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 0x7efc401a5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3ea00700 1 -- 172.21.15.116:0/3225407138 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc401a63e0 con 0x7efc40074b20 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3f400700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 0x7efc401a5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3ea00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc40074b20 0x7efc4019e780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efc3000ea20 tx=0x7efc3000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc37400700 1 -- 172.21.15.116:0/3225407138 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3000c9c0 con 0x7efc40074b20 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc37400700 1 -- 172.21.15.116:0/3225407138 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc30010430 con 0x7efc40074b20 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc37400700 1 -- 172.21.15.116:0/3225407138 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc30018680 con 0x7efc40074b20 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.858+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc401a66d0 con 0x7efc40074b20 2024-03-04T07:36:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.859+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc401a6c10 con 0x7efc40074b20 2024-03-04T07:36:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.860+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc40068d80 con 0x7efc40074b20 2024-03-04T07:36:56.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.862+0000 7efc37400700 1 -- 172.21.15.116:0/3225407138 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7efc300105e0 con 0x7efc40074b20 2024-03-04T07:36:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.863+0000 7efc37400700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc240a1e10 0x7efc240a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:36:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.863+0000 7efc37400700 1 -- 172.21.15.116:0/3225407138 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efc300cda00 con 0x7efc40074b20 2024-03-04T07:36:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.863+0000 7efc3e000700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc240a1e10 0x7efc240a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:36:56.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.864+0000 7efc3e000700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc240a1e10 0x7efc240a42d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc40075e20 tx=0x7efc2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:36:56.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:56.865+0000 7efc37400700 1 -- 172.21.15.116:0/3225407138 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efc3009b230 con 0x7efc40074b20 2024-03-04T07:36:57.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.085+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7efc40062cd0 con 0x7efc40074b20 2024-03-04T07:36:57.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.085+0000 7efc37400700 1 -- 172.21.15.116:0/3225407138 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7efc3009ab30 con 0x7efc40074b20 2024-03-04T07:36:57.085 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:36:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc240a1e10 msgr2=0x7efc240a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc240a1e10 0x7efc240a42d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc40075e20 tx=0x7efc2c009260 comp rx=0 tx=0).stop 2024-03-04T07:36:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc40074b20 msgr2=0x7efc4019e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:36:57.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc40074b20 0x7efc4019e780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efc3000ea20 tx=0x7efc3000ed30 comp rx=0 tx=0).stop 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 shutdown_connections 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010e610 0x7efc401a5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc240a1e10 0x7efc240a42d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc40074b20 0x7efc4019e780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 --2- 172.21.15.116:0/3225407138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc400760f0 0x7efc4019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.088+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 >> 172.21.15.116:0/3225407138 conn(0x7efc400fd5d0 msgr2=0x7efc401019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.089+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 shutdown_connections 2024-03-04T07:36:57.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:36:57.089+0000 7efc3fe00700 1 -- 172.21.15.116:0/3225407138 wait complete. 2024-03-04T07:36:57.706 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:57 smithi152 ceph-mon[37154]: purged_snaps scrub starts 2024-03-04T07:36:57.706 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:57 smithi152 ceph-mon[37154]: purged_snaps scrub ok 2024-03-04T07:36:57.706 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:57 smithi152 ceph-mon[37154]: osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] boot 2024-03-04T07:36:57.706 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:57 smithi152 ceph-mon[37154]: osdmap e80: 12 total, 12 up, 12 in 2024-03-04T07:36:57.706 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:57.706 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:57 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3225407138' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T07:36:57.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:57 smithi116 ceph-mon[32150]: purged_snaps scrub starts 2024-03-04T07:36:57.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:57 smithi116 ceph-mon[32150]: purged_snaps scrub ok 2024-03-04T07:36:57.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:57 smithi116 ceph-mon[32150]: osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] boot 2024-03-04T07:36:57.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:57 smithi116 ceph-mon[32150]: osdmap e80: 12 total, 12 up, 12 in 2024-03-04T07:36:57.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:57.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:57 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3225407138' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T07:36:57.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:57 smithi133 ceph-mon[36505]: purged_snaps scrub starts 2024-03-04T07:36:57.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:57 smithi133 ceph-mon[36505]: purged_snaps scrub ok 2024-03-04T07:36:57.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:57 smithi133 ceph-mon[36505]: osd.11 [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] boot 2024-03-04T07:36:57.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:57 smithi133 ceph-mon[36505]: osdmap e80: 12 total, 12 up, 12 in 2024-03-04T07:36:57.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T07:36:57.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:57 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3225407138' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T07:36:57.919 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1709537816,"num_in_osds":12,"osd_in_since":1709537796,"num_remapped_pgs":0} 2024-03-04T07:36:57.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T07:36:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: Detected new or changed devices on smithi152 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi152 to 4270M 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:58.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:58.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:36:58 smithi116 ceph-mon[32150]: osdmap e81: 12 total, 12 up, 12 in 2024-03-04T07:36:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:36:59.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: Detected new or changed devices on smithi152 2024-03-04T07:36:59.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi152 to 4270M 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:59.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:36:58 smithi152 ceph-mon[37154]: osdmap e81: 12 total, 12 up, 12 in 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: Detected new or changed devices on smithi152 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi152 to 4270M 2024-03-04T07:36:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:36:59.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:36:58 smithi133 ceph-mon[36505]: osdmap e81: 12 total, 12 up, 12 in 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 -- 172.21.15.116:0/1508259881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc108fd0 msgr2=0x7fd6fc115440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 --2- 172.21.15.116:0/1508259881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc108fd0 0x7fd6fc115440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400b0d0 tx=0x7fd6f400b3e0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 -- 172.21.15.116:0/1508259881 shutdown_connections 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 --2- 172.21.15.116:0/1508259881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc108fd0 0x7fd6fc115440 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 --2- 172.21.15.116:0/1508259881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6fc10e670 0x7fd6fc10ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 --2- 172.21.15.116:0/1508259881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6fc108610 0x7fd6fc108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 -- 172.21.15.116:0/1508259881 >> 172.21.15.116:0/1508259881 conn(0x7fd6fc077d60 msgr2=0x7fd6fc07a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:00.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.727+0000 7fd701e00700 1 -- 172.21.15.116:0/1508259881 shutdown_connections 2024-03-04T07:37:00.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.728+0000 7fd701e00700 1 -- 172.21.15.116:0/1508259881 wait complete. 2024-03-04T07:37:00.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.729+0000 7fd701e00700 1 Processor -- start 2024-03-04T07:37:00.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.729+0000 7fd701e00700 1 -- start start 2024-03-04T07:37:00.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6fc108610 0x7fd6fc1a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:00.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6fc108fd0 0x7fd6fc1a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:00.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc10e670 0x7fd6fc19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:00.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc118970 con 0x7fd6fc10e670 2024-03-04T07:37:00.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701e00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1187f0 con 0x7fd6fc108fd0 2024-03-04T07:37:00.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc118af0 con 0x7fd6fc108610 2024-03-04T07:37:00.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc10e670 0x7fd6fc19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:00.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd700a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6fc108610 0x7fd6fc1a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:00.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc10e670 0x7fd6fc19eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53440/0 (socket says 172.21.15.116:53440) 2024-03-04T07:37:00.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd701400700 1 -- 172.21.15.116:0/1151909327 learned_addr learned my addr 172.21.15.116:0/1151909327 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.730+0000 7fd6fbe00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6fc108fd0 0x7fd6fc1a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd700a00700 1 -- 172.21.15.116:0/1151909327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6fc108fd0 msgr2=0x7fd6fc1a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd700a00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6fc108fd0 0x7fd6fc1a4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd700a00700 1 -- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc10e670 msgr2=0x7fd6fc19eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd700a00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc10e670 0x7fd6fc19eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd700a00700 1 -- 172.21.15.116:0/1151909327 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc19f250 con 0x7fd6fc108610 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd701400700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc10e670 0x7fd6fc19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd700a00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6fc108610 0x7fd6fc1a4930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000e9f0 tx=0x7fd6f000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd6f9600700 1 -- 172.21.15.116:0/1151909327 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f000c9c0 con 0x7fd6fc108610 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc19f540 con 0x7fd6fc108610 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd6f9600700 1 -- 172.21.15.116:0/1151909327 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6f000cb60 con 0x7fd6fc108610 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd6f9600700 1 -- 172.21.15.116:0/1151909327 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0018680 con 0x7fd6fc108610 2024-03-04T07:37:00.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.731+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc1ab800 con 0x7fd6fc108610 2024-03-04T07:37:00.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.733+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc068d80 con 0x7fd6fc108610 2024-03-04T07:37:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.735+0000 7fd6f9600700 1 -- 172.21.15.116:0/1151909327 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd6f0010430 con 0x7fd6fc108610 2024-03-04T07:37:00.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.736+0000 7fd6f9600700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6e40a1e10 0x7fd6e40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:00.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.736+0000 7fd6f9600700 1 -- 172.21.15.116:0/1151909327 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd6f00cd9e0 con 0x7fd6fc108610 2024-03-04T07:37:00.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.736+0000 7fd6fbe00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6e40a1e10 0x7fd6e40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:00.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.737+0000 7fd6fbe00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6e40a1e10 0x7fd6e40a42d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc07a5e0 tx=0x7fd6e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:00.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.739+0000 7fd6f9600700 1 -- 172.21.15.116:0/1151909327 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd6f009b210 con 0x7fd6fc108610 2024-03-04T07:37:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.958+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6fc02d9b0 con 0x7fd6fc108610 2024-03-04T07:37:00.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:00 smithi116 ceph-mon[32150]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:00.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.959+0000 7fd6f9600700 1 -- 172.21.15.116:0/1151909327 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18539 (secure 0 0 0) 0x7fd6f009ab10 con 0x7fd6fc108610 2024-03-04T07:37:00.960 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:00.960 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:36:58.362248+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6e40a1e10 msgr2=0x7fd6e40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6e40a1e10 0x7fd6e40a42d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc07a5e0 tx=0x7fd6e8009260 comp rx=0 tx=0).stop 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6fc108610 msgr2=0x7fd6fc1a4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6fc108610 0x7fd6fc1a4930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6f000e9f0 tx=0x7fd6f000ed00 comp rx=0 tx=0).stop 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 shutdown_connections 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6fc10e670 0x7fd6fc19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6e40a1e10 0x7fd6e40a42d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6fc108610 0x7fd6fc1a4930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 --2- 172.21.15.116:0/1151909327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6fc108fd0 0x7fd6fc1a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:00.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 >> 172.21.15.116:0/1151909327 conn(0x7fd6fc077d60 msgr2=0x7fd6fc079360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 shutdown_connections 2024-03-04T07:37:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:00.963+0000 7fd701e00700 1 -- 172.21.15.116:0/1151909327 wait complete. 2024-03-04T07:37:01.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:00 smithi152 ceph-mon[37154]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:01.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:00 smithi133 ceph-mon[36505]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:01.862 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:37:01.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-04T07:37:02.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:01 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1151909327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:02.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:01 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1151909327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:02.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:01 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1151909327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:02.310 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:37:03.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:02 smithi133 ceph-mon[36505]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:02 smithi116 ceph-mon[32150]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:02 smithi152 ceph-mon[37154]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:04.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 -- 172.21.15.116:0/2846832640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e406a710 msgr2=0x7fb8e410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:04.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 --2- 172.21.15.116:0/2846832640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e406a710 0x7fb8e410ef50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc009fc0 tx=0x7fb8dc00b330 comp rx=0 tx=0).stop 2024-03-04T07:37:04.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 -- 172.21.15.116:0/2846832640 shutdown_connections 2024-03-04T07:37:04.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 --2- 172.21.15.116:0/2846832640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e4108110 0x7fb8e41084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 --2- 172.21.15.116:0/2846832640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e406a710 0x7fb8e410ef50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 --2- 172.21.15.116:0/2846832640 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8e4069d70 0x7fb8e406a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 -- 172.21.15.116:0/2846832640 >> 172.21.15.116:0/2846832640 conn(0x7fb8e4077ef0 msgr2=0x7fb8e4078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 -- 172.21.15.116:0/2846832640 shutdown_connections 2024-03-04T07:37:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.505+0000 7fb8ec200700 1 -- 172.21.15.116:0/2846832640 wait complete. 2024-03-04T07:37:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.506+0000 7fb8ec200700 1 Processor -- start 2024-03-04T07:37:04.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.506+0000 7fb8ec200700 1 -- start start 2024-03-04T07:37:04.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8ec200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e4069d70 0x7fb8e419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8ec200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e406a710 0x7fb8e419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8ec200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8e4108110 0x7fb8e4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8ec200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e41123e0 con 0x7fb8e406a710 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8ec200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4112260 con 0x7fb8e4108110 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8ec200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4112560 con 0x7fb8e4069d70 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8e9800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e4069d70 0x7fb8e419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8e8e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e406a710 0x7fb8e419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8e9800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e4069d70 0x7fb8e419e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:34422/0 (socket says 172.21.15.116:34422) 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8e9800700 1 -- 172.21.15.116:0/1345363755 learned_addr learned my addr 172.21.15.116:0/1345363755 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.507+0000 7fb8ea200700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8e4108110 0x7fb8e4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e9800700 1 -- 172.21.15.116:0/1345363755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8e4108110 msgr2=0x7fb8e4198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e9800700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8e4108110 0x7fb8e4198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e9800700 1 -- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e406a710 msgr2=0x7fb8e419e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e9800700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e406a710 0x7fb8e419e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e9800700 1 -- 172.21.15.116:0/1345363755 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4198d60 con 0x7fb8e4069d70 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e8e00700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e406a710 0x7fb8e419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e9800700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e4069d70 0x7fb8e419e440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800e9c0 tx=0x7fb8d800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e2000700 1 -- 172.21.15.116:0/1345363755 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8004a30 con 0x7fb8e4069d70 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e2000700 1 -- 172.21.15.116:0/1345363755 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8d800cca0 con 0x7fb8e4069d70 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e4199050 con 0x7fb8e4069d70 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.508+0000 7fb8e2000700 1 -- 172.21.15.116:0/1345363755 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8005610 con 0x7fb8e4069d70 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.509+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e41a5310 con 0x7fb8e4069d70 2024-03-04T07:37:04.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.510+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e4068d80 con 0x7fb8e4069d70 2024-03-04T07:37:04.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.511+0000 7fb8e2000700 1 -- 172.21.15.116:0/1345363755 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb8d8004cf0 con 0x7fb8e4069d70 2024-03-04T07:37:04.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.513+0000 7fb8e2000700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8cc0a1e10 0x7fb8cc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.513+0000 7fb8e2000700 1 -- 172.21.15.116:0/1345363755 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb8d80cda20 con 0x7fb8e4069d70 2024-03-04T07:37:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.513+0000 7fb8e8e00700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8cc0a1e10 0x7fb8cc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:04.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.514+0000 7fb8e8e00700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8cc0a1e10 0x7fb8cc0a42d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40ff7f0 tx=0x7fb8d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:04.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.517+0000 7fb8e2000700 1 -- 172.21.15.116:0/1345363755 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb8d809b250 con 0x7fb8e4069d70 2024-03-04T07:37:04.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.735+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8e4199b30 con 0x7fb8e4069d70 2024-03-04T07:37:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.736+0000 7fb8e2000700 1 -- 172.21.15.116:0/1345363755 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8d809ab50 con 0x7fb8e4069d70 2024-03-04T07:37:04.737 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:37:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8cc0a1e10 msgr2=0x7fb8cc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8cc0a1e10 0x7fb8cc0a42d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40ff7f0 tx=0x7fb8d0009260 comp rx=0 tx=0).stop 2024-03-04T07:37:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e4069d70 msgr2=0x7fb8e419e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e4069d70 0x7fb8e419e440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800e9c0 tx=0x7fb8d800ecd0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 shutdown_connections 2024-03-04T07:37:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8cc0a1e10 0x7fb8cc0a42d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8e4108110 0x7fb8e4198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8e4069d70 0x7fb8e419e440 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 --2- 172.21.15.116:0/1345363755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e406a710 0x7fb8e419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 >> 172.21.15.116:0/1345363755 conn(0x7fb8e4077ef0 msgr2=0x7fb8e40ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.739+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 shutdown_connections 2024-03-04T07:37:04.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:04.740+0000 7fb8ec200700 1 -- 172.21.15.116:0/1345363755 wait complete. 2024-03-04T07:37:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:04 smithi116 ceph-mon[32150]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:04 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1345363755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:05.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:04 smithi133 ceph-mon[36505]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:05.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:04 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1345363755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:04 smithi152 ceph-mon[37154]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:04 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1345363755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:05.607 INFO:tasks.cephadm:Adding mds.a on smithi116 2024-03-04T07:37:05.607 INFO:tasks.cephadm:Adding mds.d on smithi116 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.g on smithi116 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.j on smithi116 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.b on smithi133 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.e on smithi133 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.h on smithi133 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.k on smithi133 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.c on smithi152 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.f on smithi152 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.i on smithi152 2024-03-04T07:37:05.608 INFO:tasks.cephadm:Adding mds.l on smithi152 2024-03-04T07:37:05.608 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph orch apply mds all '12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l' 2024-03-04T07:37:05.860 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T07:37:07.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:06 smithi152 ceph-mon[37154]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:07.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:06 smithi133 ceph-mon[36505]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:06 smithi116 ceph-mon[32150]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 -- 172.21.15.152:0/643527850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36341033e0 msgr2=0x7f3634103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 --2- 172.21.15.152:0/643527850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36341033e0 0x7f3634103840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3628009990 tx=0x7f3628009ca0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 -- 172.21.15.152:0/643527850 shutdown_connections 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 --2- 172.21.15.152:0/643527850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3634103d80 0x7f363410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 --2- 172.21.15.152:0/643527850 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3634100400 0x7f3634100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 --2- 172.21.15.152:0/643527850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36341033e0 0x7f3634103840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 -- 172.21.15.152:0/643527850 >> 172.21.15.152:0/643527850 conn(0x7f36340fbd60 msgr2=0x7f36340fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 -- 172.21.15.152:0/643527850 shutdown_connections 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 -- 172.21.15.152:0/643527850 wait complete. 2024-03-04T07:37:07.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 Processor -- start 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.430+0000 7f363a6bb700 1 -- start start 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f363a6bb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3634100400 0x7f3634198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f363a6bb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36341033e0 0x7f3634198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f363a6bb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3634103d80 0x7f363419f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f363a6bb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363410dda0 con 0x7f36341033e0 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f363a6bb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363410dc20 con 0x7f3634100400 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f363a6bb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363410df20 con 0x7f3634103d80 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3634103d80 0x7f363419f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3634103d80 0x7f363419f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:58448/0 (socket says 172.21.15.152:58448) 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 -- 172.21.15.152:0/1374988932 learned_addr learned my addr 172.21.15.152:0/1374988932 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3633fff700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3634100400 0x7f3634198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:07.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 -- 172.21.15.152:0/1374988932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3634100400 msgr2=0x7f3634198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:07.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3634100400 0x7f3634198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 -- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36341033e0 msgr2=0x7f3634198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:07.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36341033e0 0x7f3634198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 -- 172.21.15.152:0/1374988932 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363419ffb0 con 0x7f3634103d80 2024-03-04T07:37:07.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.431+0000 7f3638c58700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3634103d80 0x7f363419f910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f362c00b9a0 tx=0x7f362c00bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:07.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.432+0000 7f36317fa700 1 -- 172.21.15.152:0/1374988932 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c019070 con 0x7f3634103d80 2024-03-04T07:37:07.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.432+0000 7f36317fa700 1 -- 172.21.15.152:0/1374988932 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f362c00ce10 con 0x7f3634103d80 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.432+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36341a02a0 con 0x7f3634103d80 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.432+0000 7f36317fa700 1 -- 172.21.15.152:0/1374988932 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0073d0 con 0x7f3634103d80 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.432+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36341a08b0 con 0x7f3634103d80 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.432+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634068d80 con 0x7f3634103d80 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.434+0000 7f36317fa700 1 -- 172.21.15.152:0/1374988932 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f362c01c070 con 0x7f3634103d80 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.434+0000 7f36317fa700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f361c0a1e90 0x7f361c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.434+0000 7f36317fa700 1 -- 172.21.15.152:0/1374988932 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f362c0cd0b0 con 0x7f3634103d80 2024-03-04T07:37:07.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.435+0000 7f3633fff700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f361c0a1e90 0x7f361c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:07.436 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.436+0000 7f3633fff700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f361c0a1e90 0x7f361c0a4350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f363406a5e0 tx=0x7f362400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:07.437 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.438+0000 7f36317fa700 1 -- 172.21.15.152:0/1374988932 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f362c09a980 con 0x7f3634103d80 2024-03-04T07:37:07.641 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.641+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f3634062700 con 0x7f361c0a1e90 2024-03-04T07:37:07.658 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.657+0000 7f36317fa700 1 -- 172.21.15.152:0/1374988932 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f3634062700 con 0x7f361c0a1e90 2024-03-04T07:37:07.658 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mds.all update... 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f361c0a1e90 msgr2=0x7f361c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f361c0a1e90 0x7f361c0a4350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f363406a5e0 tx=0x7f362400b410 comp rx=0 tx=0).stop 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3634103d80 msgr2=0x7f363419f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3634103d80 0x7f363419f910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f362c00b9a0 tx=0x7f362c00bcb0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 shutdown_connections 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f361c0a1e90 0x7f361c0a4350 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3634103d80 0x7f363419f910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3634100400 0x7f3634198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 --2- 172.21.15.152:0/1374988932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36341033e0 0x7f3634198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 >> 172.21.15.152:0/1374988932 conn(0x7f36340fbd60 msgr2=0x7f3634106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 shutdown_connections 2024-03-04T07:37:07.661 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-04T07:37:07.660+0000 7f363a6bb700 1 -- 172.21.15.152:0/1374988932 wait complete. 2024-03-04T07:37:08.425 DEBUG:teuthology.orchestra.run.smithi116:mds.a> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.a.service 2024-03-04T07:37:08.432 DEBUG:teuthology.orchestra.run.smithi116:mds.d> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.d.service 2024-03-04T07:37:08.477 DEBUG:teuthology.orchestra.run.smithi116:mds.g> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.g.service 2024-03-04T07:37:08.523 DEBUG:teuthology.orchestra.run.smithi116:mds.j> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.j.service 2024-03-04T07:37:08.568 DEBUG:teuthology.orchestra.run.smithi133:mds.b> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.b.service 2024-03-04T07:37:08.571 DEBUG:teuthology.orchestra.run.smithi133:mds.e> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.e.service 2024-03-04T07:37:08.614 DEBUG:teuthology.orchestra.run.smithi133:mds.h> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.h.service 2024-03-04T07:37:08.657 DEBUG:teuthology.orchestra.run.smithi133:mds.k> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.k.service 2024-03-04T07:37:08.700 DEBUG:teuthology.orchestra.run.smithi152:mds.c> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.c.service 2024-03-04T07:37:08.702 DEBUG:teuthology.orchestra.run.smithi152:mds.f> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.f.service 2024-03-04T07:37:08.745 DEBUG:teuthology.orchestra.run.smithi152:mds.i> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.i.service 2024-03-04T07:37:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:08 smithi116 ceph-mon[32150]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:08 smithi116 ceph-mon[32150]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:37:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:08 smithi116 ceph-mon[32150]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-04T07:37:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:08.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:08 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:08.787 DEBUG:teuthology.orchestra.run.smithi152:mds.l> sudo journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.l.service 2024-03-04T07:37:08.830 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-04T07:37:08.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:37:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:08 smithi152 ceph-mon[37154]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:08 smithi152 ceph-mon[37154]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:37:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:08 smithi152 ceph-mon[37154]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-04T07:37:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:08 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:09.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 -- 172.21.15.116:0/4294902558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d0107c10 msgr2=0x7fc2d0108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 --2- 172.21.15.116:0/4294902558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d0107c10 0x7fc2d0108030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009900 tx=0x7fc2bc02bc50 comp rx=0 tx=0).stop 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 -- 172.21.15.116:0/4294902558 shutdown_connections 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 --2- 172.21.15.116:0/4294902558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d01097c0 0x7fc2d0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 --2- 172.21.15.116:0/4294902558 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc2d0108e00 0x7fc2d0109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 --2- 172.21.15.116:0/4294902558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d0107c10 0x7fc2d0108030 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 -- 172.21.15.116:0/4294902558 >> 172.21.15.116:0/4294902558 conn(0x7fc2d0076be0 msgr2=0x7fc2d0079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 -- 172.21.15.116:0/4294902558 shutdown_connections 2024-03-04T07:37:09.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.064+0000 7fc2d7800640 1 -- 172.21.15.116:0/4294902558 wait complete. 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 Processor -- start 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 -- start start 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d0107c10 0x7fc2d019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc2d0108e00 0x7fc2d019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d01097c0 0x7fc2d01a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d01138b0 con 0x7fc2d0107c10 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d0113730 con 0x7fc2d0108e00 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d7800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d0113a30 con 0x7fc2d01097c0 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.065+0000 7fc2d4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d0107c10 0x7fc2d019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2cfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc2d0108e00 0x7fc2d019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2d4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d0107c10 0x7fc2d019def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58692/0 (socket says 172.21.15.116:58692) 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2d4e00640 1 -- 172.21.15.116:0/2338652228 learned_addr learned my addr 172.21.15.116:0/2338652228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2d5800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d01097c0 0x7fc2d01a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2cfe00640 1 -- 172.21.15.116:0/2338652228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d01097c0 msgr2=0x7fc2d01a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2cfe00640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d01097c0 0x7fc2d01a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2cfe00640 1 -- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d0107c10 msgr2=0x7fc2d019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2cfe00640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d0107c10 0x7fc2d019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2cfe00640 1 -- 172.21.15.116:0/2338652228 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d01a5b10 con 0x7fc2d0108e00 2024-03-04T07:37:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2d5800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d01097c0 0x7fc2d01a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:09.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.066+0000 7fc2cfe00640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc2d0108e00 0x7fc2d019e430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005fd0 tx=0x7fc2c400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.067+0000 7fc2cd600640 1 -- 172.21.15.116:0/2338652228 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4009c30 con 0x7fc2d0108e00 2024-03-04T07:37:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.067+0000 7fc2cd600640 1 -- 172.21.15.116:0/2338652228 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2c4009dd0 con 0x7fc2d0108e00 2024-03-04T07:37:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.067+0000 7fc2cd600640 1 -- 172.21.15.116:0/2338652228 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4010460 con 0x7fc2d0108e00 2024-03-04T07:37:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.067+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d01a5e00 con 0x7fc2d0108e00 2024-03-04T07:37:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.067+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d01a6270 con 0x7fc2d0108e00 2024-03-04T07:37:09.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.068+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d010eed0 con 0x7fc2d0108e00 2024-03-04T07:37:09.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.070+0000 7fc2cd600640 1 -- 172.21.15.116:0/2338652228 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fc2c4003ef0 con 0x7fc2d0108e00 2024-03-04T07:37:09.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.071+0000 7fc2cd600640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc2b80a1e00 0x7fc2b80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.071+0000 7fc2cd600640 1 -- 172.21.15.116:0/2338652228 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc2c40cd3a0 con 0x7fc2d0108e00 2024-03-04T07:37:09.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.071+0000 7fc2d4e00640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc2b80a1e00 0x7fc2b80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.072+0000 7fc2d4e00640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc2b80a1e00 0x7fc2b80a42c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc02c900 tx=0x7fc2bc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.075+0000 7fc2cd600640 1 -- 172.21.15.116:0/2338652228 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc2c4017020 con 0x7fc2d0108e00 2024-03-04T07:37:09.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:08 smithi133 ceph-mon[36505]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:08 smithi133 ceph-mon[36505]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:37:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:08 smithi133 ceph-mon[36505]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-04T07:37:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:08 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:09.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.272+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2d002d1a0 con 0x7fc2d0108e00 2024-03-04T07:37:09.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.274+0000 7fc2cd600640 1 -- 172.21.15.116:0/2338652228 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18539 (secure 0 0 0) 0x7fc2c409a620 con 0x7fc2d0108e00 2024-03-04T07:37:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:09.275 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:36:58.362248+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:37:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.276+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc2b80a1e00 msgr2=0x7fc2b80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.276+0000 7fc2d7800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc2b80a1e00 0x7fc2b80a42c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc02c900 tx=0x7fc2bc000bc0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc2d0108e00 msgr2=0x7fc2d019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc2d0108e00 0x7fc2d019e430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005fd0 tx=0x7fc2c400dde0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 shutdown_connections 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc2b80a1e00 0x7fc2b80a42c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2d01097c0 0x7fc2d01a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc2d0108e00 0x7fc2d019e430 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 --2- 172.21.15.116:0/2338652228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2d0107c10 0x7fc2d019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 >> 172.21.15.116:0/2338652228 conn(0x7fc2d0076be0 msgr2=0x7fc2d010c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 shutdown_connections 2024-03-04T07:37:09.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.277+0000 7fc2d7800640 1 -- 172.21.15.116:0/2338652228 wait complete. 2024-03-04T07:37:09.299 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:37:09.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:37:09.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.519+0000 7f1cec800640 1 -- 172.21.15.116:0/3110733054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 msgr2=0x7f1ce4103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.519+0000 7f1cec800640 1 --2- 172.21.15.116:0/3110733054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 0x7f1ce4103c30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009900 tx=0x7f1cd802bc50 comp rx=0 tx=0).stop 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.519+0000 7f1cec800640 1 -- 172.21.15.116:0/3110733054 shutdown_connections 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.519+0000 7f1cec800640 1 --2- 172.21.15.116:0/3110733054 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ce41053c0 0x7f1ce410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.519+0000 7f1cec800640 1 --2- 172.21.15.116:0/3110733054 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ce4104a00 0x7f1ce4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.519+0000 7f1cec800640 1 --2- 172.21.15.116:0/3110733054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 0x7f1ce4103c30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.519+0000 7f1cec800640 1 -- 172.21.15.116:0/3110733054 >> 172.21.15.116:0/3110733054 conn(0x7f1ce40fed80 msgr2=0x7f1ce41011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.520+0000 7f1cec800640 1 -- 172.21.15.116:0/3110733054 shutdown_connections 2024-03-04T07:37:09.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.520+0000 7f1cec800640 1 -- 172.21.15.116:0/3110733054 wait complete. 2024-03-04T07:37:09.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 Processor -- start 2024-03-04T07:37:09.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 -- start start 2024-03-04T07:37:09.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 0x7f1ce4199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ce4104a00 0x7f1ce419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ce41053c0 0x7f1ce41a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce410f460 con 0x7f1ce4103810 2024-03-04T07:37:09.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce410f2e0 con 0x7f1ce4104a00 2024-03-04T07:37:09.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cec800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce410f5e0 con 0x7f1ce41053c0 2024-03-04T07:37:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1ce9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 0x7f1ce4199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1ce9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 0x7f1ce4199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58718/0 (socket says 172.21.15.116:58718) 2024-03-04T07:37:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1ce9e00640 1 -- 172.21.15.116:0/946537223 learned_addr learned my addr 172.21.15.116:0/946537223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1cea800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ce41053c0 0x7f1ce41a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.521+0000 7f1ce9400640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ce4104a00 0x7f1ce419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cea800640 1 -- 172.21.15.116:0/946537223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ce4104a00 msgr2=0x7f1ce419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cea800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ce4104a00 0x7f1ce419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cea800640 1 -- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 msgr2=0x7f1ce4199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cea800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 0x7f1ce4199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cea800640 1 -- 172.21.15.116:0/946537223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce41a17f0 con 0x7f1ce41053c0 2024-03-04T07:37:09.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cea800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ce41053c0 0x7f1ce41a1180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009fd0 tx=0x7f1ce000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:09.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cdea00640 1 -- 172.21.15.116:0/946537223 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce000ce20 con 0x7f1ce41053c0 2024-03-04T07:37:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce41a1ae0 con 0x7f1ce41053c0 2024-03-04T07:37:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cdea00640 1 -- 172.21.15.116:0/946537223 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0005020 con 0x7f1ce41053c0 2024-03-04T07:37:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cdea00640 1 -- 172.21.15.116:0/946537223 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0010720 con 0x7f1ce41053c0 2024-03-04T07:37:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.522+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce41a2030 con 0x7f1ce41053c0 2024-03-04T07:37:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.524+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce410ab80 con 0x7f1ce41053c0 2024-03-04T07:37:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.525+0000 7f1cdea00640 1 -- 172.21.15.116:0/946537223 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1ce0005540 con 0x7f1ce41053c0 2024-03-04T07:37:09.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.527+0000 7f1cdea00640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ccc0a1d80 0x7f1ccc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.527+0000 7f1cdea00640 1 -- 172.21.15.116:0/946537223 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1ce00ce0a0 con 0x7f1ce41053c0 2024-03-04T07:37:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.527+0000 7f1ce9e00640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ccc0a1d80 0x7f1ccc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.527+0000 7f1ce9e00640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ccc0a1d80 0x7f1ccc0a4240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1cd802c900 tx=0x7f1cd8005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:09.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.531+0000 7f1cdea00640 1 -- 172.21.15.116:0/946537223 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1ce009b8d0 con 0x7f1ce41053c0 2024-03-04T07:37:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.731+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ce40621d0 con 0x7f1ce41053c0 2024-03-04T07:37:09.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.731+0000 7f1cdea00640 1 -- 172.21.15.116:0/946537223 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ce009b1d0 con 0x7f1ce41053c0 2024-03-04T07:37:09.732 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:37:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ccc0a1d80 msgr2=0x7f1ccc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ccc0a1d80 0x7f1ccc0a4240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1cd802c900 tx=0x7f1cd8005b50 comp rx=0 tx=0).stop 2024-03-04T07:37:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ce41053c0 msgr2=0x7f1ce41a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ce41053c0 0x7f1ce41a1180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009fd0 tx=0x7f1ce000ef10 comp rx=0 tx=0).stop 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 shutdown_connections 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ccc0a1d80 0x7f1ccc0a4240 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ce41053c0 0x7f1ce41a1180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1ce4104a00 0x7f1ce419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 --2- 172.21.15.116:0/946537223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ce4103810 0x7f1ce4199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.734+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 >> 172.21.15.116:0/946537223 conn(0x7f1ce40fed80 msgr2=0x7f1ce4107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.735+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 shutdown_connections 2024-03-04T07:37:09.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.735+0000 7f1cec800640 1 -- 172.21.15.116:0/946537223 wait complete. 2024-03-04T07:37:09.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:09.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:09.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:09.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:09.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:09.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:09 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:09.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:09 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2338652228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:09.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:09.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:09.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:09.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:09.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:09.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:09.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:09 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:09.958 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:09 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2338652228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.969+0000 7fd037600640 1 -- 172.21.15.116:0/3804755289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301015f0 msgr2=0x7fd030101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.969+0000 7fd037600640 1 --2- 172.21.15.116:0/3804755289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301015f0 0x7fd030101a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd024009860 tx=0x7fd02402bb70 comp rx=0 tx=0).stop 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.970+0000 7fd037600640 1 -- 172.21.15.116:0/3804755289 shutdown_connections 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.970+0000 7fd037600640 1 --2- 172.21.15.116:0/3804755289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301031a0 0x7fd030109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.970+0000 7fd037600640 1 --2- 172.21.15.116:0/3804755289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301027e0 0x7fd030102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.970+0000 7fd037600640 1 --2- 172.21.15.116:0/3804755289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301015f0 0x7fd030101a10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.970+0000 7fd037600640 1 -- 172.21.15.116:0/3804755289 >> 172.21.15.116:0/3804755289 conn(0x7fd0300fcb80 msgr2=0x7fd0300fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.970+0000 7fd037600640 1 -- 172.21.15.116:0/3804755289 shutdown_connections 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.970+0000 7fd037600640 1 -- 172.21.15.116:0/3804755289 wait complete. 2024-03-04T07:37:09.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.971+0000 7fd037600640 1 Processor -- start 2024-03-04T07:37:09.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.971+0000 7fd037600640 1 -- start start 2024-03-04T07:37:09.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.971+0000 7fd037600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301015f0 0x7fd030197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.971+0000 7fd037600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301027e0 0x7fd030197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.971+0000 7fd037600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301031a0 0x7fd03019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.971+0000 7fd037600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03010d240 con 0x7fd0301031a0 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd037600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03010d0c0 con 0x7fd0301027e0 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd037600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03010d3c0 con 0x7fd0301015f0 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd02fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301027e0 0x7fd030197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301015f0 0x7fd030197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301015f0 0x7fd030197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:38746/0 (socket says 172.21.15.116:38746) 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd02fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301027e0 0x7fd030197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50034/0 (socket says 172.21.15.116:50034) 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 -- 172.21.15.116:0/3187874682 learned_addr learned my addr 172.21.15.116:0/3187874682 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd035600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301031a0 0x7fd03019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 -- 172.21.15.116:0/3187874682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301027e0 msgr2=0x7fd030197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301027e0 0x7fd030197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 -- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301031a0 msgr2=0x7fd03019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301031a0 0x7fd03019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.972+0000 7fd034c00640 1 -- 172.21.15.116:0/3187874682 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03019f530 con 0x7fd0301015f0 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.973+0000 7fd034c00640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301015f0 0x7fd030197910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd02402c0b0 tx=0x7fd024032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.973+0000 7fd02d600640 1 -- 172.21.15.116:0/3187874682 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024041020 con 0x7fd0301015f0 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.973+0000 7fd02d600640 1 -- 172.21.15.116:0/3187874682 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd02402c9f0 con 0x7fd0301015f0 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.973+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03019f7c0 con 0x7fd0301015f0 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.973+0000 7fd02d600640 1 -- 172.21.15.116:0/3187874682 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024039040 con 0x7fd0301015f0 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.973+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03019fcb0 con 0x7fd0301015f0 2024-03-04T07:37:09.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.974+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd030108910 con 0x7fd0301015f0 2024-03-04T07:37:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.976+0000 7fd02d600640 1 -- 172.21.15.116:0/3187874682 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd024044020 con 0x7fd0301015f0 2024-03-04T07:37:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.977+0000 7fd02d600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd0180a1d80 0x7fd0180a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.977+0000 7fd02d600640 1 -- 172.21.15.116:0/3187874682 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd0240ef320 con 0x7fd0301015f0 2024-03-04T07:37:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.977+0000 7fd02fe00640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd0180a1d80 0x7fd0180a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.978+0000 7fd02fe00640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd0180a1d80 0x7fd0180a4240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd030198e20 tx=0x7fd01c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:09.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:09.981+0000 7fd02d600640 1 -- 172.21.15.116:0/3187874682 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd02404b020 con 0x7fd0301015f0 2024-03-04T07:37:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:09 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:10.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:09 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2338652228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.222+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd03002cff0 con 0x7fd0301015f0 2024-03-04T07:37:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.224+0000 7fd02d600640 1 -- 172.21.15.116:0/3187874682 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fd0240bcb50 con 0x7fd0301015f0 2024-03-04T07:37:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:10.225 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-04T07:37:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 1 2024-03-04T07:37:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd0180a1d80 msgr2=0x7fd0180a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd0180a1d80 0x7fd0180a4240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd030198e20 tx=0x7fd01c005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301015f0 msgr2=0x7fd030197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301015f0 0x7fd030197910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd02402c0b0 tx=0x7fd024032430 comp rx=0 tx=0).stop 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 shutdown_connections 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301031a0 0x7fd03019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0301027e0 0x7fd030197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd0180a1d80 0x7fd0180a4240 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 --2- 172.21.15.116:0/3187874682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0301015f0 0x7fd030197910 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.226+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 >> 172.21.15.116:0/3187874682 conn(0x7fd0300fcb80 msgr2=0x7fd030105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.227+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 shutdown_connections 2024-03-04T07:37:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:10.227+0000 7fd037600640 1 -- 172.21.15.116:0/3187874682 wait complete. 2024-03-04T07:37:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:10 smithi152 ceph-mon[37154]: Deploying daemon mds.l on smithi152 2024-03-04T07:37:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:10 smithi152 ceph-mon[37154]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:10 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/946537223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:10.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:10 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3187874682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:10 smithi116 ceph-mon[32150]: Deploying daemon mds.l on smithi152 2024-03-04T07:37:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:10 smithi116 ceph-mon[32150]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:10 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/946537223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:11.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:10 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3187874682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:10 smithi133 ceph-mon[36505]: Deploying daemon mds.l on smithi152 2024-03-04T07:37:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:10 smithi133 ceph-mon[36505]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:10 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/946537223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:10 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3187874682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:11.193 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:10 smithi152 systemd[1]: Starting Ceph mds.l for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:11.448 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 podman[67011]: 2024-03-04 07:37:11.194150256 +0000 UTC m=+0.113700617 container create 48ac649e0e10ebbaeb0bd90cd120f0a2567e16dc7745cf60871c4cdd1ce41225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:37:11.448 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 podman[67011]: 2024-03-04 07:37:11.098052278 +0000 UTC m=+0.017602646 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:11.448 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 podman[67011]: 2024-03-04 07:37:11.422413835 +0000 UTC m=+0.341964191 container init 48ac649e0e10ebbaeb0bd90cd120f0a2567e16dc7745cf60871c4cdd1ce41225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-04T07:37:11.448 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 podman[67011]: 2024-03-04 07:37:11.424945988 +0000 UTC m=+0.344496355 container start 48ac649e0e10ebbaeb0bd90cd120f0a2567e16dc7745cf60871c4cdd1ce41225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T07:37:11.448 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 ceph-mds[67030]: -- 172.21.15.152:0/1775211469 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5631d9261a40 con 0x5631d9234c00 2024-03-04T07:37:11.771 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l[67026]: starting mds.l at 2024-03-04T07:37:11.771 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 bash[67011]: 48ac649e0e10ebbaeb0bd90cd120f0a2567e16dc7745cf60871c4cdd1ce41225 2024-03-04T07:37:11.771 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:37:11 smithi152 systemd[1]: Started Ceph mds.l for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:12.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 -- 172.21.15.116:0/711037936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c81051d0 msgr2=0x7fa0c810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 --2- 172.21.15.116:0/711037936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c81051d0 0x7fa0c810ba60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009be0 tx=0x7fa0c002bef0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 -- 172.21.15.116:0/711037936 shutdown_connections 2024-03-04T07:37:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 --2- 172.21.15.116:0/711037936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c81051d0 0x7fa0c810ba60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 --2- 172.21.15.116:0/711037936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c8104810 0x7fa0c8104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 --2- 172.21.15.116:0/711037936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0c8103620 0x7fa0c8103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 -- 172.21.15.116:0/711037936 >> 172.21.15.116:0/711037936 conn(0x7fa0c80febb0 msgr2=0x7fa0c8100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 -- 172.21.15.116:0/711037936 shutdown_connections 2024-03-04T07:37:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.473+0000 7fa0cd400640 1 -- 172.21.15.116:0/711037936 wait complete. 2024-03-04T07:37:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 Processor -- start 2024-03-04T07:37:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 -- start start 2024-03-04T07:37:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c8103620 0x7fa0c81998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c8104810 0x7fa0c8199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0c81051d0 0x7fa0c81a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c810f2f0 con 0x7fa0c8104810 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c810f170 con 0x7fa0c8103620 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0cd400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c810f470 con 0x7fa0c81051d0 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.474+0000 7fa0c7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c8103620 0x7fa0c81998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c8104810 0x7fa0c8199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c8103620 0x7fa0c81998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50050/0 (socket says 172.21.15.116:50050) 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 -- 172.21.15.116:0/2564384533 learned_addr learned my addr 172.21.15.116:0/2564384533 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0cca00640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0c81051d0 0x7fa0c81a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:12.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 -- 172.21.15.116:0/2564384533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0c81051d0 msgr2=0x7fa0c81a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0c81051d0 0x7fa0c81a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 -- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c8104810 msgr2=0x7fa0c8199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c8104810 0x7fa0c8199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 -- 172.21.15.116:0/2564384533 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c81a14f0 con 0x7fa0c8103620 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0cca00640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0c81051d0 0x7fa0c81a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c7e00640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c8103620 0x7fa0c81998d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9f0 tx=0x7fa0b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.475+0000 7fa0c4c00640 1 -- 172.21.15.116:0/2564384533 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800ccc0 con 0x7fa0c8103620 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.476+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c81a17e0 con 0x7fa0c8103620 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.476+0000 7fa0c4c00640 1 -- 172.21.15.116:0/2564384533 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0b80035f0 con 0x7fa0c8103620 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.476+0000 7fa0c4c00640 1 -- 172.21.15.116:0/2564384533 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800ce60 con 0x7fa0c8103620 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.476+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c81a1d30 con 0x7fa0c8103620 2024-03-04T07:37:12.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.477+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c810a940 con 0x7fa0c8103620 2024-03-04T07:37:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.478+0000 7fa0c4c00640 1 -- 172.21.15.116:0/2564384533 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa0b8003b10 con 0x7fa0c8103620 2024-03-04T07:37:12.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.480+0000 7fa0c4c00640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0ac0a1d80 0x7fa0ac0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.480+0000 7fa0c4c00640 1 -- 172.21.15.116:0/2564384533 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa0b80cdc80 con 0x7fa0c8103620 2024-03-04T07:37:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.480+0000 7fa0c7400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0ac0a1d80 0x7fa0ac0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.480+0000 7fa0c7400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0ac0a1d80 0x7fa0ac0a4240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa0c819ade0 tx=0x7fa0b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:12.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.483+0000 7fa0c4c00640 1 -- 172.21.15.116:0/2564384533 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa0b809b550 con 0x7fa0c8103620 2024-03-04T07:37:12.510 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:12.510 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.152:6832/2796423979,v1:172.21.15.152:6833/2796423979] up:boot 2024-03-04T07:37:12.510 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: fsmap 1 up:standby 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:12.511 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:12 smithi133 ceph-mon[36505]: Deploying daemon mds.k on smithi133 2024-03-04T07:37:12.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.726+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0c802cff0 con 0x7fa0c8103620 2024-03-04T07:37:12.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.727+0000 7fa0c4c00640 1 -- 172.21.15.116:0/2564384533 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fa0b809ae50 con 0x7fa0c8103620 2024-03-04T07:37:12.728 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:12.728 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-04T07:37:12.729 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 2 2024-03-04T07:37:12.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0ac0a1d80 msgr2=0x7fa0ac0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:12.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0ac0a1d80 0x7fa0ac0a4240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa0c819ade0 tx=0x7fa0b4005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:12.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c8103620 msgr2=0x7fa0c81998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:12.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c8103620 0x7fa0c81998d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9f0 tx=0x7fa0b800eeb0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 shutdown_connections 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0ac0a1d80 0x7fa0ac0a4240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0c81051d0 0x7fa0c81a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0c8104810 0x7fa0c8199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 --2- 172.21.15.116:0/2564384533 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0c8103620 0x7fa0c81998d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 >> 172.21.15.116:0/2564384533 conn(0x7fa0c80febb0 msgr2=0x7fa0c8107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 shutdown_connections 2024-03-04T07:37:12.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:12.730+0000 7fa0cd400640 1 -- 172.21.15.116:0/2564384533 wait complete. 2024-03-04T07:37:12.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:12.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.152:6832/2796423979,v1:172.21.15.152:6833/2796423979] up:boot 2024-03-04T07:37:12.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: fsmap 1 up:standby 2024-03-04T07:37:12.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T07:37:12.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.747 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.748 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:12.748 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:12.748 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:12.748 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:12 smithi116 ceph-mon[32150]: Deploying daemon mds.k on smithi133 2024-03-04T07:37:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.152:6832/2796423979,v1:172.21.15.152:6833/2796423979] up:boot 2024-03-04T07:37:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: fsmap 1 up:standby 2024-03-04T07:37:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T07:37:12.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:12.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:12.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:12.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:12.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:12 smithi152 ceph-mon[37154]: Deploying daemon mds.k on smithi133 2024-03-04T07:37:13.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:13 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2564384533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:13.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:13 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2564384533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:13.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:13 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2564384533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:14.052 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:13 smithi133 systemd[1]: Starting Ceph mds.k for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:14.303 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 podman[66129]: 2024-03-04 07:37:13.953553601 +0000 UTC m=+0.017320706 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:14.304 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 podman[66129]: 2024-03-04 07:37:14.051398211 +0000 UTC m=+0.115165294 container create da4094da1eadb24e7b240bc2ccb19a71a1b19f6f497d46bf63e358a15ac27657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-04T07:37:14.304 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 podman[66129]: 2024-03-04 07:37:14.304113463 +0000 UTC m=+0.367880557 container init da4094da1eadb24e7b240bc2ccb19a71a1b19f6f497d46bf63e358a15ac27657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:14 smithi133 ceph-mon[36505]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:14 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 podman[66129]: 2024-03-04 07:37:14.306863278 +0000 UTC m=+0.370630367 container start da4094da1eadb24e7b240bc2ccb19a71a1b19f6f497d46bf63e358a15ac27657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 ceph-mds[66148]: -- 172.21.15.133:0/3368475385 <== mon.2 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b393e1da40 con 0x55b393df1000 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k[66144]: starting mds.k at 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 bash[66129]: da4094da1eadb24e7b240bc2ccb19a71a1b19f6f497d46bf63e358a15ac27657 2024-03-04T07:37:14.587 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:37:14 smithi133 systemd[1]: Started Ceph mds.k for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:14.753 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:14.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:14 smithi116 ceph-mon[32150]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:14.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:14.761 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:14 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:14 smithi152 ceph-mon[37154]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:14.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:14 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 -- 172.21.15.116:0/3973701303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1015f0 msgr2=0x7f4b8c101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 --2- 172.21.15.116:0/3973701303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1015f0 0x7f4b8c101a10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009900 tx=0x7f4b8002bc40 comp rx=0 tx=0).stop 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 -- 172.21.15.116:0/3973701303 shutdown_connections 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 --2- 172.21.15.116:0/3973701303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b8c1031a0 0x7f4b8c109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 --2- 172.21.15.116:0/3973701303 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b8c1027e0 0x7f4b8c102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 --2- 172.21.15.116:0/3973701303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1015f0 0x7f4b8c101a10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 -- 172.21.15.116:0/3973701303 >> 172.21.15.116:0/3973701303 conn(0x7f4b8c0fcb80 msgr2=0x7f4b8c0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 -- 172.21.15.116:0/3973701303 shutdown_connections 2024-03-04T07:37:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.960+0000 7f4b94800640 1 -- 172.21.15.116:0/3973701303 wait complete. 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.961+0000 7f4b94800640 1 Processor -- start 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.961+0000 7f4b94800640 1 -- start start 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.961+0000 7f4b94800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b8c1015f0 0x7f4b8c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.961+0000 7f4b94800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b8c1027e0 0x7f4b8c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b94800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1031a0 0x7f4b8c1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b94800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c10d2b0 con 0x7f4b8c1031a0 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b94800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c10d130 con 0x7f4b8c1015f0 2024-03-04T07:37:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b94800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c10d430 con 0x7f4b8c1027e0 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b8c1015f0 0x7f4b8c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b92800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1031a0 0x7f4b8c1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b8c1015f0 0x7f4b8c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50060/0 (socket says 172.21.15.116:50060) 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91e00640 1 -- 172.21.15.116:0/200408316 learned_addr learned my addr 172.21.15.116:0/200408316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91400640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b8c1027e0 0x7f4b8c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91400640 1 -- 172.21.15.116:0/200408316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b8c1015f0 msgr2=0x7f4b8c199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91400640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b8c1015f0 0x7f4b8c199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91400640 1 -- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1031a0 msgr2=0x7f4b8c1a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91400640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1031a0 0x7f4b8c1a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.962+0000 7f4b91400640 1 -- 172.21.15.116:0/200408316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c1a1710 con 0x7f4b8c1027e0 2024-03-04T07:37:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.963+0000 7f4b91400640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b8c1027e0 0x7f4b8c19a030 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00e9f0 tx=0x7f4b7c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.963+0000 7f4b86a00640 1 -- 172.21.15.116:0/200408316 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c00ccc0 con 0x7f4b8c1027e0 2024-03-04T07:37:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.963+0000 7f4b86a00640 1 -- 172.21.15.116:0/200408316 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b7c010910 con 0x7f4b8c1027e0 2024-03-04T07:37:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.963+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c1a1a00 con 0x7f4b8c1027e0 2024-03-04T07:37:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.963+0000 7f4b86a00640 1 -- 172.21.15.116:0/200408316 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c018960 con 0x7f4b8c1027e0 2024-03-04T07:37:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.963+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c1a1f50 con 0x7f4b8c1027e0 2024-03-04T07:37:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.964+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c1a2180 con 0x7f4b8c1027e0 2024-03-04T07:37:14.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.966+0000 7f4b86a00640 1 -- 172.21.15.116:0/200408316 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4b7c010ac0 con 0x7f4b8c1027e0 2024-03-04T07:37:14.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.968+0000 7f4b86a00640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4b740a1d80 0x7f4b740a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:14.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.968+0000 7f4b86a00640 1 -- 172.21.15.116:0/200408316 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4b7c0cdd10 con 0x7f4b8c1027e0 2024-03-04T07:37:14.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.968+0000 7f4b91e00640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4b740a1d80 0x7f4b740a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.969+0000 7f4b91e00640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4b740a1d80 0x7f4b740a4240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4b80005ab0 tx=0x7f4b8002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:14.972+0000 7f4b86a00640 1 -- 172.21.15.116:0/200408316 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4b7c0d3020 con 0x7f4b8c1027e0 2024-03-04T07:37:15.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.210+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b8c1a2350 con 0x7f4b8c1027e0 2024-03-04T07:37:15.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.212+0000 7f4b86a00640 1 -- 172.21.15.116:0/200408316 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7f4b7c09aef0 con 0x7f4b8c1027e0 2024-03-04T07:37:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:15.213 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-04T07:37:15.213 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-04T07:37:15.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4b740a1d80 msgr2=0x7f4b740a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:15.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4b740a1d80 0x7f4b740a4240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4b80005ab0 tx=0x7f4b8002cf90 comp rx=0 tx=0).stop 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b8c1027e0 msgr2=0x7f4b8c19a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b8c1027e0 0x7f4b8c19a030 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00e9f0 tx=0x7f4b7c00eeb0 comp rx=0 tx=0).stop 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 shutdown_connections 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1031a0 0x7f4b8c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4b740a1d80 0x7f4b740a4240 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b8c1027e0 0x7f4b8c19a030 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 --2- 172.21.15.116:0/200408316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4b8c1015f0 0x7f4b8c199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.214+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 >> 172.21.15.116:0/200408316 conn(0x7f4b8c0fcb80 msgr2=0x7f4b8c073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.215+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 shutdown_connections 2024-03-04T07:37:15.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:15.215+0000 7f4b94800640 1 -- 172.21.15.116:0/200408316 wait complete. 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.133:6832/20614029,v1:172.21.15.133:6833/20614029] up:boot 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: fsmap 2 up:standby 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: Deploying daemon mds.j on smithi116 2024-03-04T07:37:15.708 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/200408316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:15.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.133:6832/20614029,v1:172.21.15.133:6833/20614029] up:boot 2024-03-04T07:37:15.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: fsmap 2 up:standby 2024-03-04T07:37:15.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T07:37:15.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:15.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:15.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:15.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:15.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: Deploying daemon mds.j on smithi116 2024-03-04T07:37:15.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/200408316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.133:6832/20614029,v1:172.21.15.133:6833/20614029] up:boot 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: fsmap 2 up:standby 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: Deploying daemon mds.j on smithi116 2024-03-04T07:37:15.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/200408316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:16.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:16 smithi116 ceph-mon[32150]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:16.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:16 smithi152 ceph-mon[37154]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:16.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:16 smithi133 ceph-mon[36505]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:17.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:17.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.447+0000 7fba36e00640 1 -- 172.21.15.116:0/3204742824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 msgr2=0x7fba30104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.447+0000 7fba36e00640 1 --2- 172.21.15.116:0/3204742824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30104510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fba20009900 tx=0x7fba2002bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.447+0000 7fba36e00640 1 -- 172.21.15.116:0/3204742824 shutdown_connections 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.447+0000 7fba36e00640 1 --2- 172.21.15.116:0/3204742824 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba30107380 0x7fba30109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.448+0000 7fba36e00640 1 --2- 172.21.15.116:0/3204742824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba30104a50 0x7fba30106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.448+0000 7fba36e00640 1 --2- 172.21.15.116:0/3204742824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30104510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.448+0000 7fba36e00640 1 -- 172.21.15.116:0/3204742824 >> 172.21.15.116:0/3204742824 conn(0x7fba300fba70 msgr2=0x7fba300fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.448+0000 7fba36e00640 1 -- 172.21.15.116:0/3204742824 shutdown_connections 2024-03-04T07:37:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.448+0000 7fba36e00640 1 -- 172.21.15.116:0/3204742824 wait complete. 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 Processor -- start 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 -- start start 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba30104a50 0x7fba301000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba30107380 0x7fba30100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3010d340 con 0x7fba30102120 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3010d1c0 con 0x7fba30107380 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba36e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3010d4c0 con 0x7fba30104a50 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30101a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54420/0 (socket says 172.21.15.116:54420) 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.449+0000 7fba2fe00640 1 -- 172.21.15.116:0/2176424855 learned_addr learned my addr 172.21.15.116:0/2176424855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:17.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba34e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba30107380 0x7fba30100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba2f400640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba30104a50 0x7fba301000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba2fe00640 1 -- 172.21.15.116:0/2176424855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba30104a50 msgr2=0x7fba301000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba2fe00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba30104a50 0x7fba301000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba2fe00640 1 -- 172.21.15.116:0/2176424855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba30107380 msgr2=0x7fba30100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba2fe00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba30107380 0x7fba30100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba2fe00640 1 -- 172.21.15.116:0/2176424855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30100b50 con 0x7fba30102120 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba34e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba30107380 0x7fba30100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.450+0000 7fba2fe00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30101a90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fba20005ab0 tx=0x7fba2002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.451+0000 7fba2cc00640 1 -- 172.21.15.116:0/2176424855 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba20043020 con 0x7fba30102120 2024-03-04T07:37:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.451+0000 7fba2cc00640 1 -- 172.21.15.116:0/2176424855 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba200054e0 con 0x7fba30102120 2024-03-04T07:37:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.451+0000 7fba2cc00640 1 -- 172.21.15.116:0/2176424855 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba20003c70 con 0x7fba30102120 2024-03-04T07:37:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.451+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba30100de0 con 0x7fba30102120 2024-03-04T07:37:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.451+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba301a63f0 con 0x7fba30102120 2024-03-04T07:37:17.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.452+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba300fd560 con 0x7fba30102120 2024-03-04T07:37:17.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.454+0000 7fba2cc00640 1 -- 172.21.15.116:0/2176424855 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fba20046020 con 0x7fba30102120 2024-03-04T07:37:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.455+0000 7fba2cc00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fba140a1e00 0x7fba140a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.456+0000 7fba2cc00640 1 -- 172.21.15.116:0/2176424855 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fba200efdf0 con 0x7fba30102120 2024-03-04T07:37:17.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.456+0000 7fba2f400640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fba140a1e00 0x7fba140a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.457+0000 7fba2f400640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fba140a1e00 0x7fba140a42c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fba30101800 tx=0x7fba1c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.459+0000 7fba2cc00640 1 -- 172.21.15.116:0/2176424855 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba2004d020 con 0x7fba30102120 2024-03-04T07:37:17.614 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:17 smithi116 systemd[1]: Starting Ceph mds.j for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.700+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba3002cff0 con 0x7fba30102120 2024-03-04T07:37:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.701+0000 7fba2cc00640 1 -- 172.21.15.116:0/2176424855 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7fba200ef020 con 0x7fba30102120 2024-03-04T07:37:17.702 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:17.702 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-04T07:37:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-04T07:37:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.703+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fba140a1e00 msgr2=0x7fba140a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.703+0000 7fba36e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fba140a1e00 0x7fba140a42c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fba30101800 tx=0x7fba1c008040 comp rx=0 tx=0).stop 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.703+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 msgr2=0x7fba30101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.703+0000 7fba36e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30101a90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fba20005ab0 tx=0x7fba2002c6c0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 shutdown_connections 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fba30107380 0x7fba30100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fba140a1e00 0x7fba140a42c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fba30104a50 0x7fba301000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 --2- 172.21.15.116:0/2176424855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba30102120 0x7fba30101a90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 >> 172.21.15.116:0/2176424855 conn(0x7fba300fba70 msgr2=0x7fba30102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 shutdown_connections 2024-03-04T07:37:17.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:17.704+0000 7fba36e00640 1 -- 172.21.15.116:0/2176424855 wait complete. 2024-03-04T07:37:18.255 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:17 smithi116 podman[70229]: 2024-03-04 07:37:17.865074478 +0000 UTC m=+0.035937583 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:18.255 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:17 smithi116 podman[70229]: 2024-03-04 07:37:17.991420745 +0000 UTC m=+0.162283878 container create 61a6923b3de48b55eef09dfbe9edf2b778d9a1b39e6b7667fc8d12d6f6677243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T07:37:18.255 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:18 smithi116 podman[70229]: 2024-03-04 07:37:18.254104707 +0000 UTC m=+0.424967876 container init 61a6923b3de48b55eef09dfbe9edf2b778d9a1b39e6b7667fc8d12d6f6677243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T07:37:18.509 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:18 smithi116 podman[70229]: 2024-03-04 07:37:18.264348882 +0000 UTC m=+0.435212018 container start 61a6923b3de48b55eef09dfbe9edf2b778d9a1b39e6b7667fc8d12d6f6677243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-04T07:37:18.509 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mds[70248]: -- 172.21.15.116:0/2122411072 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561185771a40 con 0x561185744c00 2024-03-04T07:37:18.509 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j[70244]: starting mds.j at 2024-03-04T07:37:18.509 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:18 smithi116 bash[70229]: 61a6923b3de48b55eef09dfbe9edf2b778d9a1b39e6b7667fc8d12d6f6677243 2024-03-04T07:37:18.509 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:37:18 smithi116 systemd[1]: Started Ceph mds.j for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2176424855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:18.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:18 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2176424855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:19.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:18 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2176424855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:18 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:19.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:19 smithi116 ceph-mon[32150]: Deploying daemon mds.d on smithi116 2024-03-04T07:37:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:19 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.116:6834/666560370,v1:172.21.15.116:6835/666560370] up:boot 2024-03-04T07:37:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:19 smithi116 ceph-mon[32150]: fsmap 3 up:standby 2024-03-04T07:37:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:19 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T07:37:19.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:19 smithi133 ceph-mon[36505]: Deploying daemon mds.d on smithi116 2024-03-04T07:37:19.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:19 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.116:6834/666560370,v1:172.21.15.116:6835/666560370] up:boot 2024-03-04T07:37:19.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:19 smithi133 ceph-mon[36505]: fsmap 3 up:standby 2024-03-04T07:37:19.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:19 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T07:37:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 -- 172.21.15.116:0/2063973549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58781053c0 msgr2=0x7f587810bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 --2- 172.21.15.116:0/2063973549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58781053c0 0x7f587810bc50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5870009be0 tx=0x7f587002bef0 comp rx=0 tx=0).stop 2024-03-04T07:37:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 -- 172.21.15.116:0/2063973549 shutdown_connections 2024-03-04T07:37:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 --2- 172.21.15.116:0/2063973549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58781053c0 0x7f587810bc50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 --2- 172.21.15.116:0/2063973549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5878104a00 0x7f5878104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 --2- 172.21.15.116:0/2063973549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5878103810 0x7f5878103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 -- 172.21.15.116:0/2063973549 >> 172.21.15.116:0/2063973549 conn(0x7f58780fedc0 msgr2=0x7f58781011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:19.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 -- 172.21.15.116:0/2063973549 shutdown_connections 2024-03-04T07:37:19.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.929+0000 7f5880000640 1 -- 172.21.15.116:0/2063973549 wait complete. 2024-03-04T07:37:19.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.930+0000 7f5880000640 1 Processor -- start 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.930+0000 7f5880000640 1 -- start start 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f5880000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5878103810 0x7f5878199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f5880000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5878104a00 0x7f587819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f5880000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58781053c0 0x7f58781a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f5880000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587810f540 con 0x7f5878103810 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f5880000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587810f3c0 con 0x7f58781053c0 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f5880000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587810f6c0 con 0x7f5878104a00 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5878103810 0x7f5878199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5878103810 0x7f5878199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54444/0 (socket says 172.21.15.116:54444) 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 -- 172.21.15.116:0/2110891734 learned_addr learned my addr 172.21.15.116:0/2110891734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587e000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58781053c0 0x7f58781a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587cc00640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5878104a00 0x7f587819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:19.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 -- 172.21.15.116:0/2110891734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5878104a00 msgr2=0x7f587819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5878104a00 0x7f587819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 -- 172.21.15.116:0/2110891734 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58781053c0 msgr2=0x7f58781a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58781053c0 0x7f58781a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587d600640 1 -- 172.21.15.116:0/2110891734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58781a1770 con 0x7f5878103810 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.931+0000 7f587cc00640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5878104a00 0x7f587819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.932+0000 7f587d600640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5878103810 0x7f5878199b50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f586c00e9a0 tx=0x7f586c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.932+0000 7f5876000640 1 -- 172.21.15.116:0/2110891734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c00cd80 con 0x7f5878103810 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.932+0000 7f5876000640 1 -- 172.21.15.116:0/2110891734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f586c010910 con 0x7f5878103810 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.932+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58781a1a60 con 0x7f5878103810 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.932+0000 7f5876000640 1 -- 172.21.15.116:0/2110891734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c018a30 con 0x7f5878103810 2024-03-04T07:37:19.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.932+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58781a1fb0 con 0x7f5878103810 2024-03-04T07:37:19.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.933+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587810aad0 con 0x7f5878103810 2024-03-04T07:37:19.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.935+0000 7f5876000640 1 -- 172.21.15.116:0/2110891734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f586c010ac0 con 0x7f5878103810 2024-03-04T07:37:19.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.936+0000 7f5876000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58600a1d80 0x7f58600a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:19.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.936+0000 7f5876000640 1 -- 172.21.15.116:0/2110891734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f586c0cdd70 con 0x7f5878103810 2024-03-04T07:37:19.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.937+0000 7f587cc00640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58600a1d80 0x7f58600a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:19.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.937+0000 7f587cc00640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58600a1d80 0x7f58600a4240 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f587819b060 tx=0x7f5864008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:19.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:19.940+0000 7f5876000640 1 -- 172.21.15.116:0/2110891734 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f586c053280 con 0x7f5878103810 2024-03-04T07:37:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:19 smithi152 ceph-mon[37154]: Deploying daemon mds.d on smithi116 2024-03-04T07:37:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:19 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.116:6834/666560370,v1:172.21.15.116:6835/666560370] up:boot 2024-03-04T07:37:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:19 smithi152 ceph-mon[37154]: fsmap 3 up:standby 2024-03-04T07:37:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:19 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T07:37:20.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.178+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f587802cff0 con 0x7f5878103810 2024-03-04T07:37:20.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.179+0000 7f5876000640 1 -- 172.21.15.116:0/2110891734 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7f586c09b5a0 con 0x7f5878103810 2024-03-04T07:37:20.181 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:20.181 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-04T07:37:20.181 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 4 2024-03-04T07:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.182+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58600a1d80 msgr2=0x7f58600a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:20.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.182+0000 7f5880000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58600a1d80 0x7f58600a4240 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f587819b060 tx=0x7f5864008040 comp rx=0 tx=0).stop 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.182+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5878103810 msgr2=0x7f5878199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.182+0000 7f5880000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5878103810 0x7f5878199b50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f586c00e9a0 tx=0x7f586c00ee60 comp rx=0 tx=0).stop 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 shutdown_connections 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58781053c0 0x7f58781a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58600a1d80 0x7f58600a4240 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5878104a00 0x7f587819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 --2- 172.21.15.116:0/2110891734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5878103810 0x7f5878199b50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 >> 172.21.15.116:0/2110891734 conn(0x7f58780fedc0 msgr2=0x7f5878107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:20.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 shutdown_connections 2024-03-04T07:37:20.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:20.183+0000 7f5880000640 1 -- 172.21.15.116:0/2110891734 wait complete. 2024-03-04T07:37:20.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:20 smithi133 ceph-mon[36505]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:20.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:20 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2110891734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:20.917 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:20 smithi116 ceph-mon[32150]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:20.917 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:20 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2110891734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:20 smithi152 ceph-mon[37154]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:20 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2110891734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:21.921 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:21 smithi116 systemd[1]: Starting Ceph mds.d for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:22.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:22.233 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:21 smithi116 podman[70570]: 2024-03-04 07:37:21.82554426 +0000 UTC m=+0.031897882 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:22.234 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:21 smithi116 podman[70570]: 2024-03-04 07:37:21.921908943 +0000 UTC m=+0.128262597 container create dcb27b9d2112add20d58a298f88ef1e1a21b6077a1d8324a9fdcecfdfd00f5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0) 2024-03-04T07:37:22.234 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:22 smithi116 podman[70570]: 2024-03-04 07:37:22.233626418 +0000 UTC m=+0.439980053 container init dcb27b9d2112add20d58a298f88ef1e1a21b6077a1d8324a9fdcecfdfd00f5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-04T07:37:22.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 -- 172.21.15.116:0/346334037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff881011c0 msgr2=0x7eff88101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 --2- 172.21.15.116:0/346334037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff881011c0 0x7eff88101640 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7eff80009be0 tx=0x7eff8002bef0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 -- 172.21.15.116:0/346334037 shutdown_connections 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 --2- 172.21.15.116:0/346334037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff881011c0 0x7eff88101640 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 --2- 172.21.15.116:0/346334037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff88100820 0x7eff88100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 --2- 172.21.15.116:0/346334037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff880ffec0 0x7eff881002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 -- 172.21.15.116:0/346334037 >> 172.21.15.116:0/346334037 conn(0x7eff880fba60 msgr2=0x7eff880fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.410+0000 7eff8da00640 1 -- 172.21.15.116:0/346334037 shutdown_connections 2024-03-04T07:37:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.411+0000 7eff8da00640 1 -- 172.21.15.116:0/346334037 wait complete. 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.411+0000 7eff8da00640 1 Processor -- start 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff8da00640 1 -- start start 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff8da00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff880ffec0 0x7eff88199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff8da00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff88100820 0x7eff8819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff8da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff881011c0 0x7eff881a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff8da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8810f4b0 con 0x7eff881011c0 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff8da00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8810f330 con 0x7eff88100820 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff8da00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8810f630 con 0x7eff880ffec0 2024-03-04T07:37:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff87400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff881011c0 0x7eff881a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff87400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff881011c0 0x7eff881a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54466/0 (socket says 172.21.15.116:54466) 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff7fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff88100820 0x7eff8819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff87400640 1 -- 172.21.15.116:0/685882369 learned_addr learned my addr 172.21.15.116:0/685882369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.412+0000 7eff86a00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff880ffec0 0x7eff88199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff87400640 1 -- 172.21.15.116:0/685882369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff880ffec0 msgr2=0x7eff88199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff87400640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff880ffec0 0x7eff88199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff87400640 1 -- 172.21.15.116:0/685882369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff88100820 msgr2=0x7eff8819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff87400640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff88100820 0x7eff8819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff87400640 1 -- 172.21.15.116:0/685882369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff881a1740 con 0x7eff881011c0 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff7fe00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff88100820 0x7eff8819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff87400640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff881011c0 0x7eff881a10d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7eff80036040 tx=0x7eff800337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff7f400640 1 -- 172.21.15.116:0/685882369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80043020 con 0x7eff881011c0 2024-03-04T07:37:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff7f400640 1 -- 172.21.15.116:0/685882369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff80003b80 con 0x7eff881011c0 2024-03-04T07:37:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.413+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff881a19d0 con 0x7eff881011c0 2024-03-04T07:37:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.414+0000 7eff7f400640 1 -- 172.21.15.116:0/685882369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80034690 con 0x7eff881011c0 2024-03-04T07:37:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.414+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff881a1fa0 con 0x7eff881011c0 2024-03-04T07:37:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.415+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8810ad40 con 0x7eff881011c0 2024-03-04T07:37:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.417+0000 7eff7f400640 1 -- 172.21.15.116:0/685882369 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7eff80046020 con 0x7eff881011c0 2024-03-04T07:37:22.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.419+0000 7eff7f400640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff6c0a1d80 0x7eff6c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:22.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.419+0000 7eff7f400640 1 -- 172.21.15.116:0/685882369 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7eff800f00f0 con 0x7eff881011c0 2024-03-04T07:37:22.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.419+0000 7eff86a00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff6c0a1d80 0x7eff6c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:22.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.420+0000 7eff86a00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff6c0a1d80 0x7eff6c0a4240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff78009990 tx=0x7eff78009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:22.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.423+0000 7eff7f400640 1 -- 172.21.15.116:0/685882369 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7eff8004d020 con 0x7eff881011c0 2024-03-04T07:37:22.509 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:22 smithi116 podman[70570]: 2024-03-04 07:37:22.241909916 +0000 UTC m=+0.448263542 container start dcb27b9d2112add20d58a298f88ef1e1a21b6077a1d8324a9fdcecfdfd00f5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-04T07:37:22.509 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mds[70601]: -- 172.21.15.116:0/4290353614 <== mon.2 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55879dba3a40 con 0x55879db76c00 2024-03-04T07:37:22.509 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d[70586]: starting mds.d at 2024-03-04T07:37:22.509 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:22 smithi116 bash[70570]: dcb27b9d2112add20d58a298f88ef1e1a21b6077a1d8324a9fdcecfdfd00f5ac 2024-03-04T07:37:22.509 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:37:22 smithi116 systemd[1]: Started Ceph mds.d for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.670+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff880621d0 con 0x7eff881011c0 2024-03-04T07:37:22.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.671+0000 7eff7f400640 1 -- 172.21.15.116:0/685882369 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3701 (secure 0 0 0) 0x7eff800bd920 con 0x7eff881011c0 2024-03-04T07:37:22.672 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:22.672 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-04T07:37:22.672 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 5 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.673+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff6c0a1d80 msgr2=0x7eff6c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.673+0000 7eff8da00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff6c0a1d80 0x7eff6c0a4240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff78009990 tx=0x7eff78009110 comp rx=0 tx=0).stop 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff881011c0 msgr2=0x7eff881a10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff881011c0 0x7eff881a10d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7eff80036040 tx=0x7eff800337b0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 shutdown_connections 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff881011c0 0x7eff881a10d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff88100820 0x7eff8819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff6c0a1d80 0x7eff6c0a4240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 --2- 172.21.15.116:0/685882369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff880ffec0 0x7eff88199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 >> 172.21.15.116:0/685882369 conn(0x7eff880fba60 msgr2=0x7eff880fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 shutdown_connections 2024-03-04T07:37:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:22.674+0000 7eff8da00640 1 -- 172.21.15.116:0/685882369 wait complete. 2024-03-04T07:37:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:22 smithi133 ceph-mon[36505]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:22.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:22 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mon[32150]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:23.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:22 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:22 smithi152 ceph-mon[37154]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:22 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:23.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:23 smithi133 ceph-mon[36505]: Deploying daemon mds.e on smithi133 2024-03-04T07:37:23.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:23 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.116:6836/3275850329,v1:172.21.15.116:6837/3275850329] up:boot 2024-03-04T07:37:23.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:23 smithi133 ceph-mon[36505]: fsmap 4 up:standby 2024-03-04T07:37:23.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:23 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T07:37:23.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:23 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/685882369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:23 smithi116 ceph-mon[32150]: Deploying daemon mds.e on smithi133 2024-03-04T07:37:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:23 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.116:6836/3275850329,v1:172.21.15.116:6837/3275850329] up:boot 2024-03-04T07:37:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:23 smithi116 ceph-mon[32150]: fsmap 4 up:standby 2024-03-04T07:37:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:23 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T07:37:24.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:23 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/685882369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:23 smithi152 ceph-mon[37154]: Deploying daemon mds.e on smithi133 2024-03-04T07:37:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:23 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.116:6836/3275850329,v1:172.21.15.116:6837/3275850329] up:boot 2024-03-04T07:37:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:23 smithi152 ceph-mon[37154]: fsmap 4 up:standby 2024-03-04T07:37:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:23 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T07:37:24.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:23 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/685882369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:24.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:24.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:24 smithi133 ceph-mon[36505]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:24.837 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:24 smithi133 systemd[1]: Starting Ceph mds.e for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 -- 172.21.15.116:0/2515008337 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658107c10 msgr2=0x7f7658108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 --2- 172.21.15.116:0/2515008337 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658107c10 0x7f7658108030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7648009900 tx=0x7f764802bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 -- 172.21.15.116:0/2515008337 shutdown_connections 2024-03-04T07:37:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 --2- 172.21.15.116:0/2515008337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76581097c0 0x7f7658110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 --2- 172.21.15.116:0/2515008337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7658108e00 0x7f7658109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 --2- 172.21.15.116:0/2515008337 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658107c10 0x7f7658108030 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 -- 172.21.15.116:0/2515008337 >> 172.21.15.116:0/2515008337 conn(0x7f7658076be0 msgr2=0x7f7658079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 -- 172.21.15.116:0/2515008337 shutdown_connections 2024-03-04T07:37:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.911+0000 7f765e200640 1 -- 172.21.15.116:0/2515008337 wait complete. 2024-03-04T07:37:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.912+0000 7f765e200640 1 Processor -- start 2024-03-04T07:37:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.912+0000 7f765e200640 1 -- start start 2024-03-04T07:37:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f765e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658107c10 0x7f765819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f765e200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658108e00 0x7f765819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f765e200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76581097c0 0x7f76581a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f765e200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76581138b0 con 0x7f7658107c10 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f765e200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7658113730 con 0x7f7658108e00 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f765e200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7658113a30 con 0x7f76581097c0 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658107c10 0x7f765819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658107c10 0x7f765819def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54482/0 (socket says 172.21.15.116:54482) 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 -- 172.21.15.116:0/2675821505 learned_addr learned my addr 172.21.15.116:0/2675821505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f764ea00640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658108e00 0x7f765819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657e00640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76581097c0 0x7f76581a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 -- 172.21.15.116:0/2675821505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76581097c0 msgr2=0x7f76581a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76581097c0 0x7f76581a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 -- 172.21.15.116:0/2675821505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658108e00 msgr2=0x7f765819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:24.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658108e00 0x7f765819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.913+0000 7f7657400640 1 -- 172.21.15.116:0/2675821505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76581a5b10 con 0x7f7658107c10 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.914+0000 7f7657400640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658107c10 0x7f765819def0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7648000c00 tx=0x7f764802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.914+0000 7f7654c00640 1 -- 172.21.15.116:0/2675821505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648043020 con 0x7f7658107c10 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.914+0000 7f7654c00640 1 -- 172.21.15.116:0/2675821505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76480054e0 con 0x7f7658107c10 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.914+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76581a5da0 con 0x7f7658107c10 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.914+0000 7f7654c00640 1 -- 172.21.15.116:0/2675821505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648003ce0 con 0x7f7658107c10 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.914+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76581a6260 con 0x7f7658107c10 2024-03-04T07:37:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.915+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765810ef30 con 0x7f7658107c10 2024-03-04T07:37:24.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.917+0000 7f7654c00640 1 -- 172.21.15.116:0/2675821505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f7648046020 con 0x7f7658107c10 2024-03-04T07:37:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.919+0000 7f7654c00640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f763c0a1e00 0x7f763c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:24.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.919+0000 7f7654c00640 1 -- 172.21.15.116:0/2675821505 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f76480efa70 con 0x7f7658107c10 2024-03-04T07:37:24.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.919+0000 7f764ea00640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f763c0a1e00 0x7f763c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:24.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.920+0000 7f764ea00640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f763c0a1e00 0x7f763c0a42c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f765819f400 tx=0x7f764000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:24.923+0000 7f7654c00640 1 -- 172.21.15.116:0/2675821505 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f764804d020 con 0x7f7658107c10 2024-03-04T07:37:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:24 smithi116 ceph-mon[32150]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:24 smithi152 ceph-mon[37154]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:25.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.164+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f765802cff0 con 0x7f7658107c10 2024-03-04T07:37:25.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.165+0000 7f7654c00640 1 -- 172.21.15.116:0/2675821505 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3701 (secure 0 0 0) 0x7f76480bd2a0 con 0x7f7658107c10 2024-03-04T07:37:25.166 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:25.166 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-04T07:37:25.166 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 5 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f763c0a1e00 msgr2=0x7f763c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f763c0a1e00 0x7f763c0a42c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f765819f400 tx=0x7f764000b390 comp rx=0 tx=0).stop 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658107c10 msgr2=0x7f765819def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658107c10 0x7f765819def0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7648000c00 tx=0x7f764802c6c0 comp rx=0 tx=0).stop 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 shutdown_connections 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f763c0a1e00 0x7f763c0a42c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f76581097c0 0x7f76581a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7658108e00 0x7f765819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 --2- 172.21.15.116:0/2675821505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658107c10 0x7f765819def0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 >> 172.21.15.116:0/2675821505 conn(0x7f7658076be0 msgr2=0x7f765810c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 shutdown_connections 2024-03-04T07:37:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:25.168+0000 7f765e200640 1 -- 172.21.15.116:0/2675821505 wait complete. 2024-03-04T07:37:25.266 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:24 smithi133 podman[66455]: 2024-03-04 07:37:24.894030121 +0000 UTC m=+0.017715575 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:25.266 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:25 smithi133 podman[66455]: 2024-03-04 07:37:25.014406842 +0000 UTC m=+0.138092316 container create e8b7fc0b65d68f17d5be640f336610b4f37e0db5389ac2dedfb21316d44f0fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-04T07:37:25.266 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:25 smithi133 podman[66455]: 2024-03-04 07:37:25.26691204 +0000 UTC m=+0.390597503 container init e8b7fc0b65d68f17d5be640f336610b4f37e0db5389ac2dedfb21316d44f0fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-03-04T07:37:25.577 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:25 smithi133 podman[66455]: 2024-03-04 07:37:25.269698548 +0000 UTC m=+0.393384004 container start e8b7fc0b65d68f17d5be640f336610b4f37e0db5389ac2dedfb21316d44f0fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-04T07:37:25.577 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mds[66476]: -- 172.21.15.133:0/3923426113 <== mon.2 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559105a65a40 con 0x559105a38800 2024-03-04T07:37:25.577 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e[66472]: starting mds.e at 2024-03-04T07:37:25.577 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:25 smithi133 bash[66455]: e8b7fc0b65d68f17d5be640f336610b4f37e0db5389ac2dedfb21316d44f0fb7 2024-03-04T07:37:25.577 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:37:25 smithi133 systemd[1]: Started Ceph mds.e for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:25.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2675821505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:25.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:25 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:25 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2675821505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:25 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:25 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2675821505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:25 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:26.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:26 smithi152 ceph-mon[37154]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:26.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:26 smithi152 ceph-mon[37154]: Deploying daemon mds.f on smithi152 2024-03-04T07:37:26.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:26 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.133:6834/331803361,v1:172.21.15.133:6835/331803361] up:boot 2024-03-04T07:37:26.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:26 smithi152 ceph-mon[37154]: fsmap 5 up:standby 2024-03-04T07:37:26.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:26 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T07:37:26.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:26 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T07:37:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:26 smithi116 ceph-mon[32150]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:26 smithi116 ceph-mon[32150]: Deploying daemon mds.f on smithi152 2024-03-04T07:37:27.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:26 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.133:6834/331803361,v1:172.21.15.133:6835/331803361] up:boot 2024-03-04T07:37:27.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:26 smithi116 ceph-mon[32150]: fsmap 5 up:standby 2024-03-04T07:37:27.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:26 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T07:37:27.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:26 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T07:37:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:26 smithi133 ceph-mon[36505]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:26 smithi133 ceph-mon[36505]: Deploying daemon mds.f on smithi152 2024-03-04T07:37:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:26 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.133:6834/331803361,v1:172.21.15.133:6835/331803361] up:boot 2024-03-04T07:37:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:26 smithi133 ceph-mon[36505]: fsmap 5 up:standby 2024-03-04T07:37:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:26 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T07:37:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:26 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T07:37:27.189 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:27.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 -- 172.21.15.116:0/387555644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81027e0 msgr2=0x7fbdc8102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:27.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 --2- 172.21.15.116:0/387555644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81027e0 0x7fbdc8102c60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4009860 tx=0x7fbdb402bb70 comp rx=0 tx=0).stop 2024-03-04T07:37:27.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 -- 172.21.15.116:0/387555644 shutdown_connections 2024-03-04T07:37:27.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 --2- 172.21.15.116:0/387555644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdc81031a0 0x7fbdc8109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 --2- 172.21.15.116:0/387555644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81027e0 0x7fbdc8102c60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 --2- 172.21.15.116:0/387555644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81015f0 0x7fbdc8101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 -- 172.21.15.116:0/387555644 >> 172.21.15.116:0/387555644 conn(0x7fbdc80fcb80 msgr2=0x7fbdc80fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:27.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 -- 172.21.15.116:0/387555644 shutdown_connections 2024-03-04T07:37:27.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.408+0000 7fbdd0200640 1 -- 172.21.15.116:0/387555644 wait complete. 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.409+0000 7fbdd0200640 1 Processor -- start 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.409+0000 7fbdd0200640 1 -- start start 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdd0200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdc81015f0 0x7fbdc8197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdd0200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81027e0 0x7fbdc8197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdd0200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81031a0 0x7fbdc819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdd0200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc810d250 con 0x7fbdc81015f0 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdd0200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc810d0d0 con 0x7fbdc81031a0 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdd0200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc810d3d0 con 0x7fbdc81027e0 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81027e0 0x7fbdc8197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcd800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdc81015f0 0x7fbdc8197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81027e0 0x7fbdc8197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53360/0 (socket says 172.21.15.116:53360) 2024-03-04T07:37:27.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 -- 172.21.15.116:0/3167746664 learned_addr learned my addr 172.21.15.116:0/3167746664 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdce200640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81031a0 0x7fbdc819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 -- 172.21.15.116:0/3167746664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81031a0 msgr2=0x7fbdc819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81031a0 0x7fbdc819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 -- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdc81015f0 msgr2=0x7fbdc8197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdc81015f0 0x7fbdc8197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.410+0000 7fbdcce00640 1 -- 172.21.15.116:0/3167746664 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc819f520 con 0x7fbdc81027e0 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.411+0000 7fbdcce00640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81027e0 0x7fbdc8197e40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40349e0 tx=0x7fbdb4032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.411+0000 7fbdc6000640 1 -- 172.21.15.116:0/3167746664 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4041030 con 0x7fbdc81027e0 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.411+0000 7fbdc6000640 1 -- 172.21.15.116:0/3167746664 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbdb402c510 con 0x7fbdc81027e0 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.411+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc819f7b0 con 0x7fbdc81027e0 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.411+0000 7fbdc6000640 1 -- 172.21.15.116:0/3167746664 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4039040 con 0x7fbdc81027e0 2024-03-04T07:37:27.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.411+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc819fca0 con 0x7fbdc81027e0 2024-03-04T07:37:27.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.412+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc8108910 con 0x7fbdc81027e0 2024-03-04T07:37:27.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.414+0000 7fbdc6000640 1 -- 172.21.15.116:0/3167746664 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fbdb4044070 con 0x7fbdc81027e0 2024-03-04T07:37:27.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.415+0000 7fbdc6000640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbdb00a1d80 0x7fbdb00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:27.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.416+0000 7fbdc6000640 1 -- 172.21.15.116:0/3167746664 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbdb40eeed0 con 0x7fbdc81027e0 2024-03-04T07:37:27.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.416+0000 7fbdcd800640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbdb00a1d80 0x7fbdb00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:27.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.416+0000 7fbdcd800640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbdb00a1d80 0x7fbdb00a4240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc009530 tx=0x7fbdbc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:27.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.419+0000 7fbdc6000640 1 -- 172.21.15.116:0/3167746664 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbdb40bc700 con 0x7fbdc81027e0 2024-03-04T07:37:27.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.656+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdc802cff0 con 0x7fbdc81027e0 2024-03-04T07:37:27.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.658+0000 7fbdc6000640 1 -- 172.21.15.116:0/3167746664 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4492 (secure 0 0 0) 0x7fbdb40bc000 con 0x7fbdc81027e0 2024-03-04T07:37:27.659 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:27.659 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-04T07:37:27.659 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 6 2024-03-04T07:37:27.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbdb00a1d80 msgr2=0x7fbdb00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:27.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbdb00a1d80 0x7fbdb00a4240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc009530 tx=0x7fbdbc009260 comp rx=0 tx=0).stop 2024-03-04T07:37:27.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81027e0 msgr2=0x7fbdc8197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:27.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81027e0 0x7fbdc8197e40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbdb40349e0 tx=0x7fbdb4032450 comp rx=0 tx=0).stop 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 shutdown_connections 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbdc81031a0 0x7fbdc819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbdb00a1d80 0x7fbdb00a4240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbdc81027e0 0x7fbdc8197e40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 --2- 172.21.15.116:0/3167746664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdc81015f0 0x7fbdc8197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.660+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 >> 172.21.15.116:0/3167746664 conn(0x7fbdc80fcb80 msgr2=0x7fbdc8105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.661+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 shutdown_connections 2024-03-04T07:37:27.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:27.661+0000 7fbdd0200640 1 -- 172.21.15.116:0/3167746664 wait complete. 2024-03-04T07:37:27.771 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:27 smithi152 systemd[1]: Starting Ceph mds.f for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:28.212 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:27 smithi152 podman[67322]: 2024-03-04 07:37:27.810903671 +0000 UTC m=+0.017868209 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:28.212 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:27 smithi152 podman[67322]: 2024-03-04 07:37:27.926179141 +0000 UTC m=+0.133143652 container create 15259a754df582f202c0e65a9647504263ce8df5075ab5927f6ef2e17c265a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD) 2024-03-04T07:37:28.212 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:28 smithi152 podman[67322]: 2024-03-04 07:37:28.120690209 +0000 UTC m=+0.327654730 container init 15259a754df582f202c0e65a9647504263ce8df5075ab5927f6ef2e17c265a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T07:37:28.212 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:28 smithi152 podman[67322]: 2024-03-04 07:37:28.123351858 +0000 UTC m=+0.330316377 container start 15259a754df582f202c0e65a9647504263ce8df5075ab5927f6ef2e17c265a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5) 2024-03-04T07:37:28.213 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mds[67341]: -- 172.21.15.152:0/2765627362 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e72abefa40 con 0x55e72abc2c00 2024-03-04T07:37:28.213 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f[67337]: starting mds.f at 2024-03-04T07:37:28.213 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:28 smithi152 bash[67322]: 15259a754df582f202c0e65a9647504263ce8df5075ab5927f6ef2e17c265a37 2024-03-04T07:37:28.521 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:37:28 smithi152 systemd[1]: Started Ceph mds.f for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:28.911 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:28.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3167746664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:28.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:28.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:28.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:28.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:28.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:28.912 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3167746664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:28.979 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3167746664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:29.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:29.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 -- 172.21.15.116:0/38648045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4104a00 msgr2=0x7f05c4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:29.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 --2- 172.21.15.116:0/38648045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4104a00 0x7f05c4104e80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009900 tx=0x7f05b002bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:29.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 -- 172.21.15.116:0/38648045 shutdown_connections 2024-03-04T07:37:29.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 --2- 172.21.15.116:0/38648045 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05c41053c0 0x7f05c410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:29.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 --2- 172.21.15.116:0/38648045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4104a00 0x7f05c4104e80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:29.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 --2- 172.21.15.116:0/38648045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05c4103810 0x7f05c4103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:29.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 -- 172.21.15.116:0/38648045 >> 172.21.15.116:0/38648045 conn(0x7f05c40fedc0 msgr2=0x7f05c41011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:29.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 -- 172.21.15.116:0/38648045 shutdown_connections 2024-03-04T07:37:29.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.896+0000 7f05c9800640 1 -- 172.21.15.116:0/38648045 wait complete. 2024-03-04T07:37:29.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.897+0000 7f05c9800640 1 Processor -- start 2024-03-04T07:37:29.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.897+0000 7f05c9800640 1 -- start start 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c9800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4103810 0x7f05c4199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c9800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05c4104a00 0x7f05c419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c9800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05c41053c0 0x7f05c41a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c9800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c410f4c0 con 0x7f05c4103810 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c9800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c410f340 con 0x7f05c4104a00 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c9800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c410f640 con 0x7f05c41053c0 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4103810 0x7f05c4199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4103810 0x7f05c4199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51796/0 (socket says 172.21.15.116:51796) 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 -- 172.21.15.116:0/2063304897 learned_addr learned my addr 172.21.15.116:0/2063304897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2000640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05c4104a00 0x7f05c419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c3400640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05c41053c0 0x7f05c41a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 -- 172.21.15.116:0/2063304897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05c41053c0 msgr2=0x7f05c41a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05c41053c0 0x7f05c41a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 -- 172.21.15.116:0/2063304897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05c4104a00 msgr2=0x7f05c419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:29.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05c4104a00 0x7f05c419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2a00640 1 -- 172.21.15.116:0/2063304897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c41a1720 con 0x7f05c4103810 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.898+0000 7f05c2000640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05c4104a00 0x7f05c419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.899+0000 7f05c2a00640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4103810 0x7f05c4199b00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f05b400e9c0 tx=0x7f05b400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.899+0000 7f05bb400640 1 -- 172.21.15.116:0/2063304897 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400cd80 con 0x7f05c4103810 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.899+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c41a1a10 con 0x7f05c4103810 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.899+0000 7f05bb400640 1 -- 172.21.15.116:0/2063304897 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05b40035f0 con 0x7f05c4103810 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.899+0000 7f05bb400640 1 -- 172.21.15.116:0/2063304897 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4010690 con 0x7f05c4103810 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.899+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c41a1f80 con 0x7f05c4103810 2024-03-04T07:37:29.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.900+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c4068810 con 0x7f05c4103810 2024-03-04T07:37:29.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.902+0000 7f05bb400640 1 -- 172.21.15.116:0/2063304897 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f05b40037a0 con 0x7f05c4103810 2024-03-04T07:37:29.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.903+0000 7f05bb400640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05a80a1d80 0x7f05a80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:29.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.904+0000 7f05bb400640 1 -- 172.21.15.116:0/2063304897 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f05b40cdea0 con 0x7f05c4103810 2024-03-04T07:37:29.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.904+0000 7f05c2000640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05a80a1d80 0x7f05a80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:29.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.905+0000 7f05c2000640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05a80a1d80 0x7f05a80a4240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f05b002c240 tx=0x7f05b002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:29.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:29.908+0000 7f05bb400640 1 -- 172.21.15.116:0/2063304897 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f05b409b6d0 con 0x7f05c4103810 2024-03-04T07:37:29.991 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:29 smithi152 ceph-mon[37154]: Deploying daemon mds.c on smithi152 2024-03-04T07:37:29.991 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:29 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.152:6834/3438829914,v1:172.21.15.152:6835/3438829914] up:boot 2024-03-04T07:37:29.991 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:29 smithi152 ceph-mon[37154]: fsmap 6 up:standby 2024-03-04T07:37:29.991 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T07:37:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:29 smithi116 ceph-mon[32150]: Deploying daemon mds.c on smithi152 2024-03-04T07:37:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:29 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.152:6834/3438829914,v1:172.21.15.152:6835/3438829914] up:boot 2024-03-04T07:37:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:29 smithi116 ceph-mon[32150]: fsmap 6 up:standby 2024-03-04T07:37:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T07:37:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:29 smithi133 ceph-mon[36505]: Deploying daemon mds.c on smithi152 2024-03-04T07:37:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:29 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.152:6834/3438829914,v1:172.21.15.152:6835/3438829914] up:boot 2024-03-04T07:37:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:29 smithi133 ceph-mon[36505]: fsmap 6 up:standby 2024-03-04T07:37:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T07:37:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.143+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05c410ab30 con 0x7f05c4103810 2024-03-04T07:37:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.145+0000 7f05bb400640 1 -- 172.21.15.116:0/2063304897 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5286 (secure 0 0 0) 0x7f05b409afd0 con 0x7f05c4103810 2024-03-04T07:37:30.146 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:30.146 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-04T07:37:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 7 2024-03-04T07:37:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05a80a1d80 msgr2=0x7f05a80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05a80a1d80 0x7f05a80a4240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f05b002c240 tx=0x7f05b002df90 comp rx=0 tx=0).stop 2024-03-04T07:37:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4103810 msgr2=0x7f05c4199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4103810 0x7f05c4199b00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f05b400e9c0 tx=0x7f05b400ee80 comp rx=0 tx=0).stop 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 shutdown_connections 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05a80a1d80 0x7f05a80a4240 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05c41053c0 0x7f05c41a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05c4104a00 0x7f05c419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 --2- 172.21.15.116:0/2063304897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05c4103810 0x7f05c4199b00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.147+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 >> 172.21.15.116:0/2063304897 conn(0x7f05c40fedc0 msgr2=0x7f05c4107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.148+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 shutdown_connections 2024-03-04T07:37:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:30.148+0000 7f05c9800640 1 -- 172.21.15.116:0/2063304897 wait complete. 2024-03-04T07:37:30.618 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:30 smithi152 systemd[1]: Starting Ceph mds.c for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:30 smithi116 ceph-mon[32150]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:30.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:30 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2063304897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:30.897 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:30 smithi152 ceph-mon[37154]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:30.897 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:30 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2063304897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:30.897 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:30 smithi152 podman[67629]: 2024-03-04 07:37:30.650515178 +0000 UTC m=+0.018015453 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:30.898 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:30 smithi152 podman[67629]: 2024-03-04 07:37:30.774875951 +0000 UTC m=+0.142376271 container create 44d4152152c3db31c851b8931aa768cb22b988d2a477e5c7abd497852bba0662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-04T07:37:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:30 smithi133 ceph-mon[36505]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:30 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2063304897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:31.271 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:31 smithi152 podman[67629]: 2024-03-04 07:37:31.003287446 +0000 UTC m=+0.370787723 container init 44d4152152c3db31c851b8931aa768cb22b988d2a477e5c7abd497852bba0662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-04T07:37:31.271 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:31 smithi152 podman[67629]: 2024-03-04 07:37:31.006123799 +0000 UTC m=+0.373624052 container start 44d4152152c3db31c851b8931aa768cb22b988d2a477e5c7abd497852bba0662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-04T07:37:31.271 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:31 smithi152 ceph-mds[67648]: -- 172.21.15.152:0/4212675516 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f85c07fa40 con 0x55f85c052800 2024-03-04T07:37:31.271 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:31 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c[67644]: starting mds.c at 2024-03-04T07:37:31.271 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:31 smithi152 bash[67629]: 44d4152152c3db31c851b8931aa768cb22b988d2a477e5c7abd497852bba0662 2024-03-04T07:37:31.271 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:37:31 smithi152 systemd[1]: Started Ceph mds.c for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:32.167 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: Deploying daemon mds.h on smithi133 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.152:6836/603594525,v1:172.21.15.152:6837/603594525] up:boot 2024-03-04T07:37:32.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: fsmap 7 up:standby 2024-03-04T07:37:32.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:32 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T07:37:32.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 -- 172.21.15.116:0/2164361328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b8104810 msgr2=0x7f43b8104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 --2- 172.21.15.116:0/2164361328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b8104810 0x7f43b8104c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009900 tx=0x7f43a402bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 -- 172.21.15.116:0/2164361328 shutdown_connections 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 --2- 172.21.15.116:0/2164361328 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f43b81051d0 0x7f43b810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 --2- 172.21.15.116:0/2164361328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b8104810 0x7f43b8104c90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 --2- 172.21.15.116:0/2164361328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 0x7f43b8103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 -- 172.21.15.116:0/2164361328 >> 172.21.15.116:0/2164361328 conn(0x7f43b80febb0 msgr2=0x7f43b8100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 -- 172.21.15.116:0/2164361328 shutdown_connections 2024-03-04T07:37:32.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.387+0000 7f43bee00640 1 -- 172.21.15.116:0/2164361328 wait complete. 2024-03-04T07:37:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.388+0000 7f43bee00640 1 Processor -- start 2024-03-04T07:37:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bee00640 1 -- start start 2024-03-04T07:37:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bee00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 0x7f43b81998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bee00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f43b8104810 0x7f43b8199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b81051d0 0x7f43b81a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b810f2e0 con 0x7f43b81051d0 2024-03-04T07:37:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bee00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b810f160 con 0x7f43b8104810 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bee00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b810f460 con 0x7f43b8103620 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 0x7f43b81998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 0x7f43b81998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53392/0 (socket says 172.21.15.116:53392) 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 -- 172.21.15.116:0/2793173431 learned_addr learned my addr 172.21.15.116:0/2793173431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43be400640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b81051d0 0x7f43b81a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bd000640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f43b8104810 0x7f43b8199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 -- 172.21.15.116:0/2793173431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f43b8104810 msgr2=0x7f43b8199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f43b8104810 0x7f43b8199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 -- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b81051d0 msgr2=0x7f43b81a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b81051d0 0x7f43b81a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.389+0000 7f43bda00640 1 -- 172.21.15.116:0/2793173431 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b81a1510 con 0x7f43b8103620 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.390+0000 7f43bda00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 0x7f43b81998f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00d770 tx=0x7f43ac00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.390+0000 7f43b6000640 1 -- 172.21.15.116:0/2793173431 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac017070 con 0x7f43b8103620 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.390+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b81a1800 con 0x7f43b8103620 2024-03-04T07:37:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.390+0000 7f43b6000640 1 -- 172.21.15.116:0/2793173431 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43ac009c30 con 0x7f43b8103620 2024-03-04T07:37:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.390+0000 7f43b6000640 1 -- 172.21.15.116:0/2793173431 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac003dd0 con 0x7f43b8103620 2024-03-04T07:37:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.390+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b81a1d20 con 0x7f43b8103620 2024-03-04T07:37:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.391+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b810a940 con 0x7f43b8103620 2024-03-04T07:37:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.393+0000 7f43b6000640 1 -- 172.21.15.116:0/2793173431 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f43ac003f70 con 0x7f43b8103620 2024-03-04T07:37:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.394+0000 7f43b6000640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f43a00a1d80 0x7f43a00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.394+0000 7f43b6000640 1 -- 172.21.15.116:0/2793173431 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43ac0cd270 con 0x7f43b8103620 2024-03-04T07:37:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.394+0000 7f43bd000640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f43a00a1d80 0x7f43a00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.395+0000 7f43bd000640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f43a00a1d80 0x7f43a00a4240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f43a402c240 tx=0x7f43a402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:32.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.399+0000 7f43b6000640 1 -- 172.21.15.116:0/2793173431 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f43ac009de0 con 0x7f43b8103620 2024-03-04T07:37:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: Deploying daemon mds.h on smithi133 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.152:6836/603594525,v1:172.21.15.152:6837/603594525] up:boot 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: fsmap 7 up:standby 2024-03-04T07:37:32.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:32 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: Deploying daemon mds.h on smithi133 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.152:6836/603594525,v1:172.21.15.152:6837/603594525] up:boot 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: fsmap 7 up:standby 2024-03-04T07:37:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:32 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T07:37:32.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.636+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43b80621d0 con 0x7f43b8103620 2024-03-04T07:37:32.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.637+0000 7f43b6000640 1 -- 172.21.15.116:0/2793173431 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7f43ac09a450 con 0x7f43b8103620 2024-03-04T07:37:32.638 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:32.638 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-04T07:37:32.639 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f43a00a1d80 msgr2=0x7f43a00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f43a00a1d80 0x7f43a00a4240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f43a402c240 tx=0x7f43a402df90 comp rx=0 tx=0).stop 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 msgr2=0x7f43b81998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 0x7f43b81998f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00d770 tx=0x7f43ac00dc30 comp rx=0 tx=0).stop 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 shutdown_connections 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b81051d0 0x7f43b81a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f43b8104810 0x7f43b8199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f43a00a1d80 0x7f43a00a4240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 --2- 172.21.15.116:0/2793173431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b8103620 0x7f43b81998f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 >> 172.21.15.116:0/2793173431 conn(0x7f43b80febb0 msgr2=0x7f43b8107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:32.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 shutdown_connections 2024-03-04T07:37:32.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:32.640+0000 7f43bee00640 1 -- 172.21.15.116:0/2793173431 wait complete. 2024-03-04T07:37:33.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:33 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2793173431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:33.520 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:33 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2793173431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:33.545 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:33 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2793173431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:33.545 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 systemd[1]: Starting Ceph mds.h for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:33.545 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 podman[66766]: 2024-03-04 07:37:33.448497363 +0000 UTC m=+0.018541459 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:33.836 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 podman[66766]: 2024-03-04 07:37:33.577661502 +0000 UTC m=+0.147705576 container create fca8568dd783e833847f42bc164e928abc46314f02ed44f5db148e778c0ca501 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:37:34.200 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 podman[66766]: 2024-03-04 07:37:33.847045094 +0000 UTC m=+0.417089184 container init fca8568dd783e833847f42bc164e928abc46314f02ed44f5db148e778c0ca501 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-04T07:37:34.200 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 podman[66766]: 2024-03-04 07:37:33.849636076 +0000 UTC m=+0.419680160 container start fca8568dd783e833847f42bc164e928abc46314f02ed44f5db148e778c0ca501 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:37:34.200 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 ceph-mds[66785]: -- 172.21.15.133:0/227600955 <== mon.2 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c9b9a6da40 con 0x55c9b9a40c00 2024-03-04T07:37:34.200 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h[66781]: starting mds.h at 2024-03-04T07:37:34.201 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 bash[66766]: fca8568dd783e833847f42bc164e928abc46314f02ed44f5db148e778c0ca501 2024-03-04T07:37:34.201 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:37:33 smithi133 systemd[1]: Started Ceph mds.h for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:34 smithi116 ceph-mon[32150]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:34.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:34.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:34.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:34 smithi152 ceph-mon[37154]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:34.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:34.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:34 smithi133 ceph-mon[36505]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:34.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:34.663 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:34.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.882+0000 7f93b0e00640 1 -- 172.21.15.116:0/169960219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1031a0 msgr2=0x7f93ac109a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.882+0000 7f93b0e00640 1 --2- 172.21.15.116:0/169960219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1031a0 0x7f93ac109a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009900 tx=0x7f93a402bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.883+0000 7f93b0e00640 1 -- 172.21.15.116:0/169960219 shutdown_connections 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.883+0000 7f93b0e00640 1 --2- 172.21.15.116:0/169960219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1031a0 0x7f93ac109a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.883+0000 7f93b0e00640 1 --2- 172.21.15.116:0/169960219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1027e0 0x7f93ac102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.883+0000 7f93b0e00640 1 --2- 172.21.15.116:0/169960219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f93ac1015f0 0x7f93ac101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.883+0000 7f93b0e00640 1 -- 172.21.15.116:0/169960219 >> 172.21.15.116:0/169960219 conn(0x7f93ac0fcb80 msgr2=0x7f93ac0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.883+0000 7f93b0e00640 1 -- 172.21.15.116:0/169960219 shutdown_connections 2024-03-04T07:37:34.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.883+0000 7f93b0e00640 1 -- 172.21.15.116:0/169960219 wait complete. 2024-03-04T07:37:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 Processor -- start 2024-03-04T07:37:34.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 -- start start 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1015f0 0x7f93ac199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1027e0 0x7f93ac19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f93ac1031a0 0x7f93ac1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac10d2d0 con 0x7f93ac1027e0 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac10d150 con 0x7f93ac1031a0 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93b0e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac10d450 con 0x7f93ac1015f0 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93a9e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1015f0 0x7f93ac199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93a9e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1015f0 0x7f93ac199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53410/0 (socket says 172.21.15.116:53410) 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93a9e00640 1 -- 172.21.15.116:0/2705414506 learned_addr learned my addr 172.21.15.116:0/2705414506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93a9400640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1027e0 0x7f93ac19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:34.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.884+0000 7f93aa800640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f93ac1031a0 0x7f93ac1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a9e00640 1 -- 172.21.15.116:0/2705414506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f93ac1031a0 msgr2=0x7f93ac1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a9e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f93ac1031a0 0x7f93ac1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a9e00640 1 -- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1027e0 msgr2=0x7f93ac19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a9e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1027e0 0x7f93ac19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a9e00640 1 -- 172.21.15.116:0/2705414506 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac1a1720 con 0x7f93ac1015f0 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a9e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1015f0 0x7f93ac199b00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f939c00ebc0 tx=0x7f939c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a2a00640 1 -- 172.21.15.116:0/2705414506 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c003b70 con 0x7f93ac1015f0 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a2a00640 1 -- 172.21.15.116:0/2705414506 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f939c003d10 con 0x7f93ac1015f0 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93a2a00640 1 -- 172.21.15.116:0/2705414506 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c019980 con 0x7f93ac1015f0 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ac1a1a10 con 0x7f93ac1015f0 2024-03-04T07:37:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.885+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ac1a1f60 con 0x7f93ac1015f0 2024-03-04T07:37:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.886+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ac1a21f0 con 0x7f93ac1015f0 2024-03-04T07:37:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.888+0000 7f93a2a00640 1 -- 172.21.15.116:0/2705414506 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f939c004b60 con 0x7f93ac1015f0 2024-03-04T07:37:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.890+0000 7f93a2a00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93900a1d80 0x7f93900a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.890+0000 7f93a2a00640 1 -- 172.21.15.116:0/2705414506 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f939c0ce1a0 con 0x7f93ac1015f0 2024-03-04T07:37:34.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.890+0000 7f93a9400640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93900a1d80 0x7f93900a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:34.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.890+0000 7f93a9400640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93900a1d80 0x7f93900a4240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f93ac19b010 tx=0x7f9394005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:34.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:34.892+0000 7f93a2a00640 1 -- 172.21.15.116:0/2705414506 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f939c09b9d0 con 0x7f93ac1015f0 2024-03-04T07:37:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.130+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ac1a28c0 con 0x7f93ac1015f0 2024-03-04T07:37:35.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.131+0000 7f93a2a00640 1 -- 172.21.15.116:0/2705414506 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6868 (secure 0 0 0) 0x7f939c09b2d0 con 0x7f93ac1015f0 2024-03-04T07:37:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:35.238 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-04T07:37:35.239 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 9 2024-03-04T07:37:35.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93900a1d80 msgr2=0x7f93900a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:35.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93900a1d80 0x7f93900a4240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f93ac19b010 tx=0x7f9394005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:35.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1015f0 msgr2=0x7f93ac199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:35.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1015f0 0x7f93ac199b00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f939c00ebc0 tx=0x7f939c00c6a0 comp rx=0 tx=0).stop 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 shutdown_connections 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f93ac1031a0 0x7f93ac1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93ac1027e0 0x7f93ac19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93900a1d80 0x7f93900a4240 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 --2- 172.21.15.116:0/2705414506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93ac1015f0 0x7f93ac199b00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 >> 172.21.15.116:0/2705414506 conn(0x7f93ac0fcb80 msgr2=0x7f93ac073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 shutdown_connections 2024-03-04T07:37:35.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:35.134+0000 7f93b0e00640 1 -- 172.21.15.116:0/2705414506 wait complete. 2024-03-04T07:37:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:35 smithi116 ceph-mon[32150]: Deploying daemon mds.i on smithi152 2024-03-04T07:37:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:35 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.133:6836/336393981,v1:172.21.15.133:6837/336393981] up:boot 2024-03-04T07:37:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:35 smithi116 ceph-mon[32150]: fsmap 8 up:standby 2024-03-04T07:37:35.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:35 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T07:37:35.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:35 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2705414506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:35.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:35 smithi152 ceph-mon[37154]: Deploying daemon mds.i on smithi152 2024-03-04T07:37:35.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:35 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.133:6836/336393981,v1:172.21.15.133:6837/336393981] up:boot 2024-03-04T07:37:35.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:35 smithi152 ceph-mon[37154]: fsmap 8 up:standby 2024-03-04T07:37:35.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:35 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T07:37:35.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:35 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2705414506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:35.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:35 smithi133 ceph-mon[36505]: Deploying daemon mds.i on smithi152 2024-03-04T07:37:35.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:35 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.133:6836/336393981,v1:172.21.15.133:6837/336393981] up:boot 2024-03-04T07:37:35.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:35 smithi133 ceph-mon[36505]: fsmap 8 up:standby 2024-03-04T07:37:35.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:35 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T07:37:35.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:35 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2705414506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:36.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:36 smithi152 ceph-mon[37154]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:36.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:36 smithi116 ceph-mon[32150]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:36.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:36 smithi133 ceph-mon[36505]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:37.021 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 04 07:37:36 smithi152 systemd[1]: Started Ceph mds.i for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:37.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 -- 172.21.15.116:0/1877218111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec8101ea0 msgr2=0x7f3ec8104290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 --2- 172.21.15.116:0/1877218111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8104290 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009900 tx=0x7f3ebc02bc40 comp rx=0 tx=0).stop 2024-03-04T07:37:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 -- 172.21.15.116:0/1877218111 shutdown_connections 2024-03-04T07:37:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 --2- 172.21.15.116:0/1877218111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ec8107100 0x7f3ec8109550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 --2- 172.21.15.116:0/1877218111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec81047d0 0x7f3ec8106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 --2- 172.21.15.116:0/1877218111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8104290 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 -- 172.21.15.116:0/1877218111 >> 172.21.15.116:0/1877218111 conn(0x7f3ec80fb8b0 msgr2=0x7f3ec80fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 -- 172.21.15.116:0/1877218111 shutdown_connections 2024-03-04T07:37:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.450+0000 7f3ece400640 1 -- 172.21.15.116:0/1877218111 wait complete. 2024-03-04T07:37:37.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.451+0000 7f3ece400640 1 Processor -- start 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.451+0000 7f3ece400640 1 -- start start 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.451+0000 7f3ece400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ece400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec81047d0 0x7f3ec8199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ece400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ec8107100 0x7f3ec81a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ece400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec810d0b0 con 0x7f3ec8101ea0 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ece400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec810cf30 con 0x7f3ec81047d0 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ece400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec810d230 con 0x7f3ec8107100 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33070/0 (socket says 172.21.15.116:33070) 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 -- 172.21.15.116:0/1200055763 learned_addr learned my addr 172.21.15.116:0/1200055763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecda00640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ec8107100 0x7f3ec81a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ec7e00640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec81047d0 0x7f3ec8199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 -- 172.21.15.116:0/1200055763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ec8107100 msgr2=0x7f3ec81a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ec8107100 0x7f3ec81a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 -- 172.21.15.116:0/1200055763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec81047d0 msgr2=0x7f3ec8199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec81047d0 0x7f3ec8199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.452+0000 7f3ecd000640 1 -- 172.21.15.116:0/1200055763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec81a1580 con 0x7f3ec8101ea0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.453+0000 7f3ecd000640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8199960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009230 tx=0x7f3ebc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.453+0000 7f3ec5600640 1 -- 172.21.15.116:0/1200055763 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc041070 con 0x7f3ec8101ea0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.453+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec81a1810 con 0x7f3ec8101ea0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.453+0000 7f3ec5600640 1 -- 172.21.15.116:0/1200055763 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ebc0044d0 con 0x7f3ec8101ea0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.453+0000 7f3ec5600640 1 -- 172.21.15.116:0/1200055763 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc034440 con 0x7f3ec8101ea0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.453+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec81a1d00 con 0x7f3ec8101ea0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.455+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8108c50 con 0x7f3ec8101ea0 2024-03-04T07:37:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.456+0000 7f3ec5600640 1 -- 172.21.15.116:0/1200055763 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3ebc044070 con 0x7f3ec8101ea0 2024-03-04T07:37:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.457+0000 7f3ec5600640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3eb00a1d80 0x7f3eb00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.458+0000 7f3ec5600640 1 -- 172.21.15.116:0/1200055763 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3ebc0ef5c0 con 0x7f3ec8101ea0 2024-03-04T07:37:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.458+0000 7f3ec7e00640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3eb00a1d80 0x7f3eb00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.459+0000 7f3ec7e00640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3eb00a1d80 0x7f3eb00a4240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ec819ae70 tx=0x7f3eb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.463+0000 7f3ec5600640 1 -- 172.21.15.116:0/1200055763 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3ebc0bc7a0 con 0x7f3ec8101ea0 2024-03-04T07:37:37.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.734+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ec80621d0 con 0x7f3ec8101ea0 2024-03-04T07:37:37.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.735+0000 7f3ec5600640 1 -- 172.21.15.116:0/1200055763 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7664 (secure 0 0 0) 0x7f3ebc0c0840 con 0x7f3ec8101ea0 2024-03-04T07:37:37.735 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:37.735 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-04T07:37:37.735 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 10 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3eb00a1d80 msgr2=0x7f3eb00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3eb00a1d80 0x7f3eb00a4240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ec819ae70 tx=0x7f3eb4005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec8101ea0 msgr2=0x7f3ec8199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8199960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009230 tx=0x7f3ebc005d50 comp rx=0 tx=0).stop 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 shutdown_connections 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3eb00a1d80 0x7f3eb00a4240 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ec8107100 0x7f3ec81a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3ec81047d0 0x7f3ec8199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 --2- 172.21.15.116:0/1200055763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec8101ea0 0x7f3ec8199960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:37.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 >> 172.21.15.116:0/1200055763 conn(0x7f3ec80fb8b0 msgr2=0x7f3ec81015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:37.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 shutdown_connections 2024-03-04T07:37:37.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:37.738+0000 7f3ece400640 1 -- 172.21.15.116:0/1200055763 wait complete. 2024-03-04T07:37:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.152:6838/3832975468,v1:172.21.15.152:6839/3832975468] up:boot 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: fsmap 9 up:standby 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T07:37:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:37 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1200055763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.152:6838/3832975468,v1:172.21.15.152:6839/3832975468] up:boot 2024-03-04T07:37:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: fsmap 9 up:standby 2024-03-04T07:37:38.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T07:37:38.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:37 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1200055763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.152:6838/3832975468,v1:172.21.15.152:6839/3832975468] up:boot 2024-03-04T07:37:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: fsmap 9 up:standby 2024-03-04T07:37:38.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T07:37:38.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:37 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1200055763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:38 smithi133 ceph-mon[36505]: Deploying daemon mds.g on smithi116 2024-03-04T07:37:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:38 smithi133 ceph-mon[36505]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:39.200 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:38 smithi116 ceph-mon[32150]: Deploying daemon mds.g on smithi116 2024-03-04T07:37:39.201 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:38 smithi116 ceph-mon[32150]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:38 smithi152 ceph-mon[37154]: Deploying daemon mds.g on smithi116 2024-03-04T07:37:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:38 smithi152 ceph-mon[37154]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:39.757 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:39.976 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:39 smithi116 ceph-mon[32150]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:40.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:39 smithi133 ceph-mon[36505]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:40.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.104+0000 7f64d4c00640 1 -- 172.21.15.116:0/2463976595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 msgr2=0x7f64d0073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.104+0000 7f64d4c00640 1 --2- 172.21.15.116:0/2463976595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 0x7f64d0073a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009900 tx=0x7f64c002bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.105+0000 7f64d4c00640 1 -- 172.21.15.116:0/2463976595 shutdown_connections 2024-03-04T07:37:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.105+0000 7f64d4c00640 1 --2- 172.21.15.116:0/2463976595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64d01094b0 0x7f64d010b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.105+0000 7f64d4c00640 1 --2- 172.21.15.116:0/2463976595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d0074030 0x7f64d00744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.105+0000 7f64d4c00640 1 --2- 172.21.15.116:0/2463976595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 0x7f64d0073a60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.105+0000 7f64d4c00640 1 -- 172.21.15.116:0/2463976595 >> 172.21.15.116:0/2463976595 conn(0x7f64d00fcb80 msgr2=0x7f64d00fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:40.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.105+0000 7f64d4c00640 1 -- 172.21.15.116:0/2463976595 shutdown_connections 2024-03-04T07:37:40.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.105+0000 7f64d4c00640 1 -- 172.21.15.116:0/2463976595 wait complete. 2024-03-04T07:37:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 Processor -- start 2024-03-04T07:37:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 -- start start 2024-03-04T07:37:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64d0074030 0x7f64d0199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 0x7f64d0199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d01094b0 0x7f64d01a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d010f450 con 0x7f64d01094b0 2024-03-04T07:37:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d010f2d0 con 0x7f64d0074030 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64d4c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d010f5d0 con 0x7f64d0075610 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64ce800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d01094b0 0x7f64d01a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64cde00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64d0074030 0x7f64d0199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64c7e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 0x7f64d0199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64ce800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d01094b0 0x7f64d01a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33076/0 (socket says 172.21.15.116:33076) 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.106+0000 7f64ce800640 1 -- 172.21.15.116:0/4034546316 learned_addr learned my addr 172.21.15.116:0/4034546316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c7e00640 1 -- 172.21.15.116:0/4034546316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64d0074030 msgr2=0x7f64d0199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c7e00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64d0074030 0x7f64d0199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c7e00640 1 -- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d01094b0 msgr2=0x7f64d01a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c7e00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d01094b0 0x7f64d01a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c7e00640 1 -- 172.21.15.116:0/4034546316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d01a16c0 con 0x7f64d0075610 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64ce800640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d01094b0 0x7f64d01a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64cde00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64d0074030 0x7f64d0199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c7e00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 0x7f64d0199fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f64b800e9f0 tx=0x7f64b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c6a00640 1 -- 172.21.15.116:0/4034546316 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b800cd80 con 0x7f64d0075610 2024-03-04T07:37:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64c6a00640 1 -- 172.21.15.116:0/4034546316 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64b80035f0 con 0x7f64d0075610 2024-03-04T07:37:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.107+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d01a19b0 con 0x7f64d0075610 2024-03-04T07:37:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.108+0000 7f64c6a00640 1 -- 172.21.15.116:0/4034546316 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8010690 con 0x7f64d0075610 2024-03-04T07:37:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.108+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d01a1f00 con 0x7f64d0075610 2024-03-04T07:37:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.109+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d01081e0 con 0x7f64d0075610 2024-03-04T07:37:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.111+0000 7f64c6a00640 1 -- 172.21.15.116:0/4034546316 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f64b8003b30 con 0x7f64d0075610 2024-03-04T07:37:40.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.112+0000 7f64c6a00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f64b40a1d80 0x7f64b40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:40.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.112+0000 7f64c6a00640 1 -- 172.21.15.116:0/4034546316 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64b80cdc60 con 0x7f64d0075610 2024-03-04T07:37:40.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.112+0000 7f64cde00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f64b40a1d80 0x7f64b40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:40.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.113+0000 7f64cde00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f64b40a1d80 0x7f64b40a4240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64c002c7b0 tx=0x7f64c0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:40.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.116+0000 7f64c6a00640 1 -- 172.21.15.116:0/4034546316 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f64b809b490 con 0x7f64d0075610 2024-03-04T07:37:40.172 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:39 smithi152 ceph-mon[37154]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:40.258 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 systemd[1]: Starting Ceph mds.g for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.385+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64d00621d0 con 0x7f64d0075610 2024-03-04T07:37:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.386+0000 7f64c6a00640 1 -- 172.21.15.116:0/4034546316 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7664 (secure 0 0 0) 0x7f64b809ad90 con 0x7f64d0075610 2024-03-04T07:37:40.386 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:40.387 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-04T07:37:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 10 2024-03-04T07:37:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f64b40a1d80 msgr2=0x7f64b40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f64b40a1d80 0x7f64b40a4240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64c002c7b0 tx=0x7f64c0005fd0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 msgr2=0x7f64d0199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 0x7f64d0199fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f64b800e9f0 tx=0x7f64b800eeb0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 shutdown_connections 2024-03-04T07:37:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64d01094b0 0x7f64d01a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f64b40a1d80 0x7f64b40a4240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64d0075610 0x7f64d0199fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 --2- 172.21.15.116:0/4034546316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64d0074030 0x7f64d0199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 >> 172.21.15.116:0/4034546316 conn(0x7f64d00fcb80 msgr2=0x7f64d01053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 shutdown_connections 2024-03-04T07:37:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:40.389+0000 7f64d4c00640 1 -- 172.21.15.116:0/4034546316 wait complete. 2024-03-04T07:37:40.683 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 podman[71216]: 2024-03-04 07:37:40.281082667 +0000 UTC m=+0.031287901 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:40.683 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 podman[71216]: 2024-03-04 07:37:40.393858399 +0000 UTC m=+0.144063665 container create 6f4a49fbf911f726872920823c5a75a33dbbbe82cfaadd9a65f3028f86475704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:37:40.683 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 podman[71216]: 2024-03-04 07:37:40.681599772 +0000 UTC m=+0.431805045 container init 6f4a49fbf911f726872920823c5a75a33dbbbe82cfaadd9a65f3028f86475704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-04T07:37:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:40 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4034546316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:41.008 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 podman[71216]: 2024-03-04 07:37:40.691816535 +0000 UTC m=+0.442021807 container start 6f4a49fbf911f726872920823c5a75a33dbbbe82cfaadd9a65f3028f86475704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-04T07:37:41.008 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 ceph-mds[71237]: -- 172.21.15.116:0/4215993996 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d320dafa40 con 0x55d320d83000 2024-03-04T07:37:41.008 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g[71233]: starting mds.g at 2024-03-04T07:37:41.008 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 bash[71216]: 6f4a49fbf911f726872920823c5a75a33dbbbe82cfaadd9a65f3028f86475704 2024-03-04T07:37:41.008 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:37:40 smithi116 systemd[1]: Started Ceph mds.g for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:40 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4034546316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:41.185 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:40 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4034546316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.116:6838/1000386690,v1:172.21.15.116:6839/1000386690] up:boot 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: fsmap 10 up:standby 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:42.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: Deploying daemon mds.a on smithi116 2024-03-04T07:37:42.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:41 smithi133 ceph-mon[36505]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:42.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.116:6838/1000386690,v1:172.21.15.116:6839/1000386690] up:boot 2024-03-04T07:37:42.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: fsmap 10 up:standby 2024-03-04T07:37:42.123 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: Deploying daemon mds.a on smithi116 2024-03-04T07:37:42.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:41 smithi116 ceph-mon[32150]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:42.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.116:6838/1000386690,v1:172.21.15.116:6839/1000386690] up:boot 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: fsmap 10 up:standby 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: Deploying daemon mds.a on smithi116 2024-03-04T07:37:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:41 smithi152 ceph-mon[37154]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:42.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 -- 172.21.15.116:0/3832522690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 msgr2=0x7fcfec104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 --2- 172.21.15.116:0/3832522690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec104c90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009900 tx=0x7fcfd802bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 -- 172.21.15.116:0/3832522690 shutdown_connections 2024-03-04T07:37:42.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 --2- 172.21.15.116:0/3832522690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfec1051d0 0x7fcfec10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 --2- 172.21.15.116:0/3832522690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec104c90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 --2- 172.21.15.116:0/3832522690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfec103620 0x7fcfec103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 -- 172.21.15.116:0/3832522690 >> 172.21.15.116:0/3832522690 conn(0x7fcfec0febb0 msgr2=0x7fcfec100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:42.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 -- 172.21.15.116:0/3832522690 shutdown_connections 2024-03-04T07:37:42.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.626+0000 7fcff3000640 1 -- 172.21.15.116:0/3832522690 wait complete. 2024-03-04T07:37:42.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.627+0000 7fcff3000640 1 Processor -- start 2024-03-04T07:37:42.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.627+0000 7fcff3000640 1 -- start start 2024-03-04T07:37:42.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff3000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfec103620 0x7fcfec1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:42.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff3000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:42.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff3000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfec1051d0 0x7fcfec1a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:42.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff3000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec10f2e0 con 0x7fcfec104810 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff3000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec10f160 con 0x7fcfec103620 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff3000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec10f460 con 0x7fcfec1051d0 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33118/0 (socket says 172.21.15.116:33118) 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 -- 172.21.15.116:0/3094893804 learned_addr learned my addr 172.21.15.116:0/3094893804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1c00640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfec103620 0x7fcfec1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff2600640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfec1051d0 0x7fcfec1a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 -- 172.21.15.116:0/3094893804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfec1051d0 msgr2=0x7fcfec1a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfec1051d0 0x7fcfec1a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 -- 172.21.15.116:0/3094893804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfec103620 msgr2=0x7fcfec1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfec103620 0x7fcfec1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1200640 1 -- 172.21.15.116:0/3094893804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec1a1510 con 0x7fcfec104810 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff1c00640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfec103620 0x7fcfec1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:42.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.628+0000 7fcff2600640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfec1051d0 0x7fcfec1a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:42.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.629+0000 7fcff1200640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec199e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8000c00 tx=0x7fcfd802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:42.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.629+0000 7fcfea000640 1 -- 172.21.15.116:0/3094893804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8043020 con 0x7fcfec104810 2024-03-04T07:37:42.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.629+0000 7fcfea000640 1 -- 172.21.15.116:0/3094893804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd80054e0 con 0x7fcfec104810 2024-03-04T07:37:42.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.629+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec1a17a0 con 0x7fcfec104810 2024-03-04T07:37:42.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.629+0000 7fcfea000640 1 -- 172.21.15.116:0/3094893804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8003c40 con 0x7fcfec104810 2024-03-04T07:37:42.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.629+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec1a1cf0 con 0x7fcfec104810 2024-03-04T07:37:42.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.630+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec10a940 con 0x7fcfec104810 2024-03-04T07:37:42.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.632+0000 7fcfea000640 1 -- 172.21.15.116:0/3094893804 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fcfd8046020 con 0x7fcfec104810 2024-03-04T07:37:42.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.633+0000 7fcfea000640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfd40a1d80 0x7fcfd40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:42.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.634+0000 7fcfea000640 1 -- 172.21.15.116:0/3094893804 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcfd80ef990 con 0x7fcfec104810 2024-03-04T07:37:42.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.634+0000 7fcff1c00640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfd40a1d80 0x7fcfd40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:42.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.634+0000 7fcff1c00640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfd40a1d80 0x7fcfd40a4240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcfec104670 tx=0x7fcfe0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.638+0000 7fcfea000640 1 -- 172.21.15.116:0/3094893804 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcfd804d020 con 0x7fcfec104810 2024-03-04T07:37:42.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.882+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfec02cff0 con 0x7fcfec104810 2024-03-04T07:37:42.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.883+0000 7fcfea000640 1 -- 172.21.15.116:0/3094893804 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8459 (secure 0 0 0) 0x7fcfd80bd1c0 con 0x7fcfec104810 2024-03-04T07:37:42.884 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:42.884 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-04T07:37:42.884 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 11 2024-03-04T07:37:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfd40a1d80 msgr2=0x7fcfd40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfd40a1d80 0x7fcfd40a4240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcfec104670 tx=0x7fcfe0005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 msgr2=0x7fcfec199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:42.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec199e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8000c00 tx=0x7fcfd802c650 comp rx=0 tx=0).stop 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 shutdown_connections 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfd40a1d80 0x7fcfd40a4240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfec1051d0 0x7fcfec1a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfec104810 0x7fcfec199e30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 --2- 172.21.15.116:0/3094893804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfec103620 0x7fcfec1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 >> 172.21.15.116:0/3094893804 conn(0x7fcfec0febb0 msgr2=0x7fcfec107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.886+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 shutdown_connections 2024-03-04T07:37:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:42.887+0000 7fcff3000640 1 -- 172.21.15.116:0/3094893804 wait complete. 2024-03-04T07:37:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:42 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3094893804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:42 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3094893804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:43.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:42 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3094893804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:44.065 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:43 smithi116 ceph-mon[32150]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:43 smithi152 ceph-mon[37154]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:44.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:43 smithi133 ceph-mon[36505]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:44.416 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:44 smithi116 systemd[1]: Starting Ceph mds.a for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:44.758 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:44 smithi116 podman[71556]: 2024-03-04 07:37:44.319219724 +0000 UTC m=+0.033704432 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:44.758 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:44 smithi116 podman[71556]: 2024-03-04 07:37:44.576688916 +0000 UTC m=+0.291173651 container create e387a3fde33126a95b50b72e22860bacb4107e97fe03f5522ff38a14a6b895c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-04T07:37:44.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 -- 172.21.15.116:0/2599940087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 msgr2=0x7f4ffc102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 --2- 172.21.15.116:0/2599940087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 0x7f4ffc102c60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009860 tx=0x7f4fe802bb70 comp rx=0 tx=0).stop 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 -- 172.21.15.116:0/2599940087 shutdown_connections 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 --2- 172.21.15.116:0/2599940087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc1031a0 0x7f4ffc109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 --2- 172.21.15.116:0/2599940087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 0x7f4ffc102c60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 --2- 172.21.15.116:0/2599940087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 0x7f4ffc101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 -- 172.21.15.116:0/2599940087 >> 172.21.15.116:0/2599940087 conn(0x7f4ffc0fcb80 msgr2=0x7f4ffc0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 -- 172.21.15.116:0/2599940087 shutdown_connections 2024-03-04T07:37:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.103+0000 7f5003600640 1 -- 172.21.15.116:0/2599940087 wait complete. 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.104+0000 7f5003600640 1 Processor -- start 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.104+0000 7f5003600640 1 -- start start 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5003600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 0x7f4ffc075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5003600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 0x7f4ffc0739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5003600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc1031a0 0x7f4ffc073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5003600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc10d250 con 0x7f4ffc1031a0 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5003600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc10d0d0 con 0x7f4ffc1015f0 2024-03-04T07:37:45.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5003600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc10d3d0 con 0x7f4ffc1027e0 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5001600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc1031a0 0x7f4ffc073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 0x7f4ffc075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 0x7f4ffc075380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43900/0 (socket says 172.21.15.116:43900) 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 -- 172.21.15.116:0/1222700957 learned_addr learned my addr 172.21.15.116:0/1222700957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f4ffbe00640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 0x7f4ffc0739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 -- 172.21.15.116:0/1222700957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 msgr2=0x7f4ffc0739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 0x7f4ffc0739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 -- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc1031a0 msgr2=0x7f4ffc073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc1031a0 0x7f4ffc073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5000c00640 1 -- 172.21.15.116:0/1222700957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc074790 con 0x7f4ffc1015f0 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.105+0000 7f5001600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc1031a0 0x7f4ffc073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.106+0000 7f5000c00640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 0x7f4ffc075380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000e9c0 tx=0x7f4ff000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.106+0000 7f4ff9600640 1 -- 172.21.15.116:0/1222700957 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff000cd80 con 0x7f4ffc1015f0 2024-03-04T07:37:45.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.106+0000 7f4ff9600640 1 -- 172.21.15.116:0/1222700957 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ff00035f0 con 0x7f4ffc1015f0 2024-03-04T07:37:45.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.106+0000 7f4ff9600640 1 -- 172.21.15.116:0/1222700957 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0010690 con 0x7f4ffc1015f0 2024-03-04T07:37:45.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.106+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc1a19e0 con 0x7f4ffc1015f0 2024-03-04T07:37:45.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.106+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc1a1eb0 con 0x7f4ffc1015f0 2024-03-04T07:37:45.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.108+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc068810 con 0x7f4ffc1015f0 2024-03-04T07:37:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.109+0000 7f4ff9600640 1 -- 172.21.15.116:0/1222700957 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4ff00037a0 con 0x7f4ffc1015f0 2024-03-04T07:37:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.110+0000 7f4ff9600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4fe40a1d80 0x7f4fe40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.111+0000 7f4ff9600640 1 -- 172.21.15.116:0/1222700957 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4ff00cde40 con 0x7f4ffc1015f0 2024-03-04T07:37:45.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.111+0000 7f4ffbe00640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4fe40a1d80 0x7f4fe40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:45.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.112+0000 7f4ffbe00640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4fe40a1d80 0x7f4fe40a4240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009830 tx=0x7f4fe8034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:45.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.114+0000 7f4ff9600640 1 -- 172.21.15.116:0/1222700957 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4ff009b670 con 0x7f4ffc1015f0 2024-03-04T07:37:45.258 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:44 smithi116 podman[71556]: 2024-03-04 07:37:44.888857044 +0000 UTC m=+0.603341822 container init e387a3fde33126a95b50b72e22860bacb4107e97fe03f5522ff38a14a6b895c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-04T07:37:45.259 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:44 smithi116 podman[71556]: 2024-03-04 07:37:44.901199464 +0000 UTC m=+0.615684186 container start e387a3fde33126a95b50b72e22860bacb4107e97fe03f5522ff38a14a6b895c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:37:45.259 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:44 smithi116 ceph-mds[71575]: -- 172.21.15.116:0/2489076776 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561977e95a40 con 0x561977e68800 2024-03-04T07:37:45.259 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:44 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a[71571]: starting mds.a at 2024-03-04T07:37:45.259 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:45 smithi116 bash[71556]: e387a3fde33126a95b50b72e22860bacb4107e97fe03f5522ff38a14a6b895c5 2024-03-04T07:37:45.259 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:37:45 smithi116 systemd[1]: Started Ceph mds.a for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.356+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ffc108910 con 0x7f4ffc1015f0 2024-03-04T07:37:45.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.357+0000 7f4ff9600640 1 -- 172.21.15.116:0/1222700957 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9248 (secure 0 0 0) 0x7f4ff009af70 con 0x7f4ffc1015f0 2024-03-04T07:37:45.357 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:45.357 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-04T07:37:45.357 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 12 2024-03-04T07:37:45.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4fe40a1d80 msgr2=0x7f4fe40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4fe40a1d80 0x7f4fe40a4240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009830 tx=0x7f4fe8034ab0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 msgr2=0x7f4ffc075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 0x7f4ffc075380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000e9c0 tx=0x7f4ff000ee80 comp rx=0 tx=0).stop 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 shutdown_connections 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc1031a0 0x7f4ffc073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4fe40a1d80 0x7f4fe40a4240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1027e0 0x7f4ffc0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 --2- 172.21.15.116:0/1222700957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ffc1015f0 0x7f4ffc075380 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 >> 172.21.15.116:0/1222700957 conn(0x7f4ffc0fcb80 msgr2=0x7f4ffc105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 shutdown_connections 2024-03-04T07:37:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:45.360+0000 7f5003600640 1 -- 172.21.15.116:0/1222700957 wait complete. 2024-03-04T07:37:46.174 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.116:6840/18076009,v1:172.21.15.116:6841/18076009] up:boot 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: fsmap 11 up:standby 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: Deploying daemon mds.b on smithi133 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1222700957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:46.175 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:45 smithi133 ceph-mon[36505]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.116:6840/18076009,v1:172.21.15.116:6841/18076009] up:boot 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: fsmap 11 up:standby 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:46.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: Deploying daemon mds.b on smithi133 2024-03-04T07:37:46.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1222700957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:46.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:45 smithi116 ceph-mon[32150]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.116:6840/18076009,v1:172.21.15.116:6841/18076009] up:boot 2024-03-04T07:37:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: fsmap 11 up:standby 2024-03-04T07:37:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T07:37:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:46.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T07:37:46.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T07:37:46.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:46.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: Deploying daemon mds.b on smithi133 2024-03-04T07:37:46.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1222700957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:46.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:45 smithi152 ceph-mon[37154]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:47.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:47.491 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 systemd[1]: Starting Ceph mds.b for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:37:47.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 -- 172.21.15.116:0/2303164778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6258108c20 msgr2=0x7f62581090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 --2- 172.21.15.116:0/2303164778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6258108c20 0x7f62581090a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6244009900 tx=0x7f624402bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 -- 172.21.15.116:0/2303164778 shutdown_connections 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 --2- 172.21.15.116:0/2303164778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62581095e0 0x7f625810fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 --2- 172.21.15.116:0/2303164778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6258108c20 0x7f62581090a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 --2- 172.21.15.116:0/2303164778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 0x7f6258107e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 -- 172.21.15.116:0/2303164778 >> 172.21.15.116:0/2303164778 conn(0x7f62580769e0 msgr2=0x7f6258078e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 -- 172.21.15.116:0/2303164778 shutdown_connections 2024-03-04T07:37:47.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.603+0000 7f625f000640 1 -- 172.21.15.116:0/2303164778 wait complete. 2024-03-04T07:37:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.604+0000 7f625f000640 1 Processor -- start 2024-03-04T07:37:47.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.604+0000 7f625f000640 1 -- start start 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625f000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 0x7f625819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625f000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6258108c20 0x7f625819e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625f000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62581095e0 0x7f62581a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625f000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62581136b0 con 0x7f62581095e0 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625f000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6258113530 con 0x7f6258108c20 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625f000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6258113830 con 0x7f6258107a30 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625dc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 0x7f625819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625dc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 0x7f625819dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:38860/0 (socket says 172.21.15.116:38860) 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625dc00640 1 -- 172.21.15.116:0/1273219339 learned_addr learned my addr 172.21.15.116:0/1273219339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:47.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625d200640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6258108c20 0x7f625819e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.605+0000 7f625e600640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62581095e0 0x7f62581a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625dc00640 1 -- 172.21.15.116:0/1273219339 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6258108c20 msgr2=0x7f625819e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625dc00640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6258108c20 0x7f625819e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625dc00640 1 -- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62581095e0 msgr2=0x7f62581a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625dc00640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62581095e0 0x7f62581a5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625dc00640 1 -- 172.21.15.116:0/1273219339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62581a5970 con 0x7f6258107a30 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625dc00640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 0x7f625819dd50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f624c00e9c0 tx=0x7f624c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f6256000640 1 -- 172.21.15.116:0/1273219339 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c00ccc0 con 0x7f6258107a30 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f6256000640 1 -- 172.21.15.116:0/1273219339 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f624c0035f0 con 0x7f6258107a30 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62581a5c60 con 0x7f6258107a30 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f6256000640 1 -- 172.21.15.116:0/1273219339 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c00ce60 con 0x7f6258107a30 2024-03-04T07:37:47.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.606+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62581a6130 con 0x7f6258107a30 2024-03-04T07:37:47.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.608+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625810ed50 con 0x7f6258107a30 2024-03-04T07:37:47.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.609+0000 7f6256000640 1 -- 172.21.15.116:0/1273219339 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f624c0037a0 con 0x7f6258107a30 2024-03-04T07:37:47.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.610+0000 7f6256000640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62400a1d80 0x7f62400a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:47.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.611+0000 7f6256000640 1 -- 172.21.15.116:0/1273219339 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f624c0cdd10 con 0x7f6258107a30 2024-03-04T07:37:47.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.611+0000 7f625d200640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62400a1d80 0x7f62400a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:47.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.611+0000 7f625d200640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62400a1d80 0x7f62400a4240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f624402c7b0 tx=0x7f6244005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.615+0000 7f6256000640 1 -- 172.21.15.116:0/1273219339 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f624c0d3020 con 0x7f6258107a30 2024-03-04T07:37:47.750 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 podman[67074]: 2024-03-04 07:37:47.394029233 +0000 UTC m=+0.018348860 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:37:47.750 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 podman[67074]: 2024-03-04 07:37:47.52516261 +0000 UTC m=+0.149482246 container create 104f6c82bd6c3afd1904912c8eb96f04db43be68ad382116bd338a8c80c00c0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:37:47.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.855+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f625802cff0 con 0x7f6258107a30 2024-03-04T07:37:47.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.857+0000 7f6256000640 1 -- 172.21.15.116:0/1273219339 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9248 (secure 0 0 0) 0x7f624c09b540 con 0x7f6258107a30 2024-03-04T07:37:47.857 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:47.857 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-04T07:37:47.857 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 12 2024-03-04T07:37:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.859+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62400a1d80 msgr2=0x7f62400a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:47.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.859+0000 7f625f000640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62400a1d80 0x7f62400a4240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f624402c7b0 tx=0x7f6244005fd0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.859+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 msgr2=0x7f625819dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.859+0000 7f625f000640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 0x7f625819dd50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f624c00e9c0 tx=0x7f624c00edc0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 shutdown_connections 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62581095e0 0x7f62581a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6258108c20 0x7f625819e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62400a1d80 0x7f62400a4240 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 --2- 172.21.15.116:0/1273219339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6258107a30 0x7f625819dd50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 >> 172.21.15.116:0/1273219339 conn(0x7f62580769e0 msgr2=0x7f625810bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 shutdown_connections 2024-03-04T07:37:47.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:47.860+0000 7f625f000640 1 -- 172.21.15.116:0/1273219339 wait complete. 2024-03-04T07:37:48.087 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 podman[67074]: 2024-03-04 07:37:47.75120381 +0000 UTC m=+0.375523427 container init 104f6c82bd6c3afd1904912c8eb96f04db43be68ad382116bd338a8c80c00c0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-03-04T07:37:48.087 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 podman[67074]: 2024-03-04 07:37:47.753872284 +0000 UTC m=+0.378191901 container start 104f6c82bd6c3afd1904912c8eb96f04db43be68ad382116bd338a8c80c00c0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-04T07:37:48.087 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 ceph-mds[67093]: -- 172.21.15.133:0/192641617 <== mon.2 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5609d4331a40 con 0x5609d4304c00 2024-03-04T07:37:48.087 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b[67089]: starting mds.b at 2024-03-04T07:37:48.087 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 bash[67074]: 104f6c82bd6c3afd1904912c8eb96f04db43be68ad382116bd338a8c80c00c0a 2024-03-04T07:37:48.087 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:47 smithi133 systemd[1]: Started Ceph mds.b for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:37:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1273219339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.772 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1273219339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:48.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1273219339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:49.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:49 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] up:boot 2024-03-04T07:37:49.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:49 smithi133 ceph-mon[36505]: fsmap 12 up:standby 2024-03-04T07:37:49.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:49 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:37:49.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:49 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:37:49.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:37:49.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:49 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] up:boot 2024-03-04T07:37:49.967 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:49 smithi116 ceph-mon[32150]: fsmap 12 up:standby 2024-03-04T07:37:49.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:49 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:37:49.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:49 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:37:50.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:49 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] up:boot 2024-03-04T07:37:50.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:49 smithi152 ceph-mon[37154]: fsmap 12 up:standby 2024-03-04T07:37:50.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:49 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:37:50.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:49 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:37:50.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.102+0000 7fb886200640 1 -- 172.21.15.116:0/1171701922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 msgr2=0x7fb880102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.102+0000 7fb886200640 1 --2- 172.21.15.116:0/1171701922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880102ab0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009900 tx=0x7fb86c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.103+0000 7fb886200640 1 -- 172.21.15.116:0/1171701922 shutdown_connections 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.103+0000 7fb886200640 1 --2- 172.21.15.116:0/1171701922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880102ff0 0x7fb880109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.103+0000 7fb886200640 1 --2- 172.21.15.116:0/1171701922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880102ab0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.103+0000 7fb886200640 1 --2- 172.21.15.116:0/1171701922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb880101440 0x7fb880101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.103+0000 7fb886200640 1 -- 172.21.15.116:0/1171701922 >> 172.21.15.116:0/1171701922 conn(0x7fb8800fc9b0 msgr2=0x7fb8800fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.103+0000 7fb886200640 1 -- 172.21.15.116:0/1171701922 shutdown_connections 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.103+0000 7fb886200640 1 -- 172.21.15.116:0/1171701922 wait complete. 2024-03-04T07:37:50.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 Processor -- start 2024-03-04T07:37:50.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 -- start start 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880101440 0x7fb880199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb880102ff0 0x7fb8801a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88010d0d0 con 0x7fb880101440 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88010cf50 con 0x7fb880102630 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.104+0000 7fb886200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88010d250 con 0x7fb880102ff0 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb884e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880101440 0x7fb880199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59820/0 (socket says 172.21.15.116:59820) 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 -- 172.21.15.116:0/290672725 learned_addr learned my addr 172.21.15.116:0/290672725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb884e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880101440 0x7fb880199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57418/0 (socket says 172.21.15.116:57418) 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb885800640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb880102ff0 0x7fb8801a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 -- 172.21.15.116:0/290672725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb880102ff0 msgr2=0x7fb8801a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb880102ff0 0x7fb8801a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 -- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880101440 msgr2=0x7fb880199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880101440 0x7fb880199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb87fe00640 1 -- 172.21.15.116:0/290672725 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8801a14f0 con 0x7fb880102630 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.105+0000 7fb884e00640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880101440 0x7fb880199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.106+0000 7fb87fe00640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880199eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009230 tx=0x7fb86c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.106+0000 7fb87d600640 1 -- 172.21.15.116:0/290672725 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c041070 con 0x7fb880102630 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.106+0000 7fb87d600640 1 -- 172.21.15.116:0/290672725 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb86c004450 con 0x7fb880102630 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.106+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8801a1780 con 0x7fb880102630 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.106+0000 7fb87d600640 1 -- 172.21.15.116:0/290672725 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c003e20 con 0x7fb880102630 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.106+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8801a1cc0 con 0x7fb880102630 2024-03-04T07:37:50.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.107+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8801a1f20 con 0x7fb880102630 2024-03-04T07:37:50.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.109+0000 7fb87d600640 1 -- 172.21.15.116:0/290672725 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb86c044070 con 0x7fb880102630 2024-03-04T07:37:50.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.111+0000 7fb87d600640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8680a1e00 0x7fb8680a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.111+0000 7fb87d600640 1 -- 172.21.15.116:0/290672725 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb86c0ef4e0 con 0x7fb880102630 2024-03-04T07:37:50.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.112+0000 7fb884e00640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8680a1e00 0x7fb8680a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.112+0000 7fb884e00640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8680a1e00 0x7fb8680a42c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb880102490 tx=0x7fb874005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:50.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.114+0000 7fb87d600640 1 -- 172.21.15.116:0/290672725 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb86c0bc6c0 con 0x7fb880102630 2024-03-04T07:37:50.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.398+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8801a20f0 con 0x7fb880102630 2024-03-04T07:37:50.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.399+0000 7fb87d600640 1 -- 172.21.15.116:0/290672725 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10043 (secure 0 0 0) 0x7fb86c0c0760 con 0x7fb880102630 2024-03-04T07:37:50.399 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:50.399 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-04T07:37:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 13 2024-03-04T07:37:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8680a1e00 msgr2=0x7fb8680a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8680a1e00 0x7fb8680a42c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb880102490 tx=0x7fb874005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 msgr2=0x7fb880199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880199eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009230 tx=0x7fb86c02c650 comp rx=0 tx=0).stop 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 shutdown_connections 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb8680a1e00 0x7fb8680a42c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb880102ff0 0x7fb8801a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb880102630 0x7fb880199eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 --2- 172.21.15.116:0/290672725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb880101440 0x7fb880199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 >> 172.21.15.116:0/290672725 conn(0x7fb8800fc9b0 msgr2=0x7fb880073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 shutdown_connections 2024-03-04T07:37:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.402+0000 7fb886200640 1 -- 172.21.15.116:0/290672725 wait complete. 2024-03-04T07:37:50.421 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:37:50.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 -- 172.21.15.116:0/888742645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440101430 msgr2=0x7f4440101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 --2- 172.21.15.116:0/888742645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440101430 0x7f4440101850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4430009860 tx=0x7f443002bb70 comp rx=0 tx=0).stop 2024-03-04T07:37:50.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 -- 172.21.15.116:0/888742645 shutdown_connections 2024-03-04T07:37:50.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 --2- 172.21.15.116:0/888742645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440102fe0 0x7f4440109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 --2- 172.21.15.116:0/888742645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4440102620 0x7f4440102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 --2- 172.21.15.116:0/888742645 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440101430 0x7f4440101850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 -- 172.21.15.116:0/888742645 >> 172.21.15.116:0/888742645 conn(0x7f44400fc9a0 msgr2=0x7f44400fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:50.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 -- 172.21.15.116:0/888742645 shutdown_connections 2024-03-04T07:37:50.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.620+0000 7f443fe00640 1 -- 172.21.15.116:0/888742645 wait complete. 2024-03-04T07:37:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.621+0000 7f443fe00640 1 Processor -- start 2024-03-04T07:37:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.621+0000 7f443fe00640 1 -- start start 2024-03-04T07:37:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440101430 0x7f4440199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440102620 0x7f4440199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4440102fe0 0x7f44401a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444010d0c0 con 0x7f4440101430 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443fe00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444010cf40 con 0x7f4440102620 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443fe00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444010d240 con 0x7f4440102fe0 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440101430 0x7f4440199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440101430 0x7f4440199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57426/0 (socket says 172.21.15.116:57426) 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 -- 172.21.15.116:0/2340884104 learned_addr learned my addr 172.21.15.116:0/2340884104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443e000640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440102620 0x7f4440199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443f400640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4440102fe0 0x7f44401a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 -- 172.21.15.116:0/2340884104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4440102fe0 msgr2=0x7f44401a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4440102fe0 0x7f44401a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 -- 172.21.15.116:0/2340884104 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440102620 msgr2=0x7f4440199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440102620 0x7f4440199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.622+0000 7f443ea00640 1 -- 172.21.15.116:0/2340884104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44401a1570 con 0x7f4440101430 2024-03-04T07:37:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.623+0000 7f443ea00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440101430 0x7f4440199950 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44300349e0 tx=0x7f4430032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.623+0000 7f4437400640 1 -- 172.21.15.116:0/2340884104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430041020 con 0x7f4440101430 2024-03-04T07:37:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.623+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44401a1800 con 0x7f4440101430 2024-03-04T07:37:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.623+0000 7f4437400640 1 -- 172.21.15.116:0/2340884104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f443002c9f0 con 0x7f4440101430 2024-03-04T07:37:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.623+0000 7f4437400640 1 -- 172.21.15.116:0/2340884104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430039040 con 0x7f4440101430 2024-03-04T07:37:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.623+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44401a1d40 con 0x7f4440101430 2024-03-04T07:37:50.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.625+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44401a2000 con 0x7f4440101430 2024-03-04T07:37:50.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.626+0000 7f4437400640 1 -- 172.21.15.116:0/2340884104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4430044020 con 0x7f4440101430 2024-03-04T07:37:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.627+0000 7f4437400640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44240a1d80 0x7f44240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:50.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.628+0000 7f4437400640 1 -- 172.21.15.116:0/2340884104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f44300ef1b0 con 0x7f4440101430 2024-03-04T07:37:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.628+0000 7f443e000640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44240a1d80 0x7f44240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.628+0000 7f443e000640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44240a1d80 0x7f44240a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f444019ae60 tx=0x7f442c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.632+0000 7f4437400640 1 -- 172.21.15.116:0/2340884104 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f443004b020 con 0x7f4440101430 2024-03-04T07:37:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:50 smithi116 ceph-mon[32150]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:50 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/290672725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:50 smithi133 ceph-mon[36505]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:50 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/290672725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.884+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44401a21d0 con 0x7f4440101430 2024-03-04T07:37:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.886+0000 7f4437400640 1 -- 172.21.15.116:0/2340884104 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18539 (secure 0 0 0) 0x7f44300bc9e0 con 0x7f4440101430 2024-03-04T07:37:50.886 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:50.886 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:36:58.362248+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.888+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44240a1d80 msgr2=0x7f44240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.888+0000 7f443fe00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44240a1d80 0x7f44240a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f444019ae60 tx=0x7f442c005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.888+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440101430 msgr2=0x7f4440199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.888+0000 7f443fe00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440101430 0x7f4440199950 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44300349e0 tx=0x7f4430032430 comp rx=0 tx=0).stop 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 shutdown_connections 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44240a1d80 0x7f44240a4240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4440102fe0 0x7f44401a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4440102620 0x7f4440199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 --2- 172.21.15.116:0/2340884104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4440101430 0x7f4440199950 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 >> 172.21.15.116:0/2340884104 conn(0x7f44400fc9a0 msgr2=0x7f4440073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 shutdown_connections 2024-03-04T07:37:50.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:50.889+0000 7f443fe00640 1 -- 172.21.15.116:0/2340884104 wait complete. 2024-03-04T07:37:50.907 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:37:50.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:37:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:50 smithi152 ceph-mon[37154]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:50 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/290672725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:37:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 -- 172.21.15.116:0/3533157340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 msgr2=0x7ff51c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 --2- 172.21.15.116:0/3533157340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 0x7ff51c101a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009900 tx=0x7ff50c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 -- 172.21.15.116:0/3533157340 shutdown_connections 2024-03-04T07:37:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 --2- 172.21.15.116:0/3533157340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff51c103190 0x7ff51c109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 --2- 172.21.15.116:0/3533157340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff51c1027d0 0x7ff51c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 --2- 172.21.15.116:0/3533157340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 0x7ff51c101a00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 -- 172.21.15.116:0/3533157340 >> 172.21.15.116:0/3533157340 conn(0x7ff51c0fcb70 msgr2=0x7ff51c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 -- 172.21.15.116:0/3533157340 shutdown_connections 2024-03-04T07:37:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.099+0000 7ff523200640 1 -- 172.21.15.116:0/3533157340 wait complete. 2024-03-04T07:37:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 Processor -- start 2024-03-04T07:37:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 -- start start 2024-03-04T07:37:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 0x7ff51c197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff51c1027d0 0x7ff51c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff51c103190 0x7ff51c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c10d230 con 0x7ff51c1015e0 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c10d0b0 con 0x7ff51c1027d0 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff523200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c10d3b0 con 0x7ff51c103190 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.101+0000 7ff51be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 0x7ff51c197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 0x7ff51c197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57444/0 (socket says 172.21.15.116:57444) 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51be00640 1 -- 172.21.15.116:0/358117400 learned_addr learned my addr 172.21.15.116:0/358117400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51b400640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff51c1027d0 0x7ff51c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff521200640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff51c103190 0x7ff51c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51b400640 1 -- 172.21.15.116:0/358117400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff51c103190 msgr2=0x7ff51c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51b400640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff51c103190 0x7ff51c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51b400640 1 -- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 msgr2=0x7ff51c197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51b400640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 0x7ff51c197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51b400640 1 -- 172.21.15.116:0/358117400 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c19f580 con 0x7ff51c1027d0 2024-03-04T07:37:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff51b400640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff51c1027d0 0x7ff51c197ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff50800e9f0 tx=0x7ff50800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.102+0000 7ff518c00640 1 -- 172.21.15.116:0/358117400 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800ccc0 con 0x7ff51c1027d0 2024-03-04T07:37:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.103+0000 7ff518c00640 1 -- 172.21.15.116:0/358117400 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5080035f0 con 0x7ff51c1027d0 2024-03-04T07:37:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.103+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c19f870 con 0x7ff51c1027d0 2024-03-04T07:37:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.103+0000 7ff518c00640 1 -- 172.21.15.116:0/358117400 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800ce60 con 0x7ff51c1027d0 2024-03-04T07:37:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.103+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c19fe10 con 0x7ff51c1027d0 2024-03-04T07:37:51.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.104+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c108970 con 0x7ff51c1027d0 2024-03-04T07:37:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.107+0000 7ff518c00640 1 -- 172.21.15.116:0/358117400 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff5080037a0 con 0x7ff51c1027d0 2024-03-04T07:37:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.108+0000 7ff518c00640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5000a1e00 0x7ff5000a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.108+0000 7ff518c00640 1 -- 172.21.15.116:0/358117400 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff5080cdb50 con 0x7ff51c1027d0 2024-03-04T07:37:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.108+0000 7ff51be00640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5000a1e00 0x7ff5000a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.109+0000 7ff51be00640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5000a1e00 0x7ff5000a42c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff50c02c8e0 tx=0x7ff50c005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:51.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.112+0000 7ff518c00640 1 -- 172.21.15.116:0/358117400 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff50809b380 con 0x7ff51c1027d0 2024-03-04T07:37:51.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.318+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff51c0621d0 con 0x7ff51c1027d0 2024-03-04T07:37:51.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.319+0000 7ff518c00640 1 -- 172.21.15.116:0/358117400 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff508009440 con 0x7ff51c1027d0 2024-03-04T07:37:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:37:51.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5000a1e00 msgr2=0x7ff5000a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5000a1e00 0x7ff5000a42c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff50c02c8e0 tx=0x7ff50c005b50 comp rx=0 tx=0).stop 2024-03-04T07:37:51.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff51c1027d0 msgr2=0x7ff51c197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff51c1027d0 0x7ff51c197ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff50800e9f0 tx=0x7ff50800edf0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 shutdown_connections 2024-03-04T07:37:51.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5000a1e00 0x7ff5000a42c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff51c103190 0x7ff51c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff51c1027d0 0x7ff51c197ea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 --2- 172.21.15.116:0/358117400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff51c1015e0 0x7ff51c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.322+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 >> 172.21.15.116:0/358117400 conn(0x7ff51c0fcb70 msgr2=0x7ff51c105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:51.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.323+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 shutdown_connections 2024-03-04T07:37:51.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.323+0000 7ff523200640 1 -- 172.21.15.116:0/358117400 wait complete. 2024-03-04T07:37:51.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T07:37:51.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 -- 172.21.15.116:0/1472124471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 msgr2=0x7f8390100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 --2- 172.21.15.116:0/1472124471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f8390100c20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8378009900 tx=0x7f837802bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 -- 172.21.15.116:0/1472124471 shutdown_connections 2024-03-04T07:37:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 --2- 172.21.15.116:0/1472124471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8390101160 0x7f83901015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 --2- 172.21.15.116:0/1472124471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f8390100c20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 --2- 172.21.15.116:0/1472124471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83900ffe60 0x7f8390100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 -- 172.21.15.116:0/1472124471 >> 172.21.15.116:0/1472124471 conn(0x7f83900fba80 msgr2=0x7f83900fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 -- 172.21.15.116:0/1472124471 shutdown_connections 2024-03-04T07:37:51.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.533+0000 7f8395200640 1 -- 172.21.15.116:0/1472124471 wait complete. 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.534+0000 7f8395200640 1 Processor -- start 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.534+0000 7f8395200640 1 -- start start 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8395200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83900ffe60 0x7f8390199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8395200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f839019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8395200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101160 0x7f83901a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8395200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839010f480 con 0x7f8390101160 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8395200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839010f300 con 0x7f83900ffe60 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8395200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839010f600 con 0x7f83901007c0 2024-03-04T07:37:51.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f839019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f839019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:38916/0 (socket says 172.21.15.116:38916) 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f838de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83900ffe60 0x7f8390199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 -- 172.21.15.116:0/3623912839 learned_addr learned my addr 172.21.15.116:0/3623912839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f838e800640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101160 0x7f83901a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 -- 172.21.15.116:0/3623912839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83900ffe60 msgr2=0x7f8390199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83900ffe60 0x7f8390199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 -- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101160 msgr2=0x7f83901a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101160 0x7f83901a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f8387e00640 1 -- 172.21.15.116:0/3623912839 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83901a16e0 con 0x7f83901007c0 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.535+0000 7f838de00640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83900ffe60 0x7f8390199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:51.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.536+0000 7f8387e00640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f839019a000 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f837802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.536+0000 7f8386a00640 1 -- 172.21.15.116:0/3623912839 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378043020 con 0x7f83901007c0 2024-03-04T07:37:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.536+0000 7f8386a00640 1 -- 172.21.15.116:0/3623912839 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83780054e0 con 0x7f83901007c0 2024-03-04T07:37:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.536+0000 7f8386a00640 1 -- 172.21.15.116:0/3623912839 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378003c40 con 0x7f83901007c0 2024-03-04T07:37:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.536+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83901a1970 con 0x7f83901007c0 2024-03-04T07:37:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.536+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83901a1f10 con 0x7f83901007c0 2024-03-04T07:37:51.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.538+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839010ad40 con 0x7f83901007c0 2024-03-04T07:37:51.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.539+0000 7f8386a00640 1 -- 172.21.15.116:0/3623912839 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8378046020 con 0x7f83901007c0 2024-03-04T07:37:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.541+0000 7f8386a00640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83740a1e00 0x7f83740a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.541+0000 7f8386a00640 1 -- 172.21.15.116:0/3623912839 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f83780ef8c0 con 0x7f83901007c0 2024-03-04T07:37:51.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.541+0000 7f838de00640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83740a1e00 0x7f83740a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.542+0000 7f838de00640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83740a1e00 0x7f83740a42c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8380009fc0 tx=0x7f8380005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:51.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.545+0000 7f8386a00640 1 -- 172.21.15.116:0/3623912839 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f837804d020 con 0x7f83901007c0 2024-03-04T07:37:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.743+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f839002d240 con 0x7f83901007c0 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.744+0000 7f8386a00640 1 -- 172.21.15.116:0/3623912839 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f83780bd190 con 0x7f83901007c0 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:37:51.744 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-03-04T07:37:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83740a1e00 msgr2=0x7f83740a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83740a1e00 0x7f83740a42c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8380009fc0 tx=0x7f8380005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 msgr2=0x7f839019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f839019a000 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f837802c650 comp rx=0 tx=0).stop 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 shutdown_connections 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8390101160 0x7f83901a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83740a1e00 0x7f83740a42c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83901007c0 0x7f839019a000 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 --2- 172.21.15.116:0/3623912839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83900ffe60 0x7f8390199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 >> 172.21.15.116:0/3623912839 conn(0x7f83900fba80 msgr2=0x7f83900fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 shutdown_connections 2024-03-04T07:37:51.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.747+0000 7f8395200640 1 -- 172.21.15.116:0/3623912839 wait complete. 2024-03-04T07:37:51.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2340884104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:51.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/358117400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:51.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:51.765 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T07:37:51.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T07:37:51.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2340884104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:51.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:51.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:51.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/358117400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:51.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:51.883 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2340884104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:51.883 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.883 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.883 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:51.884 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:51.884 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:51.884 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/358117400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:37:51.884 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 -- 172.21.15.116:0/834230113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41015e0 msgr2=0x7f3cd4101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 --2- 172.21.15.116:0/834230113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41015e0 0x7f3cd4101a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8009900 tx=0x7f3cc802bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 -- 172.21.15.116:0/834230113 shutdown_connections 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 --2- 172.21.15.116:0/834230113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3cd4103190 0x7f3cd4109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 --2- 172.21.15.116:0/834230113 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3cd41027d0 0x7f3cd4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 --2- 172.21.15.116:0/834230113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41015e0 0x7f3cd4101a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 -- 172.21.15.116:0/834230113 >> 172.21.15.116:0/834230113 conn(0x7f3cd40fcb50 msgr2=0x7f3cd40fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 -- 172.21.15.116:0/834230113 shutdown_connections 2024-03-04T07:37:51.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.973+0000 7f3cdc200640 1 -- 172.21.15.116:0/834230113 wait complete. 2024-03-04T07:37:51.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 Processor -- start 2024-03-04T07:37:51.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 -- start start 2024-03-04T07:37:51.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3cd41015e0 0x7f3cd4197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41027d0 0x7f3cd4197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3cd4103190 0x7f3cd419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd410d2a0 con 0x7f3cd41027d0 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd410d120 con 0x7f3cd4103190 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cdc200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd410d420 con 0x7f3cd41015e0 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cd8e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41027d0 0x7f3cd4197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cd8e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41027d0 0x7f3cd4197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57482/0 (socket says 172.21.15.116:57482) 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cd8e00640 1 -- 172.21.15.116:0/1684769750 learned_addr learned my addr 172.21.15.116:0/1684769750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cd9800640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3cd41015e0 0x7f3cd4197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.974+0000 7f3cda200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3cd4103190 0x7f3cd419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd8e00640 1 -- 172.21.15.116:0/1684769750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3cd41015e0 msgr2=0x7f3cd4197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd8e00640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3cd41015e0 0x7f3cd4197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd8e00640 1 -- 172.21.15.116:0/1684769750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3cd4103190 msgr2=0x7f3cd419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd8e00640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3cd4103190 0x7f3cd419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:51.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd8e00640 1 -- 172.21.15.116:0/1684769750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd419f560 con 0x7f3cd41027d0 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cda200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3cd4103190 0x7f3cd419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd8e00640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41027d0 0x7f3cd4197e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000d950 tx=0x7f3cc000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd2000640 1 -- 172.21.15.116:0/1684769750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0017070 con 0x7f3cd41027d0 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.975+0000 7f3cd2000640 1 -- 172.21.15.116:0/1684769750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cc0009c30 con 0x7f3cd41027d0 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.976+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd419f850 con 0x7f3cd41027d0 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.976+0000 7f3cd2000640 1 -- 172.21.15.116:0/1684769750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000f6b0 con 0x7f3cd41027d0 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.976+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd419fda0 con 0x7f3cd41027d0 2024-03-04T07:37:51.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.977+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd4108980 con 0x7f3cd41027d0 2024-03-04T07:37:51.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.980+0000 7f3cd2000640 1 -- 172.21.15.116:0/1684769750 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3cc00109d0 con 0x7f3cd41027d0 2024-03-04T07:37:51.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.981+0000 7f3cd2000640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3cbc0a1d80 0x7f3cbc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:51.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.981+0000 7f3cd2000640 1 -- 172.21.15.116:0/1684769750 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3cc00cd520 con 0x7f3cd41027d0 2024-03-04T07:37:51.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.981+0000 7f3cd9800640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3cbc0a1d80 0x7f3cbc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:51.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.982+0000 7f3cd9800640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3cbc0a1d80 0x7f3cbc0a4240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3cc802c170 tx=0x7f3cc802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:51.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:51.986+0000 7f3cd2000640 1 -- 172.21.15.116:0/1684769750 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3cc0009de0 con 0x7f3cd41027d0 2024-03-04T07:37:52.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:52.200+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3cd402d210 con 0x7f3cd41027d0 2024-03-04T07:37:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:52 smithi116 ceph-mon[32150]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:53.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:52 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3623912839' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-04T07:37:53.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:52 smithi152 ceph-mon[37154]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:53.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:52 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3623912839' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-04T07:37:53.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:52 smithi133 ceph-mon[36505]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:53.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:52 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3623912839' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-04T07:37:53.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.091+0000 7f3cd2000640 1 -- 172.21.15.116:0/1684769750 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3cc009a780 con 0x7f3cd41027d0 2024-03-04T07:37:53.091 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2024-03-04T07:37:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.094+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3cbc0a1d80 msgr2=0x7f3cbc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.094+0000 7f3cdc200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3cbc0a1d80 0x7f3cbc0a4240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3cc802c170 tx=0x7f3cc802df90 comp rx=0 tx=0).stop 2024-03-04T07:37:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.094+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41027d0 msgr2=0x7f3cd4197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.094+0000 7f3cdc200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41027d0 0x7f3cd4197e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000d950 tx=0x7f3cc000de10 comp rx=0 tx=0).stop 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 shutdown_connections 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3cd4103190 0x7f3cd419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cd41027d0 0x7f3cd4197e80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3cbc0a1d80 0x7f3cbc0a4240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 --2- 172.21.15.116:0/1684769750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3cd41015e0 0x7f3cd4197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 >> 172.21.15.116:0/1684769750 conn(0x7f3cd40fcb50 msgr2=0x7f3cd4105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 shutdown_connections 2024-03-04T07:37:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.095+0000 7f3cdc200640 1 -- 172.21.15.116:0/1684769750 wait complete. 2024-03-04T07:37:53.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T07:37:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 -- 172.21.15.116:0/3542167368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9324074030 msgr2=0x7f93240744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 --2- 172.21.15.116:0/3542167368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9324074030 0x7f93240744b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9310009860 tx=0x7f931002bb70 comp rx=0 tx=0).stop 2024-03-04T07:37:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 -- 172.21.15.116:0/3542167368 shutdown_connections 2024-03-04T07:37:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 --2- 172.21.15.116:0/3542167368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93241094a0 0x7f932410b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 --2- 172.21.15.116:0/3542167368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9324074030 0x7f93240744b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 --2- 172.21.15.116:0/3542167368 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9324075610 0x7f9324073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 -- 172.21.15.116:0/3542167368 >> 172.21.15.116:0/3542167368 conn(0x7f93240fcb50 msgr2=0x7f93240fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:53.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 -- 172.21.15.116:0/3542167368 shutdown_connections 2024-03-04T07:37:53.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.310+0000 7f932be00640 1 -- 172.21.15.116:0/3542167368 wait complete. 2024-03-04T07:37:53.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 Processor -- start 2024-03-04T07:37:53.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 -- start start 2024-03-04T07:37:53.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9324074030 0x7f9324195760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:53.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9324075610 0x7f9324195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93241094a0 0x7f932419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932410f440 con 0x7f9324074030 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932410f2c0 con 0x7f9324075610 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f932be00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932410f5c0 con 0x7f93241094a0 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f9329400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9324074030 0x7f9324195760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f9329e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93241094a0 0x7f932419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f9329400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9324074030 0x7f9324195760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57498/0 (socket says 172.21.15.116:57498) 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f9329e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93241094a0 0x7f932419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:38958/0 (socket says 172.21.15.116:38958) 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f9329400640 1 -- 172.21.15.116:0/213796748 learned_addr learned my addr 172.21.15.116:0/213796748 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.312+0000 7f9328a00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9324075610 0x7f9324195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9329e00640 1 -- 172.21.15.116:0/213796748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9324075610 msgr2=0x7f9324195ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9329e00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9324075610 0x7f9324195ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9329e00640 1 -- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9324074030 msgr2=0x7f9324195760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:53.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9329e00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9324074030 0x7f9324195760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9329e00640 1 -- 172.21.15.116:0/213796748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93241ae3c0 con 0x7f93241094a0 2024-03-04T07:37:53.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9329e00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93241094a0 0x7f932419cd10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f931c00ef30 tx=0x7f931c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:53.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9322000640 1 -- 172.21.15.116:0/213796748 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c004b20 con 0x7f93241094a0 2024-03-04T07:37:53.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9322000640 1 -- 172.21.15.116:0/213796748 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f931c003e50 con 0x7f93241094a0 2024-03-04T07:37:53.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93241ae6b0 con 0x7f93241094a0 2024-03-04T07:37:53.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f9322000640 1 -- 172.21.15.116:0/213796748 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c00b570 con 0x7f93241094a0 2024-03-04T07:37:53.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.313+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93241aeb80 con 0x7f93241094a0 2024-03-04T07:37:53.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.315+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9324103dc0 con 0x7f93241094a0 2024-03-04T07:37:53.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.316+0000 7f9322000640 1 -- 172.21.15.116:0/213796748 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f931c01d020 con 0x7f93241094a0 2024-03-04T07:37:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.318+0000 7f9322000640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f930c0a1d80 0x7f930c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.318+0000 7f9322000640 1 -- 172.21.15.116:0/213796748 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f931c0cd990 con 0x7f93241094a0 2024-03-04T07:37:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.319+0000 7f9329400640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f930c0a1d80 0x7f930c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:53.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.319+0000 7f9329400640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f930c0a1d80 0x7f930c0a4240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9318009cd0 tx=0x7f9318005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:53.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.321+0000 7f9322000640 1 -- 172.21.15.116:0/213796748 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f931c09b050 con 0x7f93241094a0 2024-03-04T07:37:53.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:53.528+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f93240621d0 con 0x7f93241094a0 2024-03-04T07:37:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.108+0000 7f9322000640 1 -- 172.21.15.116:0/213796748 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7f931c09a950 con 0x7f93241094a0 2024-03-04T07:37:54.109 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2024-03-04T07:37:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f930c0a1d80 msgr2=0x7f930c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f930c0a1d80 0x7f930c0a4240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9318009cd0 tx=0x7f9318005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93241094a0 msgr2=0x7f932419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93241094a0 0x7f932419cd10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f931c00ef30 tx=0x7f931c00c800 comp rx=0 tx=0).stop 2024-03-04T07:37:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 shutdown_connections 2024-03-04T07:37:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f930c0a1d80 0x7f930c0a4240 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9318009cd0 tx=0x7f9318005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:54.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93241094a0 0x7f932419cd10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9324075610 0x7f9324195ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 --2- 172.21.15.116:0/213796748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9324074030 0x7f9324195760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.113+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 >> 172.21.15.116:0/213796748 conn(0x7f93240fcb50 msgr2=0x7f9324100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.114+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 shutdown_connections 2024-03-04T07:37:54.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.114+0000 7f932be00640 1 -- 172.21.15.116:0/213796748 wait complete. 2024-03-04T07:37:54.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T07:37:54.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 -- 172.21.15.116:0/593680851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 msgr2=0x7fb144108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 --2- 172.21.15.116:0/593680851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb144108010 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb134009900 tx=0x7fb13402bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 -- 172.21.15.116:0/593680851 shutdown_connections 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 --2- 172.21.15.116:0/593680851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1441097a0 0x7fb144110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 --2- 172.21.15.116:0/593680851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb144108de0 0x7fb144109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 --2- 172.21.15.116:0/593680851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb144108010 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 -- 172.21.15.116:0/593680851 >> 172.21.15.116:0/593680851 conn(0x7fb144076b90 msgr2=0x7fb144078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.323+0000 7fb14a400640 1 -- 172.21.15.116:0/593680851 shutdown_connections 2024-03-04T07:37:54.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.324+0000 7fb14a400640 1 -- 172.21.15.116:0/593680851 wait complete. 2024-03-04T07:37:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.324+0000 7fb14a400640 1 Processor -- start 2024-03-04T07:37:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb14a400640 1 -- start start 2024-03-04T07:37:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb14a400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb14419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb14a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb144108de0 0x7fb14419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb14a400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1441097a0 0x7fb1441a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb14a400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144113890 con 0x7fb144108de0 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb14a400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144113710 con 0x7fb144107bf0 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb14a400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144113a10 con 0x7fb1441097a0 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb143400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb14419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb142a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb144108de0 0x7fb14419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb143e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1441097a0 0x7fb1441a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb142a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb144108de0 0x7fb14419e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57510/0 (socket says 172.21.15.116:57510) 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb143400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb14419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59950/0 (socket says 172.21.15.116:59950) 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.325+0000 7fb142a00640 1 -- 172.21.15.116:0/359155009 learned_addr learned my addr 172.21.15.116:0/359155009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb143400640 1 -- 172.21.15.116:0/359155009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1441097a0 msgr2=0x7fb1441a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb143400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1441097a0 0x7fb1441a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb143400640 1 -- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb144108de0 msgr2=0x7fb14419e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb143400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb144108de0 0x7fb14419e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb143400640 1 -- 172.21.15.116:0/359155009 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1441a5b60 con 0x7fb144107bf0 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb143400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb14419df90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb134009230 tx=0x7fb13402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb13be00640 1 -- 172.21.15.116:0/359155009 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134043020 con 0x7fb144107bf0 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb13be00640 1 -- 172.21.15.116:0/359155009 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1340054e0 con 0x7fb144107bf0 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1441a5df0 con 0x7fb144107bf0 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb13be00640 1 -- 172.21.15.116:0/359155009 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134003c60 con 0x7fb144107bf0 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.326+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1441a6250 con 0x7fb144107bf0 2024-03-04T07:37:54.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.327+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14410efe0 con 0x7fb144107bf0 2024-03-04T07:37:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.329+0000 7fb13be00640 1 -- 172.21.15.116:0/359155009 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb134046020 con 0x7fb144107bf0 2024-03-04T07:37:54.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.330+0000 7fb13be00640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1280a1d80 0x7fb1280a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:54.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.331+0000 7fb13be00640 1 -- 172.21.15.116:0/359155009 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fb1340f0090 con 0x7fb144107bf0 2024-03-04T07:37:54.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.331+0000 7fb142a00640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1280a1d80 0x7fb1280a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:54.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.332+0000 7fb142a00640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1280a1d80 0x7fb1280a4240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb1300095f0 tx=0x7fb130005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:54.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.333+0000 7fb13be00640 1 -- 172.21.15.116:0/359155009 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb13404d020 con 0x7fb144107bf0 2024-03-04T07:37:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1684769750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-04T07:37:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1684769750' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-04T07:37:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: osdmap e82: 12 total, 12 up, 12 in 2024-03-04T07:37:54.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:54 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/213796748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T07:37:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1684769750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-04T07:37:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1684769750' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-04T07:37:54.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: osdmap e82: 12 total, 12 up, 12 in 2024-03-04T07:37:54.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:54 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/213796748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T07:37:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:54.571+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb14402cff0 con 0x7fb144107bf0 2024-03-04T07:37:54.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1684769750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-04T07:37:54.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1684769750' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-04T07:37:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: osdmap e82: 12 total, 12 up, 12 in 2024-03-04T07:37:54.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:54 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/213796748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T07:37:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.148+0000 7fb13be00640 1 -- 172.21.15.116:0/359155009 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb1340bd5d0 con 0x7fb144107bf0 2024-03-04T07:37:55.148 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 2 and data pool 3 2024-03-04T07:37:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.153+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1280a1d80 msgr2=0x7fb1280a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.153+0000 7fb14a400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1280a1d80 0x7fb1280a4240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb1300095f0 tx=0x7fb130005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.153+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 msgr2=0x7fb14419df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.153+0000 7fb14a400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb14419df90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb134009230 tx=0x7fb13402c6c0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 shutdown_connections 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1280a1d80 0x7fb1280a4240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1441097a0 0x7fb1441a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb144108de0 0x7fb14419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 --2- 172.21.15.116:0/359155009 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb144107bf0 0x7fb14419df90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 >> 172.21.15.116:0/359155009 conn(0x7fb144076b90 msgr2=0x7fb14410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 shutdown_connections 2024-03-04T07:37:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.154+0000 7fb14a400640 1 -- 172.21.15.116:0/359155009 wait complete. 2024-03-04T07:37:55.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:37:55.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 -- 172.21.15.116:0/3250919095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 msgr2=0x7f06d8103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3250919095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8103c10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009900 tx=0x7f06c802bc50 comp rx=0 tx=0).stop 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 -- 172.21.15.116:0/3250919095 shutdown_connections 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3250919095 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06d81053a0 0x7f06d810bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3250919095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d81049e0 0x7f06d8104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3250919095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8103c10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 -- 172.21.15.116:0/3250919095 >> 172.21.15.116:0/3250919095 conn(0x7f06d80fed80 msgr2=0x7f06d81011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 -- 172.21.15.116:0/3250919095 shutdown_connections 2024-03-04T07:37:55.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.355+0000 7f06dcc00640 1 -- 172.21.15.116:0/3250919095 wait complete. 2024-03-04T07:37:55.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.356+0000 7f06dcc00640 1 Processor -- start 2024-03-04T07:37:55.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06dcc00640 1 -- start start 2024-03-04T07:37:55.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06dcc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06dcc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06d81049e0 0x7f06d819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06dcc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d81053a0 0x7f06d81a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06dcc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d810f490 con 0x7f06d81037f0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06dcc00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d810f310 con 0x7f06d81049e0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06dcc00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d810f610 con 0x7f06d81053a0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06d5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06d5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57518/0 (socket says 172.21.15.116:57518) 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06d5e00640 1 -- 172.21.15.116:0/3642669555 learned_addr learned my addr 172.21.15.116:0/3642669555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06d6800640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d81053a0 0x7f06d81a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.357+0000 7f06d5400640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06d81049e0 0x7f06d819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06d5e00640 1 -- 172.21.15.116:0/3642669555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d81053a0 msgr2=0x7f06d81a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06d5e00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d81053a0 0x7f06d81a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06d5e00640 1 -- 172.21.15.116:0/3642669555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06d81049e0 msgr2=0x7f06d819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06d5e00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06d81049e0 0x7f06d819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06d5e00640 1 -- 172.21.15.116:0/3642669555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d81a1740 con 0x7f06d81037f0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06d5e00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8199bc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f06c8005ab0 tx=0x7f06c802c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06cea00640 1 -- 172.21.15.116:0/3642669555 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8041070 con 0x7f06d81037f0 2024-03-04T07:37:55.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06cea00640 1 -- 172.21.15.116:0/3642669555 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06c8004450 con 0x7f06d81037f0 2024-03-04T07:37:55.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d81a19d0 con 0x7f06d81037f0 2024-03-04T07:37:55.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.358+0000 7f06cea00640 1 -- 172.21.15.116:0/3642669555 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8034440 con 0x7f06d81037f0 2024-03-04T07:37:55.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.359+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d81a1f70 con 0x7f06d81037f0 2024-03-04T07:37:55.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.360+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d810ab10 con 0x7f06d81037f0 2024-03-04T07:37:55.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.363+0000 7f06cea00640 1 -- 172.21.15.116:0/3642669555 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f06c8044070 con 0x7f06d81037f0 2024-03-04T07:37:55.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.364+0000 7f06cea00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f06bc0a1e00 0x7f06bc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.364+0000 7f06cea00640 1 -- 172.21.15.116:0/3642669555 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f06c80efb00 con 0x7f06d81037f0 2024-03-04T07:37:55.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.364+0000 7f06d5400640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f06bc0a1e00 0x7f06bc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.365+0000 7f06d5400640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f06bc0a1e00 0x7f06bc0a42c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f06d819b030 tx=0x7f06c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:55.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.367+0000 7f06cea00640 1 -- 172.21.15.116:0/3642669555 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f06c80bca30 con 0x7f06d81037f0 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: pgmap v310: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: osdmap e83: 12 total, 12 up, 12 in 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/359155009' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T07:37:55.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T07:37:55.510 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:55 smithi116 ceph-mon[32150]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: pgmap v310: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: osdmap e83: 12 total, 12 up, 12 in 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:55.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/359155009' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T07:37:55.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T07:37:55.522 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:55 smithi152 ceph-mon[37154]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-04T07:37:55.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.565+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06d802d1c0 con 0x7f06d81037f0 2024-03-04T07:37:55.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.568+0000 7f06cea00640 1 -- 172.21.15.116:0/3642669555 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+22315 (secure 0 0 0) 0x7f06c80c0ad0 con 0x7f06d81037f0 2024-03-04T07:37:55.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:55.569 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":84,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:37:54.572966+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f06bc0a1e00 msgr2=0x7f06bc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f06bc0a1e00 0x7f06bc0a42c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f06d819b030 tx=0x7f06c4005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 msgr2=0x7f06d8199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8199bc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f06c8005ab0 tx=0x7f06c802c6e0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 shutdown_connections 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f06bc0a1e00 0x7f06bc0a42c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d81053a0 0x7f06d81a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06d81049e0 0x7f06d819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 --2- 172.21.15.116:0/3642669555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06d81037f0 0x7f06d8199bc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 >> 172.21.15.116:0/3642669555 conn(0x7f06d80fed80 msgr2=0x7f06d8107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 shutdown_connections 2024-03-04T07:37:55.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.571+0000 7f06dcc00640 1 -- 172.21.15.116:0/3642669555 wait complete. 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: pgmap v310: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: osdmap e83: 12 total, 12 up, 12 in 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/359155009' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T07:37:55.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T07:37:55.588 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:55 smithi133 ceph-mon[36505]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-04T07:37:55.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:37:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 -- 172.21.15.116:0/3612670327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8102600 msgr2=0x7fdcc8102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 --2- 172.21.15.116:0/3612670327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8102600 0x7fdcc8102a80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009900 tx=0x7fdcb002bc50 comp rx=0 tx=0).stop 2024-03-04T07:37:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 -- 172.21.15.116:0/3612670327 shutdown_connections 2024-03-04T07:37:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 --2- 172.21.15.116:0/3612670327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcc8102fc0 0x7fdcc8109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 --2- 172.21.15.116:0/3612670327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8102600 0x7fdcc8102a80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 --2- 172.21.15.116:0/3612670327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcc8101410 0x7fdcc8101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 -- 172.21.15.116:0/3612670327 >> 172.21.15.116:0/3612670327 conn(0x7fdcc80fc960 msgr2=0x7fdcc80fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 -- 172.21.15.116:0/3612670327 shutdown_connections 2024-03-04T07:37:55.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.800+0000 7fdccce00640 1 -- 172.21.15.116:0/3612670327 wait complete. 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.801+0000 7fdccce00640 1 Processor -- start 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.801+0000 7fdccce00640 1 -- start start 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdccce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8101410 0x7fdcc8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdccce00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcc8102600 0x7fdcc80739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdccce00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcc8102fc0 0x7fdcc8073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdccce00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc810d0f0 con 0x7fdcc8101410 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdccce00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc810cf70 con 0x7fdcc8102600 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdccce00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc810d270 con 0x7fdcc8102fc0 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8101410 0x7fdcc8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8101410 0x7fdcc8075380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57540/0 (socket says 172.21.15.116:57540) 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 -- 172.21.15.116:0/2995417056 learned_addr learned my addr 172.21.15.116:0/2995417056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:55.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7e00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcc8102fc0 0x7fdcc8073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcbfe00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcc8102600 0x7fdcc80739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 -- 172.21.15.116:0/2995417056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcc8102fc0 msgr2=0x7fdcc8073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcc8102fc0 0x7fdcc8073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 -- 172.21.15.116:0/2995417056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcc8102600 msgr2=0x7fdcc80739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcc8102600 0x7fdcc80739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.802+0000 7fdcc7400640 1 -- 172.21.15.116:0/2995417056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8074790 con 0x7fdcc8101410 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.803+0000 7fdcc7400640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8101410 0x7fdcc8075380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800e9c0 tx=0x7fdcb800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.803+0000 7fdcc4c00640 1 -- 172.21.15.116:0/2995417056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb800cd80 con 0x7fdcc8101410 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.803+0000 7fdcc4c00640 1 -- 172.21.15.116:0/2995417056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcb80035f0 con 0x7fdcc8101410 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.803+0000 7fdcc4c00640 1 -- 172.21.15.116:0/2995417056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8010690 con 0x7fdcc8101410 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.803+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc81a1860 con 0x7fdcc8101410 2024-03-04T07:37:55.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.803+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc81a1d30 con 0x7fdcc8101410 2024-03-04T07:37:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.804+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc8068730 con 0x7fdcc8101410 2024-03-04T07:37:55.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.806+0000 7fdcc4c00640 1 -- 172.21.15.116:0/2995417056 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fdcb80037a0 con 0x7fdcc8101410 2024-03-04T07:37:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.808+0000 7fdcc4c00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdcac0a1d80 0x7fdcac0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.809+0000 7fdcc4c00640 1 -- 172.21.15.116:0/2995417056 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdcb80ce1d0 con 0x7fdcc8101410 2024-03-04T07:37:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.809+0000 7fdcbfe00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdcac0a1d80 0x7fdcac0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:55.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.809+0000 7fdcbfe00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdcac0a1d80 0x7fdcac0a4240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdcb002c260 tx=0x7fdcb0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:55.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:55.811+0000 7fdcc4c00640 1 -- 172.21.15.116:0/2995417056 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdcb809b6b0 con 0x7fdcc8101410 2024-03-04T07:37:56.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.012+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc80621d0 con 0x7fdcc8101410 2024-03-04T07:37:56.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.016+0000 7fdcc4c00640 1 -- 172.21.15.116:0/2995417056 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+22315 (secure 0 0 0) 0x7fdcb8079e20 con 0x7fdcc8101410 2024-03-04T07:37:56.017 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:37:56.017 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":84,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:37:54.572966+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdcac0a1d80 msgr2=0x7fdcac0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdcac0a1d80 0x7fdcac0a4240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdcb002c260 tx=0x7fdcb0000bc0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8101410 msgr2=0x7fdcc8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8101410 0x7fdcc8075380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800e9c0 tx=0x7fdcb800ee80 comp rx=0 tx=0).stop 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 shutdown_connections 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdcac0a1d80 0x7fdcac0a4240 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcc8102fc0 0x7fdcc8073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcc8102600 0x7fdcc80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 --2- 172.21.15.116:0/2995417056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcc8101410 0x7fdcc8075380 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 >> 172.21.15.116:0/2995417056 conn(0x7fdcc80fc960 msgr2=0x7fdcc8105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:56.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 shutdown_connections 2024-03-04T07:37:56.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.019+0000 7fdccce00640 1 -- 172.21.15.116:0/2995417056 wait complete. 2024-03-04T07:37:56.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T07:37:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 -- 172.21.15.116:0/927667809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601036f0 msgr2=0x7ff960103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 --2- 172.21.15.116:0/927667809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601036f0 0x7ff960103b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff950009900 tx=0x7ff95002bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 -- 172.21.15.116:0/927667809 shutdown_connections 2024-03-04T07:37:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 --2- 172.21.15.116:0/927667809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601052a0 0x7ff96010bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 --2- 172.21.15.116:0/927667809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 0x7ff960104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 --2- 172.21.15.116:0/927667809 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601036f0 0x7ff960103b10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 -- 172.21.15.116:0/927667809 >> 172.21.15.116:0/927667809 conn(0x7ff9600fec80 msgr2=0x7ff9601010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 -- 172.21.15.116:0/927667809 shutdown_connections 2024-03-04T07:37:56.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.252+0000 7ff965600640 1 -- 172.21.15.116:0/927667809 wait complete. 2024-03-04T07:37:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.253+0000 7ff965600640 1 Processor -- start 2024-03-04T07:37:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.253+0000 7ff965600640 1 -- start start 2024-03-04T07:37:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff965600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601036f0 0x7ff9601999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff965600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 0x7ff960199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:56.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff965600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601052a0 0x7ff9601a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff965600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96010f340 con 0x7ff9601048e0 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff965600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96010f1c0 con 0x7ff9601052a0 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff965600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96010f4c0 con 0x7ff9601036f0 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff957400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 0x7ff960199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff95fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601036f0 0x7ff9601999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff957400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 0x7ff960199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36882/0 (socket says 172.21.15.116:36882) 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff95fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601036f0 0x7ff9601999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41202/0 (socket says 172.21.15.116:41202) 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff957400640 1 -- 172.21.15.116:0/2048270393 learned_addr learned my addr 172.21.15.116:0/2048270393 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff964c00640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601052a0 0x7ff9601a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.254+0000 7ff957400640 1 -- 172.21.15.116:0/2048270393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601036f0 msgr2=0x7ff9601999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff957400640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601036f0 0x7ff9601999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff957400640 1 -- 172.21.15.116:0/2048270393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601052a0 msgr2=0x7ff9601a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff957400640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601052a0 0x7ff9601a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff957400640 1 -- 172.21.15.116:0/2048270393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9601a1610 con 0x7ff9601048e0 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff95fe00640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601036f0 0x7ff9601999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff957400640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 0x7ff960199f30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff94800ebc0 tx=0x7ff94800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff95d600640 1 -- 172.21.15.116:0/2048270393 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948010630 con 0x7ff9601048e0 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff95d600640 1 -- 172.21.15.116:0/2048270393 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff948010cb0 con 0x7ff9601048e0 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff95d600640 1 -- 172.21.15.116:0/2048270393 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948018bd0 con 0x7ff9601048e0 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9601a1900 con 0x7ff9601048e0 2024-03-04T07:37:56.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.255+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9601a1ea0 con 0x7ff9601048e0 2024-03-04T07:37:56.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.256+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96010aa90 con 0x7ff9601048e0 2024-03-04T07:37:56.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.258+0000 7ff95d600640 1 -- 172.21.15.116:0/2048270393 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff9480107d0 con 0x7ff9601048e0 2024-03-04T07:37:56.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.259+0000 7ff95d600640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9440a1e00 0x7ff9440a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:56.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.259+0000 7ff95d600640 1 -- 172.21.15.116:0/2048270393 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff9480ce1b0 con 0x7ff9601048e0 2024-03-04T07:37:56.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.259+0000 7ff95fe00640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9440a1e00 0x7ff9440a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:56.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.260+0000 7ff95fe00640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9440a1e00 0x7ff9440a42c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff950000c00 tx=0x7ff950005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:56.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.263+0000 7ff95d600640 1 -- 172.21.15.116:0/2048270393 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff94809b0e0 con 0x7ff9601048e0 2024-03-04T07:37:56.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:56.501+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff96002cff0 con 0x7ff9601048e0 2024-03-04T07:37:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:56 smithi116 ceph-mon[32150]: osdmap e84: 12 total, 12 up, 12 in 2024-03-04T07:37:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:56 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-04T07:37:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:56 smithi116 ceph-mon[32150]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-04T07:37:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:56 smithi116 ceph-mon[32150]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-04T07:37:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:56 smithi116 ceph-mon[32150]: pgmap v313: 97 pgs: 11 creating+peering, 28 active+clean, 58 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-04T07:37:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:56 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3642669555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:56.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:56 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2995417056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:56 smithi152 ceph-mon[37154]: osdmap e84: 12 total, 12 up, 12 in 2024-03-04T07:37:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:56 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-04T07:37:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:56 smithi152 ceph-mon[37154]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-04T07:37:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:56 smithi152 ceph-mon[37154]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-04T07:37:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:56 smithi152 ceph-mon[37154]: pgmap v313: 97 pgs: 11 creating+peering, 28 active+clean, 58 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-04T07:37:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:56 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3642669555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:56 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2995417056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:56.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:56 smithi133 ceph-mon[36505]: osdmap e84: 12 total, 12 up, 12 in 2024-03-04T07:37:56.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:56 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-04T07:37:56.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:56 smithi133 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-04T07:37:56.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:56 smithi133 ceph-mon[36505]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-04T07:37:56.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:56 smithi133 ceph-mon[36505]: pgmap v313: 97 pgs: 11 creating+peering, 28 active+clean, 58 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-04T07:37:56.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:56 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3642669555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:56.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:56 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2995417056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:37:57.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.159+0000 7ff95d600640 1 -- 172.21.15.116:0/2048270393 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff94809f180 con 0x7ff9601048e0 2024-03-04T07:37:57.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9440a1e00 msgr2=0x7ff9440a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9440a1e00 0x7ff9440a42c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff950000c00 tx=0x7ff950005fd0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 msgr2=0x7ff960199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 0x7ff960199f30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff94800ebc0 tx=0x7ff94800c6a0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 shutdown_connections 2024-03-04T07:37:57.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff9601052a0 0x7ff9601a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9601048e0 0x7ff960199f30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9440a1e00 0x7ff9440a42c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 --2- 172.21.15.116:0/2048270393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9601036f0 0x7ff9601999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 >> 172.21.15.116:0/2048270393 conn(0x7ff9600fec80 msgr2=0x7ff960107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:57.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 shutdown_connections 2024-03-04T07:37:57.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.163+0000 7ff965600640 1 -- 172.21.15.116:0/2048270393 wait complete. 2024-03-04T07:37:57.183 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-04T07:37:57.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-04T07:37:57.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 -- 172.21.15.116:0/531047684 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3354107310 msgr2=0x7f3354109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 --2- 172.21.15.116:0/531047684 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3354107310 0x7f3354109700 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f334c009900 tx=0x7f334c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:57.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 -- 172.21.15.116:0/531047684 shutdown_connections 2024-03-04T07:37:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 --2- 172.21.15.116:0/531047684 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3354107310 0x7f3354109700 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 --2- 172.21.15.116:0/531047684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33541049e0 0x7f3354106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 --2- 172.21.15.116:0/531047684 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f33541020b0 0x7f33541044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 -- 172.21.15.116:0/531047684 >> 172.21.15.116:0/531047684 conn(0x7f33540fba60 msgr2=0x7f33540fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 -- 172.21.15.116:0/531047684 shutdown_connections 2024-03-04T07:37:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.401+0000 7f335b000640 1 -- 172.21.15.116:0/531047684 wait complete. 2024-03-04T07:37:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.402+0000 7f335b000640 1 Processor -- start 2024-03-04T07:37:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.402+0000 7f335b000640 1 -- start start 2024-03-04T07:37:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f335b000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33541020b0 0x7f3354197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f335b000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33541049e0 0x7f3354197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f335b000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3354107310 0x7f335419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f335b000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335410d2b0 con 0x7f33541020b0 2024-03-04T07:37:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f335b000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335410d130 con 0x7f33541049e0 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f335b000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335410d430 con 0x7f3354107310 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33541049e0 0x7f3354197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33541049e0 0x7f3354197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60164/0 (socket says 172.21.15.116:60164) 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 -- 172.21.15.116:0/1174409422 learned_addr learned my addr 172.21.15.116:0/1174409422 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3359000640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3354107310 0x7f335419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353e00640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33541020b0 0x7f3354197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 -- 172.21.15.116:0/1174409422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3354107310 msgr2=0x7f335419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3354107310 0x7f335419eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 -- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33541020b0 msgr2=0x7f3354197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33541020b0 0x7f3354197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353400640 1 -- 172.21.15.116:0/1174409422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335419f510 con 0x7f33541049e0 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.403+0000 7f3353e00640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33541020b0 0x7f3354197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:37:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.404+0000 7f3353400640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33541049e0 0x7f3354197ea0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f333c00eb90 tx=0x7f333c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.404+0000 7f3350c00640 1 -- 172.21.15.116:0/1174409422 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c010630 con 0x7f33541049e0 2024-03-04T07:37:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.404+0000 7f3350c00640 1 -- 172.21.15.116:0/1174409422 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f333c00ee90 con 0x7f33541049e0 2024-03-04T07:37:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.404+0000 7f3350c00640 1 -- 172.21.15.116:0/1174409422 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c018b90 con 0x7f33541049e0 2024-03-04T07:37:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.404+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335419f800 con 0x7f33541049e0 2024-03-04T07:37:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.404+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335419fd50 con 0x7f33541049e0 2024-03-04T07:37:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.405+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33540fd550 con 0x7f33541049e0 2024-03-04T07:37:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.407+0000 7f3350c00640 1 -- 172.21.15.116:0/1174409422 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f333c0107d0 con 0x7f33541049e0 2024-03-04T07:37:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.408+0000 7f3350c00640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33340a1d80 0x7f33340a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.408+0000 7f3350c00640 1 -- 172.21.15.116:0/1174409422 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f333c0ce300 con 0x7f33541049e0 2024-03-04T07:37:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.408+0000 7f3353e00640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33340a1d80 0x7f33340a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.409+0000 7f3353e00640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33340a1d80 0x7f33340a4240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3344009c20 tx=0x7f3344005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:57.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.412+0000 7f3350c00640 1 -- 172.21.15.116:0/1174409422 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f333c09b7e0 con 0x7f33541049e0 2024-03-04T07:37:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:57 smithi116 ceph-mon[32150]: osdmap e85: 12 total, 12 up, 12 in 2024-03-04T07:37:57.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:57 smithi116 ceph-mon[32150]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T07:37:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:57 smithi116 ceph-mon[32150]: Cluster is now healthy 2024-03-04T07:37:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:57 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] up:active 2024-03-04T07:37:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:57 smithi116 ceph-mon[32150]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:57 smithi116 ceph-mon[32150]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:57.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:57 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2048270393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-04T07:37:57.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:57 smithi152 ceph-mon[37154]: osdmap e85: 12 total, 12 up, 12 in 2024-03-04T07:37:57.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:57 smithi152 ceph-mon[37154]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T07:37:57.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:57 smithi152 ceph-mon[37154]: Cluster is now healthy 2024-03-04T07:37:57.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:57 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] up:active 2024-03-04T07:37:57.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:57 smithi152 ceph-mon[37154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:57.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:57 smithi152 ceph-mon[37154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:57.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:57 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2048270393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-04T07:37:57.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:57 smithi133 ceph-mon[36505]: osdmap e85: 12 total, 12 up, 12 in 2024-03-04T07:37:57.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:57 smithi133 ceph-mon[36505]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T07:37:57.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:57 smithi133 ceph-mon[36505]: Cluster is now healthy 2024-03-04T07:37:57.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:57 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] up:active 2024-03-04T07:37:57.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:57 smithi133 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:57.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:57 smithi133 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:57.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:57 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2048270393' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-04T07:37:57.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.655+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f33540621d0 con 0x7f33541049e0 2024-03-04T07:37:57.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.761+0000 7f3350c00640 1 -- 172.21.15.116:0/1174409422 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7f333c09b0e0 con 0x7f33541049e0 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.763+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33340a1d80 msgr2=0x7f33340a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.763+0000 7f335b000640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33340a1d80 0x7f33340a4240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3344009c20 tx=0x7f3344005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.763+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33541049e0 msgr2=0x7f3354197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.763+0000 7f335b000640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33541049e0 0x7f3354197ea0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f333c00eb90 tx=0x7f333c00c6a0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 shutdown_connections 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33340a1d80 0x7f33340a4240 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3354107310 0x7f335419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f33541049e0 0x7f3354197ea0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 --2- 172.21.15.116:0/1174409422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33541020b0 0x7f3354197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 >> 172.21.15.116:0/1174409422 conn(0x7f33540fba60 msgr2=0x7f3354102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 shutdown_connections 2024-03-04T07:37:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.764+0000 7f335b000640 1 -- 172.21.15.116:0/1174409422 wait complete. 2024-03-04T07:37:57.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 -- 172.21.15.116:0/1549949455 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c107bf0 msgr2=0x7f5f5c108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 --2- 172.21.15.116:0/1549949455 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c107bf0 0x7f5f5c108010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009900 tx=0x7f5f4c02bc50 comp rx=0 tx=0).stop 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 -- 172.21.15.116:0/1549949455 shutdown_connections 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 --2- 172.21.15.116:0/1549949455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c1097a0 0x7f5f5c110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 --2- 172.21.15.116:0/1549949455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f5c108de0 0x7f5f5c109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 --2- 172.21.15.116:0/1549949455 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c107bf0 0x7f5f5c108010 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 -- 172.21.15.116:0/1549949455 >> 172.21.15.116:0/1549949455 conn(0x7f5f5c076b90 msgr2=0x7f5f5c078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 -- 172.21.15.116:0/1549949455 shutdown_connections 2024-03-04T07:37:57.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.987+0000 7f5f62600640 1 -- 172.21.15.116:0/1549949455 wait complete. 2024-03-04T07:37:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 Processor -- start 2024-03-04T07:37:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 -- start start 2024-03-04T07:37:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f5c107bf0 0x7f5f5c19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c108de0 0x7f5f5c19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c1097a0 0x7f5f5c1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c1138b0 con 0x7f5f5c108de0 2024-03-04T07:37:57.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c113730 con 0x7f5f5c1097a0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f62600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c113a30 con 0x7f5f5c107bf0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f5be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c1097a0 0x7f5f5c1a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f53e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c108de0 0x7f5f5c19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f5be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c1097a0 0x7f5f5c1a54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60186/0 (socket says 172.21.15.116:60186) 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.988+0000 7f5f5be00640 1 -- 172.21.15.116:0/4288978655 learned_addr learned my addr 172.21.15.116:0/4288978655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f5b400640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f5c107bf0 0x7f5f5c19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f5be00640 1 -- 172.21.15.116:0/4288978655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f5c107bf0 msgr2=0x7f5f5c19df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f5be00640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f5c107bf0 0x7f5f5c19df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f5be00640 1 -- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c108de0 msgr2=0x7f5f5c19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f5be00640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c108de0 0x7f5f5c19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f5be00640 1 -- 172.21.15.116:0/4288978655 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c1a5b40 con 0x7f5f5c1097a0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f5be00640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c1097a0 0x7f5f5c1a54d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400eb10 tx=0x7f5f5400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f58c00640 1 -- 172.21.15.116:0/4288978655 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54004b20 con 0x7f5f5c1097a0 2024-03-04T07:37:57.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f58c00640 1 -- 172.21.15.116:0/4288978655 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f54004cc0 con 0x7f5f5c1097a0 2024-03-04T07:37:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c1a5e30 con 0x7f5f5c1097a0 2024-03-04T07:37:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f58c00640 1 -- 172.21.15.116:0/4288978655 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54010760 con 0x7f5f5c1097a0 2024-03-04T07:37:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.989+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c1a62f0 con 0x7f5f5c1097a0 2024-03-04T07:37:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.990+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c10ef10 con 0x7f5f5c1097a0 2024-03-04T07:37:57.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.992+0000 7f5f58c00640 1 -- 172.21.15.116:0/4288978655 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5f54010900 con 0x7f5f5c1097a0 2024-03-04T07:37:57.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.993+0000 7f5f58c00640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5f400a1e00 0x7f5f400a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:57.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.994+0000 7f5f58c00640 1 -- 172.21.15.116:0/4288978655 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5f540ce2f0 con 0x7f5f5c1097a0 2024-03-04T07:37:57.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.994+0000 7f5f5b400640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5f400a1e00 0x7f5f400a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:57.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.994+0000 7f5f5b400640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5f400a1e00 0x7f5f400a42c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c02c190 tx=0x7f5f4c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:57.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:57.996+0000 7f5f58c00640 1 -- 172.21.15.116:0/4288978655 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5f5409b7d0 con 0x7f5f5c1097a0 2024-03-04T07:37:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:58.238+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f5f5c02cff0 con 0x7f5f5c1097a0 2024-03-04T07:37:58.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:58 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2048270393' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-04T07:37:58.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:58 smithi116 ceph-mon[32150]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:58.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:58 smithi116 ceph-mon[32150]: pgmap v315: 97 pgs: 11 creating+peering, 64 active+clean, 22 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-03-04T07:37:58.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:58 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1174409422' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T07:37:58.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:58 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T07:37:58.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:58 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-04T07:37:58.509 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:58 smithi116 ceph-mon[32150]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-04T07:37:58.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:58 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2048270393' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-04T07:37:58.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:58 smithi152 ceph-mon[37154]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:58.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:58 smithi152 ceph-mon[37154]: pgmap v315: 97 pgs: 11 creating+peering, 64 active+clean, 22 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-03-04T07:37:58.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:58 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1174409422' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T07:37:58.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:58 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T07:37:58.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:58 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-04T07:37:58.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:58 smithi152 ceph-mon[37154]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-04T07:37:58.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:58 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2048270393' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-04T07:37:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:58 smithi133 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T07:37:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:58 smithi133 ceph-mon[36505]: pgmap v315: 97 pgs: 11 creating+peering, 64 active+clean, 22 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-03-04T07:37:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:58 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1174409422' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T07:37:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:58 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T07:37:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:58 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-04T07:37:58.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:58 smithi133 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-04T07:37:59.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.174+0000 7f5f58c00640 1 -- 172.21.15.116:0/4288978655 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5f5409b0d0 con 0x7f5f5c1097a0 2024-03-04T07:37:59.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5f400a1e00 msgr2=0x7f5f400a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5f400a1e00 0x7f5f400a42c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c02c190 tx=0x7f5f4c000bc0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c1097a0 msgr2=0x7f5f5c1a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c1097a0 0x7f5f5c1a54d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400eb10 tx=0x7f5f5400c2d0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 shutdown_connections 2024-03-04T07:37:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5f5c1097a0 0x7f5f5c1a54d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f5c108de0 0x7f5f5c19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5f400a1e00 0x7f5f400a42c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 --2- 172.21.15.116:0/4288978655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f5c107bf0 0x7f5f5c19df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 >> 172.21.15.116:0/4288978655 conn(0x7f5f5c076b90 msgr2=0x7f5f5c10c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 shutdown_connections 2024-03-04T07:37:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.179+0000 7f5f62600640 1 -- 172.21.15.116:0/4288978655 wait complete. 2024-03-04T07:37:59.197 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-04T07:37:59.197 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-04T07:37:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 -- 172.21.15.116:0/548840755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 msgr2=0x7fb46c109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 --2- 172.21.15.116:0/548840755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c109720 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb464009be0 tx=0x7fb46402bef0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 -- 172.21.15.116:0/548840755 shutdown_connections 2024-03-04T07:37:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 --2- 172.21.15.116:0/548840755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c109720 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 --2- 172.21.15.116:0/548840755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46c104a00 0x7fb46c106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 --2- 172.21.15.116:0/548840755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb46c1020d0 0x7fb46c1044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 -- 172.21.15.116:0/548840755 >> 172.21.15.116:0/548840755 conn(0x7fb46c0fba80 msgr2=0x7fb46c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 -- 172.21.15.116:0/548840755 shutdown_connections 2024-03-04T07:37:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.387+0000 7fb473600640 1 -- 172.21.15.116:0/548840755 wait complete. 2024-03-04T07:37:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 Processor -- start 2024-03-04T07:37:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 -- start start 2024-03-04T07:37:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb46c1020d0 0x7fb46c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46c104a00 0x7fb46c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c1a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c10d260 con 0x7fb46c104a00 2024-03-04T07:37:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c10d0e0 con 0x7fb46c107330 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb473600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c10d3e0 con 0x7fb46c1020d0 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.388+0000 7fb463e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46c104a00 0x7fb46c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c1a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c1a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60204/0 (socket says 172.21.15.116:60204) 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 -- 172.21.15.116:0/1487499878 learned_addr learned my addr 172.21.15.116:0/1487499878 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb470c00640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb46c1020d0 0x7fb46c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 -- 172.21.15.116:0/1487499878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb46c1020d0 msgr2=0x7fb46c199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb46c1020d0 0x7fb46c199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 -- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46c104a00 msgr2=0x7fb46c199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46c104a00 0x7fb46c199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 -- 172.21.15.116:0/1487499878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c1a1690 con 0x7fb46c107330 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb463e00640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46c104a00 0x7fb46c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:37:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb471600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c1a1020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb464036040 tx=0x7fb4640337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb46a000640 1 -- 172.21.15.116:0/1487499878 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464043020 con 0x7fb46c107330 2024-03-04T07:37:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb46a000640 1 -- 172.21.15.116:0/1487499878 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb46402ca60 con 0x7fb46c107330 2024-03-04T07:37:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb46a000640 1 -- 172.21.15.116:0/1487499878 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464034690 con 0x7fb46c107330 2024-03-04T07:37:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.389+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c1a1920 con 0x7fb46c107330 2024-03-04T07:37:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.390+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c1a1e70 con 0x7fb46c107330 2024-03-04T07:37:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.391+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c108e20 con 0x7fb46c107330 2024-03-04T07:37:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.392+0000 7fb46a000640 1 -- 172.21.15.116:0/1487499878 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb464046020 con 0x7fb46c107330 2024-03-04T07:37:59.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.394+0000 7fb46a000640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4500a1d80 0x7fb4500a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.394+0000 7fb46a000640 1 -- 172.21.15.116:0/1487499878 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb4640f0690 con 0x7fb46c107330 2024-03-04T07:37:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.394+0000 7fb470c00640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4500a1d80 0x7fb4500a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.394+0000 7fb470c00640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4500a1d80 0x7fb4500a4240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009bd0 tx=0x7fb45c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:59.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.400+0000 7fb46a000640 1 -- 172.21.15.116:0/1487499878 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb46404d020 con 0x7fb46c107330 2024-03-04T07:37:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:59 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4288978655' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T07:37:59.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:37:59 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T07:37:59.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:59 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4288978655' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T07:37:59.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:37:59 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T07:37:59.536 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:59 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4288978655' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T07:37:59.536 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:37:59 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T07:37:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.624+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fb46c061a10 con 0x7fb4500a1d80 2024-03-04T07:37:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.646+0000 7fb46a000640 1 -- 172.21.15.116:0/1487499878 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb46c061a10 con 0x7fb4500a1d80 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4500a1d80 msgr2=0x7fb4500a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4500a1d80 0x7fb4500a4240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009bd0 tx=0x7fb45c005f90 comp rx=0 tx=0).stop 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 msgr2=0x7fb46c1a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c1a1020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb464036040 tx=0x7fb4640337b0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 shutdown_connections 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb46c107330 0x7fb46c1a1020 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46c104a00 0x7fb46c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4500a1d80 0x7fb4500a4240 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 --2- 172.21.15.116:0/1487499878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb46c1020d0 0x7fb46c199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 >> 172.21.15.116:0/1487499878 conn(0x7fb46c0fba80 msgr2=0x7fb46c101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 shutdown_connections 2024-03-04T07:37:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.649+0000 7fb473600640 1 -- 172.21.15.116:0/1487499878 wait complete. 2024-03-04T07:37:59.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-04T07:37:59.836 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:37:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b[67089]: 2024-03-04T07:37:59.534+0000 7f60eca84700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T07:37:59.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 -- 172.21.15.116:0/2837083921 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e541071b0 msgr2=0x7f8e54109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 --2- 172.21.15.116:0/2837083921 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e541071b0 0x7f8e54109600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009900 tx=0x7f8e4c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:37:59.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 -- 172.21.15.116:0/2837083921 shutdown_connections 2024-03-04T07:37:59.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 --2- 172.21.15.116:0/2837083921 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e541071b0 0x7f8e54109600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 --2- 172.21.15.116:0/2837083921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e54104880 0x7f8e54106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 --2- 172.21.15.116:0/2837083921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e54101f50 0x7f8e54104340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 -- 172.21.15.116:0/2837083921 >> 172.21.15.116:0/2837083921 conn(0x7f8e540fb940 msgr2=0x7f8e540fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:37:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 -- 172.21.15.116:0/2837083921 shutdown_connections 2024-03-04T07:37:59.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.851+0000 7f8e53200640 1 -- 172.21.15.116:0/2837083921 wait complete. 2024-03-04T07:37:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.852+0000 7f8e53200640 1 Processor -- start 2024-03-04T07:37:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e53200640 1 -- start start 2024-03-04T07:37:59.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e53200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e54101f50 0x7f8e54197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e53200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e54104880 0x7f8e54197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e53200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e541071b0 0x7f8e5419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e53200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410d160 con 0x7f8e541071b0 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e53200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410cfe0 con 0x7f8e54101f50 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e53200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410d2e0 con 0x7f8e54104880 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e54101f50 0x7f8e54197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e54101f50 0x7f8e54197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60238/0 (socket says 172.21.15.116:60238) 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 -- 172.21.15.116:0/2050282588 learned_addr learned my addr 172.21.15.116:0/2050282588 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51400640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e54104880 0x7f8e54197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e52800640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e541071b0 0x7f8e5419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 -- 172.21.15.116:0/2050282588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e54104880 msgr2=0x7f8e54197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e54104880 0x7f8e54197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 -- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e541071b0 msgr2=0x7f8e5419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e541071b0 0x7f8e5419ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.853+0000 7f8e51e00640 1 -- 172.21.15.116:0/2050282588 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5419f3a0 con 0x7f8e54101f50 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.854+0000 7f8e51e00640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e54101f50 0x7f8e54197780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009d50 tx=0x7f8e4400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.854+0000 7f8e4aa00640 1 -- 172.21.15.116:0/2050282588 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44003b70 con 0x7f8e54101f50 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.854+0000 7f8e4aa00640 1 -- 172.21.15.116:0/2050282588 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e44003d10 con 0x7f8e54101f50 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.854+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5419f690 con 0x7f8e54101f50 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.854+0000 7f8e4aa00640 1 -- 172.21.15.116:0/2050282588 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44010a20 con 0x7f8e54101f50 2024-03-04T07:37:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.854+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5419fc30 con 0x7f8e54101f50 2024-03-04T07:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.855+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e540fd430 con 0x7f8e54101f50 2024-03-04T07:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.857+0000 7f8e4aa00640 1 -- 172.21.15.116:0/2050282588 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8e44004000 con 0x7f8e54101f50 2024-03-04T07:37:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.858+0000 7f8e4aa00640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e380a1e00 0x7f8e380a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:37:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.858+0000 7f8e4aa00640 1 -- 172.21.15.116:0/2050282588 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8e440ce2f0 con 0x7f8e54101f50 2024-03-04T07:37:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.858+0000 7f8e51400640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e380a1e00 0x7f8e380a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:37:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.859+0000 7f8e51400640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e380a1e00 0x7f8e380a42c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e54198c90 tx=0x7f8e40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:37:59.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:37:59.862+0000 7f8e4aa00640 1 -- 172.21.15.116:0/2050282588 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8e4409b7d0 con 0x7f8e54101f50 2024-03-04T07:38:00.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:37:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:37:59.633+0000 7f51db600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T07:38:00.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:37:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:37:59.633+0000 7f51db600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T07:38:00.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:37:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:37:59.633+0000 7f51db600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T07:38:00.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:37:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:37:59.633+0000 7f51db600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T07:38:00.009 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:37:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:37:59.633+0000 7f51db600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T07:38:00.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.072+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e54061a10 con 0x7f8e380a1e00 2024-03-04T07:38:00.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.101+0000 7f8e4aa00640 1 -- 172.21.15.116:0/2050282588 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8e54061a10 con 0x7f8e380a1e00 2024-03-04T07:38:00.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e380a1e00 msgr2=0x7f8e380a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e380a1e00 0x7f8e380a42c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e54198c90 tx=0x7f8e40005f90 comp rx=0 tx=0).stop 2024-03-04T07:38:00.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e54101f50 msgr2=0x7f8e54197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e54101f50 0x7f8e54197780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009d50 tx=0x7f8e4400c6a0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 shutdown_connections 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e541071b0 0x7f8e5419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e380a1e00 0x7f8e380a42c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e54104880 0x7f8e54197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 --2- 172.21.15.116:0/2050282588 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e54101f50 0x7f8e54197780 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 >> 172.21.15.116:0/2050282588 conn(0x7f8e540fb940 msgr2=0x7f8e541028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 shutdown_connections 2024-03-04T07:38:00.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.104+0000 7f8e53200640 1 -- 172.21.15.116:0/2050282588 wait complete. 2024-03-04T07:38:00.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-04T07:38:00.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:00 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-04T07:38:00.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:00 smithi133 ceph-mon[36505]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-04T07:38:00.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:00 smithi133 ceph-mon[36505]: pgmap v316: 97 pgs: 8 creating+peering, 89 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-04T07:38:00.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:00 smithi133 ceph-mon[36505]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 -- 172.21.15.116:0/2956618276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c073fd0 msgr2=0x7f465c074430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 --2- 172.21.15.116:0/2956618276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c073fd0 0x7f465c074430 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f464c009900 tx=0x7f464c02bcd0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 -- 172.21.15.116:0/2956618276 shutdown_connections 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 --2- 172.21.15.116:0/2956618276 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c109470 0x7f465c10b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 --2- 172.21.15.116:0/2956618276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c073fd0 0x7f465c074430 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 --2- 172.21.15.116:0/2956618276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f465c0755b0 0x7f465c073a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 -- 172.21.15.116:0/2956618276 >> 172.21.15.116:0/2956618276 conn(0x7f465c0fcb70 msgr2=0x7f465c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 -- 172.21.15.116:0/2956618276 shutdown_connections 2024-03-04T07:38:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.344+0000 7f4664000640 1 -- 172.21.15.116:0/2956618276 wait complete. 2024-03-04T07:38:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.345+0000 7f4664000640 1 Processor -- start 2024-03-04T07:38:00.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.345+0000 7f4664000640 1 -- start start 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4664000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c073fd0 0x7f465c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4664000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f465c0755b0 0x7f465c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4664000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c109470 0x7f465c1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4664000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c10f440 con 0x7f465c109470 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4664000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c10f2c0 con 0x7f465c073fd0 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4664000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c10f5c0 con 0x7f465c0755b0 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c073fd0 0x7f465c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c073fd0 0x7f465c199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60270/0 (socket says 172.21.15.116:60270) 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4662000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c109470 0x7f465c1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4660c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f465c0755b0 0x7f465c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 -- 172.21.15.116:0/2981885737 learned_addr learned my addr 172.21.15.116:0/2981885737 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 -- 172.21.15.116:0/2981885737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f465c0755b0 msgr2=0x7f465c19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f465c0755b0 0x7f465c19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 -- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c109470 msgr2=0x7f465c1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c109470 0x7f465c1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.346+0000 7f4661600640 1 -- 172.21.15.116:0/2981885737 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c1a1720 con 0x7f465c073fd0 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.347+0000 7f4661600640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c073fd0 0x7f465c199b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f465000e9f0 tx=0x7f465000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.347+0000 7f465a000640 1 -- 172.21.15.116:0/2981885737 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465000cd80 con 0x7f465c073fd0 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.347+0000 7f465a000640 1 -- 172.21.15.116:0/2981885737 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4650010910 con 0x7f465c073fd0 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.347+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c1a1a10 con 0x7f465c073fd0 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.347+0000 7f465a000640 1 -- 172.21.15.116:0/2981885737 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650018990 con 0x7f465c073fd0 2024-03-04T07:38:00.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.347+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465c1a1f30 con 0x7f465c073fd0 2024-03-04T07:38:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.348+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c068810 con 0x7f465c073fd0 2024-03-04T07:38:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.350+0000 7f465a000640 1 -- 172.21.15.116:0/2981885737 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4650010ac0 con 0x7f465c073fd0 2024-03-04T07:38:00.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.351+0000 7f465a000640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46440a1d80 0x7f46440a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.351+0000 7f465a000640 1 -- 172.21.15.116:0/2981885737 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f46500cdfd0 con 0x7f465c073fd0 2024-03-04T07:38:00.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.352+0000 7f4660c00640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46440a1d80 0x7f46440a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.352+0000 7f4660c00640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46440a1d80 0x7f46440a4240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f464c02c850 tx=0x7f464c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:00.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.360+0000 7f465a000640 1 -- 172.21.15.116:0/2981885737 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f465009ae60 con 0x7f465c073fd0 2024-03-04T07:38:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:00 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-04T07:38:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:00 smithi116 ceph-mon[32150]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-04T07:38:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:00 smithi116 ceph-mon[32150]: pgmap v316: 97 pgs: 8 creating+peering, 89 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-04T07:38:00.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:00 smithi116 ceph-mon[32150]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:00.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:00 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-04T07:38:00.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:00 smithi152 ceph-mon[37154]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-04T07:38:00.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:00 smithi152 ceph-mon[37154]: pgmap v316: 97 pgs: 8 creating+peering, 89 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-04T07:38:00.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:00 smithi152 ceph-mon[37154]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:00.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.585+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f465c061aa0 con 0x7f46440a1d80 2024-03-04T07:38:00.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.595+0000 7f465a000640 1 -- 172.21.15.116:0/2981885737 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f465c061aa0 con 0x7f46440a1d80 2024-03-04T07:38:00.595 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_0/6bdb905f-3166-4135-904f-544ca0df6d07 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46440a1d80 msgr2=0x7f46440a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46440a1d80 0x7f46440a4240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f464c02c850 tx=0x7f464c000bc0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c073fd0 msgr2=0x7f465c199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c073fd0 0x7f465c199b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f465000e9f0 tx=0x7f465000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 shutdown_connections 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f465c109470 0x7f465c1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46440a1d80 0x7f46440a4240 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f465c0755b0 0x7f465c19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 --2- 172.21.15.116:0/2981885737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f465c073fd0 0x7f465c199b00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 >> 172.21.15.116:0/2981885737 conn(0x7f465c0fcb70 msgr2=0x7f465c105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:00.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 shutdown_connections 2024-03-04T07:38:00.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.597+0000 7f4664000640 1 -- 172.21.15.116:0/2981885737 wait complete. 2024-03-04T07:38:00.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-04T07:38:00.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 -- 172.21.15.116:0/3105456520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3484077b70 msgr2=0x7f3484078410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 --2- 172.21.15.116:0/3105456520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3484077b70 0x7f3484078410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f347c009ca0 tx=0x7f347c02c100 comp rx=0 tx=0).stop 2024-03-04T07:38:00.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 -- 172.21.15.116:0/3105456520 shutdown_connections 2024-03-04T07:38:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 --2- 172.21.15.116:0/3105456520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3484077b70 0x7f3484078410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 --2- 172.21.15.116:0/3105456520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34840771d0 0x7f3484077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 --2- 172.21.15.116:0/3105456520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3484075f90 0x7f34840763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 -- 172.21.15.116:0/3105456520 >> 172.21.15.116:0/3105456520 conn(0x7f34840fed80 msgr2=0x7f34841011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 -- 172.21.15.116:0/3105456520 shutdown_connections 2024-03-04T07:38:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.818+0000 7f348be00640 1 -- 172.21.15.116:0/3105456520 wait complete. 2024-03-04T07:38:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.819+0000 7f348be00640 1 Processor -- start 2024-03-04T07:38:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.819+0000 7f348be00640 1 -- start start 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f348be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3484075f90 0x7f348419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f348be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34840771d0 0x7f348419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f348be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3484077b70 0x7f34841a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f348be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34841138a0 con 0x7f3484077b70 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f348be00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3484113720 con 0x7f34840771d0 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f348be00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3484113a20 con 0x7f3484075f90 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34840771d0 0x7f348419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3489e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3484077b70 0x7f34841a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34840771d0 0x7f348419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60290/0 (socket says 172.21.15.116:60290) 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 -- 172.21.15.116:0/3916225733 learned_addr learned my addr 172.21.15.116:0/3916225733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3489400640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3484075f90 0x7f348419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 -- 172.21.15.116:0/3916225733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3484075f90 msgr2=0x7f348419dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3484075f90 0x7f348419dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 -- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3484077b70 msgr2=0x7f34841a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3484077b70 0x7f34841a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.820+0000 7f3488a00640 1 -- 172.21.15.116:0/3916225733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34841a5ae0 con 0x7f34840771d0 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.821+0000 7f3488a00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34840771d0 0x7f348419e400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f347000e9f0 tx=0x7f347000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.821+0000 7f3482000640 1 -- 172.21.15.116:0/3916225733 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347000cd80 con 0x7f34840771d0 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.821+0000 7f3482000640 1 -- 172.21.15.116:0/3916225733 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3470010910 con 0x7f34840771d0 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.821+0000 7f3482000640 1 -- 172.21.15.116:0/3916225733 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470018990 con 0x7f34840771d0 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.821+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34841a5dd0 con 0x7f34840771d0 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.821+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34841a6240 con 0x7f34840771d0 2024-03-04T07:38:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.822+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3484110180 con 0x7f34840771d0 2024-03-04T07:38:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.825+0000 7f3482000640 1 -- 172.21.15.116:0/3916225733 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3470010ac0 con 0x7f34840771d0 2024-03-04T07:38:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.826+0000 7f3482000640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f346c0a1e00 0x7f346c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.826+0000 7f3482000640 1 -- 172.21.15.116:0/3916225733 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f34700ce150 con 0x7f34840771d0 2024-03-04T07:38:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.826+0000 7f3489400640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f346c0a1e00 0x7f346c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.827+0000 7f3489400640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f346c0a1e00 0x7f346c0a42c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3478009bd0 tx=0x7f3478005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:00.827+0000 7f3482000640 1 -- 172.21.15.116:0/3916225733 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f347009afe0 con 0x7f34840771d0 2024-03-04T07:38:01.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.056+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f3484061aa0 con 0x7f346c0a1e00 2024-03-04T07:38:01.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.086+0000 7f3482000640 1 -- 172.21.15.116:0/3916225733 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3484061aa0 con 0x7f346c0a1e00 2024-03-04T07:38:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f346c0a1e00 msgr2=0x7f346c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f346c0a1e00 0x7f346c0a42c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3478009bd0 tx=0x7f3478005f90 comp rx=0 tx=0).stop 2024-03-04T07:38:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34840771d0 msgr2=0x7f348419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34840771d0 0x7f348419e400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f347000e9f0 tx=0x7f347000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 shutdown_connections 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3484077b70 0x7f34841a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34840771d0 0x7f348419e400 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f346c0a1e00 0x7f346c0a42c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 --2- 172.21.15.116:0/3916225733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3484075f90 0x7f348419dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.089+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 >> 172.21.15.116:0/3916225733 conn(0x7f34840fed80 msgr2=0x7f3484100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.090+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 shutdown_connections 2024-03-04T07:38:01.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.090+0000 7f348be00640 1 -- 172.21.15.116:0/3916225733 wait complete. 2024-03-04T07:38:01.108 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 -- 172.21.15.116:0/2818153744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 msgr2=0x7fbb98104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2818153744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb98104e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009900 tx=0x7fbb8402bc50 comp rx=0 tx=0).stop 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 -- 172.21.15.116:0/2818153744 shutdown_connections 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2818153744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 0x7fbb9810bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2818153744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb98104e60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2818153744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb981037f0 0x7fbb98103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 -- 172.21.15.116:0/2818153744 >> 172.21.15.116:0/2818153744 conn(0x7fbb980fed80 msgr2=0x7fbb981011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 -- 172.21.15.116:0/2818153744 shutdown_connections 2024-03-04T07:38:01.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.319+0000 7fbb9f800640 1 -- 172.21.15.116:0/2818153744 wait complete. 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.320+0000 7fbb9f800640 1 Processor -- start 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.320+0000 7fbb9f800640 1 -- start start 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9f800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb981037f0 0x7fbb98199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9f800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb9819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9f800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 0x7fbb981a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9f800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9810f4c0 con 0x7fbb981049e0 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9f800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9810f340 con 0x7fbb981053a0 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9f800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9810f640 con 0x7fbb981037f0 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb97e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb9819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9d800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 0x7fbb981a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb97e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb9819a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36988/0 (socket says 172.21.15.116:36988) 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9d800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 0x7fbb981a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60320/0 (socket says 172.21.15.116:60320) 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb97e00640 1 -- 172.21.15.116:0/2569737463 learned_addr learned my addr 172.21.15.116:0/2569737463 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:01.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9ce00640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb981037f0 0x7fbb98199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9d800640 1 -- 172.21.15.116:0/2569737463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb981037f0 msgr2=0x7fbb98199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9d800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb981037f0 0x7fbb98199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9d800640 1 -- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 msgr2=0x7fbb9819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9d800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb9819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb9d800640 1 -- 172.21.15.116:0/2569737463 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb981a1760 con 0x7fbb981053a0 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.321+0000 7fbb97e00640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb9819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.322+0000 7fbb9d800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 0x7fbb981a10f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000ec50 tx=0x7fbb9000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.322+0000 7fbb95600640 1 -- 172.21.15.116:0/2569737463 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90003b70 con 0x7fbb981053a0 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.322+0000 7fbb95600640 1 -- 172.21.15.116:0/2569737463 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb90003d10 con 0x7fbb981053a0 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.322+0000 7fbb95600640 1 -- 172.21.15.116:0/2569737463 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90010890 con 0x7fbb981053a0 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.322+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb981a1a50 con 0x7fbb981053a0 2024-03-04T07:38:01.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.322+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb981a1f50 con 0x7fbb981053a0 2024-03-04T07:38:01.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.323+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9810ab30 con 0x7fbb981053a0 2024-03-04T07:38:01.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.325+0000 7fbb95600640 1 -- 172.21.15.116:0/2569737463 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fbb90003fc0 con 0x7fbb981053a0 2024-03-04T07:38:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.326+0000 7fbb95600640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbb7c0a1e00 0x7fbb7c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.327+0000 7fbb95600640 1 -- 172.21.15.116:0/2569737463 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbb900ce1e0 con 0x7fbb981053a0 2024-03-04T07:38:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.327+0000 7fbb9ce00640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbb7c0a1e00 0x7fbb7c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.327+0000 7fbb9ce00640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbb7c0a1e00 0x7fbb7c0a42c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb88005fd0 tx=0x7fbb8800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.330+0000 7fbb95600640 1 -- 172.21.15.116:0/2569737463 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbb9009b6c0 con 0x7fbb981053a0 2024-03-04T07:38:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:01 smithi116 ceph-mon[32150]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:01.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:01 smithi116 ceph-mon[32150]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:01.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:01 smithi152 ceph-mon[37154]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:01.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:01 smithi152 ceph-mon[37154]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:01.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.560+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb98061a10 con 0x7fbb7c0a1e00 2024-03-04T07:38:01.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.566+0000 7fbb95600640 1 -- 172.21.15.116:0/2569737463 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fbb98061a10 con 0x7fbb7c0a1e00 2024-03-04T07:38:01.566 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_1/9d10f24b-566a-421f-bbda-ac3ac0eb7a11 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbb7c0a1e00 msgr2=0x7fbb7c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbb7c0a1e00 0x7fbb7c0a42c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb88005fd0 tx=0x7fbb8800b390 comp rx=0 tx=0).stop 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 msgr2=0x7fbb981a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 0x7fbb981a10f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000ec50 tx=0x7fbb9000c6a0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 shutdown_connections 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbb981053a0 0x7fbb981a10f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb981049e0 0x7fbb9819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbb7c0a1e00 0x7fbb7c0a42c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 --2- 172.21.15.116:0/2569737463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb981037f0 0x7fbb98199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 >> 172.21.15.116:0/2569737463 conn(0x7fbb980fed80 msgr2=0x7fbb98107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:01.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 shutdown_connections 2024-03-04T07:38:01.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.569+0000 7fbb9f800640 1 -- 172.21.15.116:0/2569737463 wait complete. 2024-03-04T07:38:01.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:01 smithi133 ceph-mon[36505]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:01.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:01 smithi133 ceph-mon[36505]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:01.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.798+0000 7ff801000640 1 -- 172.21.15.116:0/177820302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc0771d0 msgr2=0x7ff7fc077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.798+0000 7ff801000640 1 --2- 172.21.15.116:0/177820302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc0771d0 0x7ff7fc077630 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009900 tx=0x7ff7e802bc30 comp rx=0 tx=0).stop 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.799+0000 7ff801000640 1 -- 172.21.15.116:0/177820302 shutdown_connections 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.799+0000 7ff801000640 1 --2- 172.21.15.116:0/177820302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7fc077b70 0x7ff7fc078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.799+0000 7ff801000640 1 --2- 172.21.15.116:0/177820302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc0771d0 0x7ff7fc077630 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.799+0000 7ff801000640 1 --2- 172.21.15.116:0/177820302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7fc075f90 0x7ff7fc0763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.799+0000 7ff801000640 1 -- 172.21.15.116:0/177820302 >> 172.21.15.116:0/177820302 conn(0x7ff7fc0fed80 msgr2=0x7ff7fc1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.799+0000 7ff801000640 1 -- 172.21.15.116:0/177820302 shutdown_connections 2024-03-04T07:38:01.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.799+0000 7ff801000640 1 -- 172.21.15.116:0/177820302 wait complete. 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 Processor -- start 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 -- start start 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc075f90 0x7ff7fc19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7fc0771d0 0x7ff7fc19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7fc077b70 0x7ff7fc1a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc1138e0 con 0x7ff7fc077b70 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc113760 con 0x7ff7fc075f90 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.800+0000 7ff801000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc113a60 con 0x7ff7fc0771d0 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc075f90 0x7ff7fc19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc075f90 0x7ff7fc19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60344/0 (socket says 172.21.15.116:60344) 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 -- 172.21.15.116:0/3122518360 learned_addr learned my addr 172.21.15.116:0/3122518360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7fa800640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7fc077b70 0x7ff7fc1a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9400640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7fc0771d0 0x7ff7fc19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 -- 172.21.15.116:0/3122518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7fc0771d0 msgr2=0x7ff7fc19e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7fc0771d0 0x7ff7fc19e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 -- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7fc077b70 msgr2=0x7ff7fc1a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7fc077b70 0x7ff7fc1a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 -- 172.21.15.116:0/3122518360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc1a5b30 con 0x7ff7fc075f90 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f9e00640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc075f90 0x7ff7fc19df10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00d950 tx=0x7ff7ec00dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.801+0000 7ff7f2a00640 1 -- 172.21.15.116:0/3122518360 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec017070 con 0x7ff7fc075f90 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.802+0000 7ff7f2a00640 1 -- 172.21.15.116:0/3122518360 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7ec00de80 con 0x7ff7fc075f90 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.802+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc1a5e20 con 0x7ff7fc075f90 2024-03-04T07:38:01.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.802+0000 7ff7f2a00640 1 -- 172.21.15.116:0/3122518360 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec00f620 con 0x7ff7fc075f90 2024-03-04T07:38:01.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.802+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc1a6290 con 0x7ff7fc075f90 2024-03-04T07:38:01.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.803+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc1102b0 con 0x7ff7fc075f90 2024-03-04T07:38:01.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.805+0000 7ff7f2a00640 1 -- 172.21.15.116:0/3122518360 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff7ec009c30 con 0x7ff7fc075f90 2024-03-04T07:38:01.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.806+0000 7ff7f2a00640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7e00a1e00 0x7ff7e00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:01.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.806+0000 7ff7f2a00640 1 -- 172.21.15.116:0/3122518360 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff7ec0cd7a0 con 0x7ff7fc075f90 2024-03-04T07:38:01.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.806+0000 7ff7f9400640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7e00a1e00 0x7ff7e00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:01.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.807+0000 7ff7f9400640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7e00a1e00 0x7ff7e00a42c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff7e802c240 tx=0x7ff7e802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:01.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:01.808+0000 7ff7f2a00640 1 -- 172.21.15.116:0/3122518360 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff7ec09ada0 con 0x7ff7fc075f90 2024-03-04T07:38:02.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.043+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7fc02d260 con 0x7ff7fc075f90 2024-03-04T07:38:02.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.044+0000 7ff7f2a00640 1 -- 172.21.15.116:0/3122518360 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7ff7ec09a6a0 con 0x7ff7fc075f90 2024-03-04T07:38:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:38:02.044 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:38:02.044 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:38:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.046+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7e00a1e00 msgr2=0x7ff7e00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7e00a1e00 0x7ff7e00a42c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff7e802c240 tx=0x7ff7e802df90 comp rx=0 tx=0).stop 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc075f90 msgr2=0x7ff7fc19df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc075f90 0x7ff7fc19df10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00d950 tx=0x7ff7ec00dd50 comp rx=0 tx=0).stop 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 shutdown_connections 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7fc077b70 0x7ff7fc1a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7e00a1e00 0x7ff7e00a42c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7fc0771d0 0x7ff7fc19e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 --2- 172.21.15.116:0/3122518360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7fc075f90 0x7ff7fc19df10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 >> 172.21.15.116:0/3122518360 conn(0x7ff7fc0fed80 msgr2=0x7ff7fc100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 shutdown_connections 2024-03-04T07:38:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.047+0000 7ff801000640 1 -- 172.21.15.116:0/3122518360 wait complete. 2024-03-04T07:38:02.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:38:02.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.287+0000 7fc5da400640 1 -- 172.21.15.116:0/3913746819 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 msgr2=0x7fc5d4107e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.287+0000 7fc5da400640 1 --2- 172.21.15.116:0/3913746819 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d4107e50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009900 tx=0x7fc5c802bc30 comp rx=0 tx=0).stop 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.287+0000 7fc5da400640 1 -- 172.21.15.116:0/3913746819 shutdown_connections 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.287+0000 7fc5da400640 1 --2- 172.21.15.116:0/3913746819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d41095e0 0x7fc5d410fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.287+0000 7fc5da400640 1 --2- 172.21.15.116:0/3913746819 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d4108c20 0x7fc5d41090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.287+0000 7fc5da400640 1 --2- 172.21.15.116:0/3913746819 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d4107e50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.287+0000 7fc5da400640 1 -- 172.21.15.116:0/3913746819 >> 172.21.15.116:0/3913746819 conn(0x7fc5d40769e0 msgr2=0x7fc5d4078e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.288+0000 7fc5da400640 1 -- 172.21.15.116:0/3913746819 shutdown_connections 2024-03-04T07:38:02.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.288+0000 7fc5da400640 1 -- 172.21.15.116:0/3913746819 wait complete. 2024-03-04T07:38:02.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 Processor -- start 2024-03-04T07:38:02.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 -- start start 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d4108c20 0x7fc5d419e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d41095e0 0x7fc5d41a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d41136a0 con 0x7fc5d4108c20 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4113520 con 0x7fc5d4107a30 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5da400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4113820 con 0x7fc5d41095e0 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5d9000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5d9000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d419dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60358/0 (socket says 172.21.15.116:60358) 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5d9000640 1 -- 172.21.15.116:0/1774254336 learned_addr learned my addr 172.21.15.116:0/1774254336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.289+0000 7fc5d3e00640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d4108c20 0x7fc5d419e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d9a00640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d41095e0 0x7fc5d41a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d9000640 1 -- 172.21.15.116:0/1774254336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d41095e0 msgr2=0x7fc5d41a52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d9000640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d41095e0 0x7fc5d41a52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d9000640 1 -- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d4108c20 msgr2=0x7fc5d419e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d9000640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d4108c20 0x7fc5d419e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d9000640 1 -- 172.21.15.116:0/1774254336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d41a5960 con 0x7fc5d4107a30 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d9000640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d419dd90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8000c00 tx=0x7fc5c802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d1600640 1 -- 172.21.15.116:0/1774254336 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8041070 con 0x7fc5d4107a30 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d1600640 1 -- 172.21.15.116:0/1774254336 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5c8004450 con 0x7fc5d4107a30 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d41a5bf0 con 0x7fc5d4107a30 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.290+0000 7fc5d1600640 1 -- 172.21.15.116:0/1774254336 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8034440 con 0x7fc5d4107a30 2024-03-04T07:38:02.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.291+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d41a6130 con 0x7fc5d4107a30 2024-03-04T07:38:02.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.292+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d410ed50 con 0x7fc5d4107a30 2024-03-04T07:38:02.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.293+0000 7fc5d1600640 1 -- 172.21.15.116:0/1774254336 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fc5c8044070 con 0x7fc5d4107a30 2024-03-04T07:38:02.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.295+0000 7fc5d1600640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc5bc0a1e90 0x7fc5bc0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.295+0000 7fc5d1600640 1 -- 172.21.15.116:0/1774254336 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc5c80ef790 con 0x7fc5d4107a30 2024-03-04T07:38:02.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.295+0000 7fc5d3e00640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc5bc0a1e90 0x7fc5bc0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.296+0000 7fc5d3e00640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc5bc0a1e90 0x7fc5bc0a4350 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc5d419f250 tx=0x7fc5c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:02.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.299+0000 7fc5d1600640 1 -- 172.21.15.116:0/1774254336 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc5c80bc660 con 0x7fc5d4107a30 2024-03-04T07:38:02.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.499+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5d402d210 con 0x7fc5d4107a30 2024-03-04T07:38:02.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.501+0000 7fc5d1600640 1 -- 172.21.15.116:0/1774254336 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22302 (secure 0 0 0) 0x7fc5c80c0700 con 0x7fc5d4107a30 2024-03-04T07:38:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:38:02.501 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":85,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:37:56.091002+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc5bc0a1e90 msgr2=0x7fc5bc0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc5bc0a1e90 0x7fc5bc0a4350 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc5d419f250 tx=0x7fc5c0005f90 comp rx=0 tx=0).stop 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 msgr2=0x7fc5d419dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d419dd90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8000c00 tx=0x7fc5c802c640 comp rx=0 tx=0).stop 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 shutdown_connections 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc5bc0a1e90 0x7fc5bc0a4350 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc5d41095e0 0x7fc5d41a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d4108c20 0x7fc5d419e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 --2- 172.21.15.116:0/1774254336 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc5d4107a30 0x7fc5d419dd90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 >> 172.21.15.116:0/1774254336 conn(0x7fc5d40769e0 msgr2=0x7fc5d410bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 shutdown_connections 2024-03-04T07:38:02.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.504+0000 7fc5da400640 1 -- 172.21.15.116:0/1774254336 wait complete. 2024-03-04T07:38:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:02 smithi116 ceph-mon[32150]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:02 smithi116 ceph-mon[32150]: pgmap v317: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 695 B/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-04T07:38:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:02 smithi116 ceph-mon[32150]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:02.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:02 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3122518360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:38:02.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:02 smithi152 ceph-mon[37154]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:02.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:02 smithi152 ceph-mon[37154]: pgmap v317: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 695 B/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-04T07:38:02.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:02 smithi152 ceph-mon[37154]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:02.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:02 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3122518360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:38:02.525 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T07:38:02.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T07:38:02.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:02 smithi133 ceph-mon[36505]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:02 smithi133 ceph-mon[36505]: pgmap v317: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 695 B/s rd, 1.9 KiB/s wr, 6 op/s 2024-03-04T07:38:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:02 smithi133 ceph-mon[36505]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:38:02.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:02 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3122518360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 -- 172.21.15.116:0/3842366512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 msgr2=0x7fccfc10bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 --2- 172.21.15.116:0/3842366512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 0x7fccfc10bc50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fccec009ca0 tx=0x7fccec02c100 comp rx=0 tx=0).stop 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 -- 172.21.15.116:0/3842366512 shutdown_connections 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 --2- 172.21.15.116:0/3842366512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 0x7fccfc10bc50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 --2- 172.21.15.116:0/3842366512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccfc104a00 0x7fccfc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 --2- 172.21.15.116:0/3842366512 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 0x7fccfc103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 -- 172.21.15.116:0/3842366512 >> 172.21.15.116:0/3842366512 conn(0x7fccfc0fed80 msgr2=0x7fccfc1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 -- 172.21.15.116:0/3842366512 shutdown_connections 2024-03-04T07:38:02.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.732+0000 7fcd02200640 1 -- 172.21.15.116:0/3842366512 wait complete. 2024-03-04T07:38:02.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 Processor -- start 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 -- start start 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 0x7fccfc199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccfc104a00 0x7fccfc199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 0x7fccfc1a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc10f470 con 0x7fccfc104a00 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc10f2f0 con 0x7fccfc103810 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.733+0000 7fcd02200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc10f5f0 con 0x7fccfc1053c0 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfaa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccfc104a00 0x7fccfc199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 0x7fccfc199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 0x7fccfc199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60374/0 (socket says 172.21.15.116:60374) 2024-03-04T07:38:02.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfb400640 1 -- 172.21.15.116:0/4141967 learned_addr learned my addr 172.21.15.116:0/4141967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfbe00640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 0x7fccfc1a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfaa00640 1 -- 172.21.15.116:0/4141967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 msgr2=0x7fccfc1a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfaa00640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 0x7fccfc1a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfaa00640 1 -- 172.21.15.116:0/4141967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 msgr2=0x7fccfc199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfaa00640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 0x7fccfc199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfaa00640 1 -- 172.21.15.116:0/4141967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc1a16b0 con 0x7fccfc104a00 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfbe00640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 0x7fccfc1a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfb400640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 0x7fccfc199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.734+0000 7fccfaa00640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccfc104a00 0x7fccfc199fd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e9f0 tx=0x7fcce400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.735+0000 7fccf3e00640 1 -- 172.21.15.116:0/4141967 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce400ccc0 con 0x7fccfc104a00 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.735+0000 7fccf3e00640 1 -- 172.21.15.116:0/4141967 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcce4010910 con 0x7fccfc104a00 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.735+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccfc1a19a0 con 0x7fccfc104a00 2024-03-04T07:38:02.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.735+0000 7fccf3e00640 1 -- 172.21.15.116:0/4141967 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4018960 con 0x7fccfc104a00 2024-03-04T07:38:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.735+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc1a1f40 con 0x7fccfc104a00 2024-03-04T07:38:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.736+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc0683c0 con 0x7fccfc104a00 2024-03-04T07:38:02.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.738+0000 7fccf3e00640 1 -- 172.21.15.116:0/4141967 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fcce4010ac0 con 0x7fccfc104a00 2024-03-04T07:38:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.739+0000 7fccf3e00640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcce00a1e90 0x7fcce00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.739+0000 7fccf3e00640 1 -- 172.21.15.116:0/4141967 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcce40ce260 con 0x7fccfc104a00 2024-03-04T07:38:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.739+0000 7fccfb400640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcce00a1e90 0x7fcce00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:02.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.740+0000 7fccfb400640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcce00a1e90 0x7fcce00a4350 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fccf40058c0 tx=0x7fccf4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:02.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.743+0000 7fccf3e00640 1 -- 172.21.15.116:0/4141967 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcce409b5e0 con 0x7fccfc104a00 2024-03-04T07:38:02.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.937+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccfc061a10 con 0x7fcce00a1e90 2024-03-04T07:38:02.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.963+0000 7fccf3e00640 1 -- 172.21.15.116:0/4141967 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258812 (secure 0 0 0) 0x7fccfc061a10 con 0x7fcce00a1e90 2024-03-04T07:38:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:38:02.969 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcce00a1e90 msgr2=0x7fcce00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcce00a1e90 0x7fcce00a4350 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fccf40058c0 tx=0x7fccf4005f90 comp rx=0 tx=0).stop 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccfc104a00 msgr2=0x7fccfc199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccfc104a00 0x7fccfc199fd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e9f0 tx=0x7fcce400eeb0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 shutdown_connections 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcce00a1e90 0x7fcce00a4350 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fccfc1053c0 0x7fccfc1a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccfc104a00 0x7fccfc199fd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 --2- 172.21.15.116:0/4141967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc103810 0x7fccfc199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:02.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 >> 172.21.15.116:0/4141967 conn(0x7fccfc0fed80 msgr2=0x7fccfc107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 shutdown_connections 2024-03-04T07:38:02.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:02.969+0000 7fcd02200640 1 -- 172.21.15.116:0/4141967 wait complete. 2024-03-04T07:38:02.989 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-03-04T07:38:01.466562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":131,"num_read_kb":114,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360168,"kb_used_data":10932,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581656,"statfs":{"total":1151940427776,"available":1151571615744,"internally_reserved":0,"allocated":11194368,"data_stored":7032016,"data_compressed":107094,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.361333"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301431+0000","last_change":"2024-03-04T07:37:55.137540+0000","last_active":"2024-03-04T07:37:56.301431+0000","last_peered":"2024-03-04T07:37:56.301431+0000","last_clean":"2024-03-04T07:37:56.301431+0000","last_became_active":"2024-03-04T07:37:55.137183+0000","last_became_peered":"2024-03-04T07:37:55.137183+0000","last_unstale":"2024-03-04T07:37:56.301431+0000","last_undegraded":"2024-03-04T07:37:56.301431+0000","last_fullsized":"2024-03-04T07:37:56.301431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:15.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301829+0000","last_change":"2024-03-04T07:37:55.136049+0000","last_active":"2024-03-04T07:37:56.301829+0000","last_peered":"2024-03-04T07:37:56.301829+0000","last_clean":"2024-03-04T07:37:56.301829+0000","last_became_active":"2024-03-04T07:37:55.132153+0000","last_became_peered":"2024-03-04T07:37:55.132153+0000","last_unstale":"2024-03-04T07:37:56.301829+0000","last_undegraded":"2024-03-04T07:37:56.301829+0000","last_fullsized":"2024-03-04T07:37:56.301829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:52.426892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302297+0000","last_change":"2024-03-04T07:37:55.133571+0000","last_active":"2024-03-04T07:37:56.302297+0000","last_peered":"2024-03-04T07:37:56.302297+0000","last_clean":"2024-03-04T07:37:56.302297+0000","last_became_active":"2024-03-04T07:37:55.133214+0000","last_became_peered":"2024-03-04T07:37:55.133214+0000","last_unstale":"2024-03-04T07:37:56.302297+0000","last_undegraded":"2024-03-04T07:37:56.302297+0000","last_fullsized":"2024-03-04T07:37:56.302297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:17.924826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300967+0000","last_change":"2024-03-04T07:37:55.118146+0000","last_active":"2024-03-04T07:37:56.300967+0000","last_peered":"2024-03-04T07:37:56.300967+0000","last_clean":"2024-03-04T07:37:56.300967+0000","last_became_active":"2024-03-04T07:37:55.117687+0000","last_became_peered":"2024-03-04T07:37:55.117687+0000","last_unstale":"2024-03-04T07:37:56.300967+0000","last_undegraded":"2024-03-04T07:37:56.300967+0000","last_fullsized":"2024-03-04T07:37:56.300967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:34:35.343400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152180+0000","last_change":"2024-03-04T07:37:55.135943+0000","last_active":"2024-03-04T07:37:56.152180+0000","last_peered":"2024-03-04T07:37:56.152180+0000","last_clean":"2024-03-04T07:37:56.152180+0000","last_became_active":"2024-03-04T07:37:55.135549+0000","last_became_peered":"2024-03-04T07:37:55.135549+0000","last_unstale":"2024-03-04T07:37:56.152180+0000","last_undegraded":"2024-03-04T07:37:56.152180+0000","last_fullsized":"2024-03-04T07:37:56.152180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:35:53.953771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096012+0000","last_change":"2024-03-04T07:37:55.116949+0000","last_active":"2024-03-04T07:37:56.096012+0000","last_peered":"2024-03-04T07:37:56.096012+0000","last_clean":"2024-03-04T07:37:56.096012+0000","last_became_active":"2024-03-04T07:37:55.116620+0000","last_became_peered":"2024-03-04T07:37:55.116620+0000","last_unstale":"2024-03-04T07:37:56.096012+0000","last_undegraded":"2024-03-04T07:37:56.096012+0000","last_fullsized":"2024-03-04T07:37:56.096012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:39.655217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336783+0000","last_change":"2024-03-04T07:37:55.100975+0000","last_active":"2024-03-04T07:37:56.336783+0000","last_peered":"2024-03-04T07:37:56.336783+0000","last_clean":"2024-03-04T07:37:56.336783+0000","last_became_active":"2024-03-04T07:37:55.100636+0000","last_became_peered":"2024-03-04T07:37:55.100636+0000","last_unstale":"2024-03-04T07:37:56.336783+0000","last_undegraded":"2024-03-04T07:37:56.336783+0000","last_fullsized":"2024-03-04T07:37:56.336783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:29.593763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152558+0000","last_change":"2024-03-04T07:37:55.131895+0000","last_active":"2024-03-04T07:37:56.152558+0000","last_peered":"2024-03-04T07:37:56.152558+0000","last_clean":"2024-03-04T07:37:56.152558+0000","last_became_active":"2024-03-04T07:37:55.129682+0000","last_became_peered":"2024-03-04T07:37:55.129682+0000","last_unstale":"2024-03-04T07:37:56.152558+0000","last_undegraded":"2024-03-04T07:37:56.152558+0000","last_fullsized":"2024-03-04T07:37:56.152558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:54.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300914+0000","last_change":"2024-03-04T07:37:55.114475+0000","last_active":"2024-03-04T07:37:56.300914+0000","last_peered":"2024-03-04T07:37:56.300914+0000","last_clean":"2024-03-04T07:37:56.300914+0000","last_became_active":"2024-03-04T07:37:55.114068+0000","last_became_peered":"2024-03-04T07:37:55.114068+0000","last_unstale":"2024-03-04T07:37:56.300914+0000","last_undegraded":"2024-03-04T07:37:56.300914+0000","last_fullsized":"2024-03-04T07:37:56.300914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:21.466143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096584+0000","last_change":"2024-03-04T07:37:55.115660+0000","last_active":"2024-03-04T07:37:56.096584+0000","last_peered":"2024-03-04T07:37:56.096584+0000","last_clean":"2024-03-04T07:37:56.096584+0000","last_became_active":"2024-03-04T07:37:55.115324+0000","last_became_peered":"2024-03-04T07:37:55.115324+0000","last_unstale":"2024-03-04T07:37:56.096584+0000","last_undegraded":"2024-03-04T07:37:56.096584+0000","last_fullsized":"2024-03-04T07:37:56.096584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:42.119728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337102+0000","last_change":"2024-03-04T07:37:55.119834+0000","last_active":"2024-03-04T07:37:56.337102+0000","last_peered":"2024-03-04T07:37:56.337102+0000","last_clean":"2024-03-04T07:37:56.337102+0000","last_became_active":"2024-03-04T07:37:55.119467+0000","last_became_peered":"2024-03-04T07:37:55.119467+0000","last_unstale":"2024-03-04T07:37:56.337102+0000","last_undegraded":"2024-03-04T07:37:56.337102+0000","last_fullsized":"2024-03-04T07:37:56.337102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:49:19.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300908+0000","last_change":"2024-03-04T07:37:55.118154+0000","last_active":"2024-03-04T07:37:56.300908+0000","last_peered":"2024-03-04T07:37:56.300908+0000","last_clean":"2024-03-04T07:37:56.300908+0000","last_became_active":"2024-03-04T07:37:55.117809+0000","last_became_peered":"2024-03-04T07:37:55.117809+0000","last_unstale":"2024-03-04T07:37:56.300908+0000","last_undegraded":"2024-03-04T07:37:56.300908+0000","last_fullsized":"2024-03-04T07:37:56.300908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:50.195946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302928+0000","last_change":"2024-03-04T07:37:55.123325+0000","last_active":"2024-03-04T07:37:56.302928+0000","last_peered":"2024-03-04T07:37:56.302928+0000","last_clean":"2024-03-04T07:37:56.302928+0000","last_became_active":"2024-03-04T07:37:55.122848+0000","last_became_peered":"2024-03-04T07:37:55.122848+0000","last_unstale":"2024-03-04T07:37:56.302928+0000","last_undegraded":"2024-03-04T07:37:56.302928+0000","last_fullsized":"2024-03-04T07:37:56.302928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:18.842011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302409+0000","last_change":"2024-03-04T07:37:55.132447+0000","last_active":"2024-03-04T07:37:56.302409+0000","last_peered":"2024-03-04T07:37:56.302409+0000","last_clean":"2024-03-04T07:37:56.302409+0000","last_became_active":"2024-03-04T07:37:55.131512+0000","last_became_peered":"2024-03-04T07:37:55.131512+0000","last_unstale":"2024-03-04T07:37:56.302409+0000","last_undegraded":"2024-03-04T07:37:56.302409+0000","last_fullsized":"2024-03-04T07:37:56.302409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:23.960029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301583+0000","last_change":"2024-03-04T07:37:55.131715+0000","last_active":"2024-03-04T07:37:56.301583+0000","last_peered":"2024-03-04T07:37:56.301583+0000","last_clean":"2024-03-04T07:37:56.301583+0000","last_became_active":"2024-03-04T07:37:55.131326+0000","last_became_peered":"2024-03-04T07:37:55.131326+0000","last_unstale":"2024-03-04T07:37:56.301583+0000","last_undegraded":"2024-03-04T07:37:56.301583+0000","last_fullsized":"2024-03-04T07:37:56.301583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:56.231578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300272+0000","last_change":"2024-03-04T07:37:55.137395+0000","last_active":"2024-03-04T07:37:56.300272+0000","last_peered":"2024-03-04T07:37:56.300272+0000","last_clean":"2024-03-04T07:37:56.300272+0000","last_became_active":"2024-03-04T07:37:55.137054+0000","last_became_peered":"2024-03-04T07:37:55.137054+0000","last_unstale":"2024-03-04T07:37:56.300272+0000","last_undegraded":"2024-03-04T07:37:56.300272+0000","last_fullsized":"2024-03-04T07:37:56.300272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:19.222481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301716+0000","last_change":"2024-03-04T07:37:55.137414+0000","last_active":"2024-03-04T07:37:56.301716+0000","last_peered":"2024-03-04T07:37:56.301716+0000","last_clean":"2024-03-04T07:37:56.301716+0000","last_became_active":"2024-03-04T07:37:55.137157+0000","last_became_peered":"2024-03-04T07:37:55.137157+0000","last_unstale":"2024-03-04T07:37:56.301716+0000","last_undegraded":"2024-03-04T07:37:56.301716+0000","last_fullsized":"2024-03-04T07:37:56.301716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:36.593491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096514+0000","last_change":"2024-03-04T07:37:55.121960+0000","last_active":"2024-03-04T07:37:56.096514+0000","last_peered":"2024-03-04T07:37:56.096514+0000","last_clean":"2024-03-04T07:37:56.096514+0000","last_became_active":"2024-03-04T07:37:55.115543+0000","last_became_peered":"2024-03-04T07:37:55.115543+0000","last_unstale":"2024-03-04T07:37:56.096514+0000","last_undegraded":"2024-03-04T07:37:56.096514+0000","last_fullsized":"2024-03-04T07:37:56.096514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:36.320475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096321+0000","last_change":"2024-03-04T07:37:55.128908+0000","last_active":"2024-03-04T07:37:56.096321+0000","last_peered":"2024-03-04T07:37:56.096321+0000","last_clean":"2024-03-04T07:37:56.096321+0000","last_became_active":"2024-03-04T07:37:55.128500+0000","last_became_peered":"2024-03-04T07:37:55.128500+0000","last_unstale":"2024-03-04T07:37:56.096321+0000","last_undegraded":"2024-03-04T07:37:56.096321+0000","last_fullsized":"2024-03-04T07:37:56.096321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:10.598424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096219+0000","last_change":"2024-03-04T07:37:54.121610+0000","last_active":"2024-03-04T07:37:56.096219+0000","last_peered":"2024-03-04T07:37:56.096219+0000","last_clean":"2024-03-04T07:37:56.096219+0000","last_became_active":"2024-03-04T07:37:54.121150+0000","last_became_peered":"2024-03-04T07:37:54.121150+0000","last_unstale":"2024-03-04T07:37:56.096219+0000","last_undegraded":"2024-03-04T07:37:56.096219+0000","last_fullsized":"2024-03-04T07:37:56.096219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:05:03.965626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154034+0000","last_change":"2024-03-04T07:37:55.138904+0000","last_active":"2024-03-04T07:37:56.154034+0000","last_peered":"2024-03-04T07:37:56.154034+0000","last_clean":"2024-03-04T07:37:56.154034+0000","last_became_active":"2024-03-04T07:37:55.138376+0000","last_became_peered":"2024-03-04T07:37:55.138376+0000","last_unstale":"2024-03-04T07:37:56.154034+0000","last_undegraded":"2024-03-04T07:37:56.154034+0000","last_fullsized":"2024-03-04T07:37:56.154034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:28.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154617+0000","last_change":"2024-03-04T07:37:54.139218+0000","last_active":"2024-03-04T07:37:56.154617+0000","last_peered":"2024-03-04T07:37:56.154617+0000","last_clean":"2024-03-04T07:37:56.154617+0000","last_became_active":"2024-03-04T07:37:54.138893+0000","last_became_peered":"2024-03-04T07:37:54.138893+0000","last_unstale":"2024-03-04T07:37:56.154617+0000","last_undegraded":"2024-03-04T07:37:56.154617+0000","last_fullsized":"2024-03-04T07:37:56.154617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:01.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302247+0000","last_change":"2024-03-04T07:37:55.135953+0000","last_active":"2024-03-04T07:37:56.302247+0000","last_peered":"2024-03-04T07:37:56.302247+0000","last_clean":"2024-03-04T07:37:56.302247+0000","last_became_active":"2024-03-04T07:37:55.132880+0000","last_became_peered":"2024-03-04T07:37:55.132880+0000","last_unstale":"2024-03-04T07:37:56.302247+0000","last_undegraded":"2024-03-04T07:37:56.302247+0000","last_fullsized":"2024-03-04T07:37:56.302247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:03.848428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152629+0000","last_change":"2024-03-04T07:37:54.122493+0000","last_active":"2024-03-04T07:37:56.152629+0000","last_peered":"2024-03-04T07:37:56.152629+0000","last_clean":"2024-03-04T07:37:56.152629+0000","last_became_active":"2024-03-04T07:37:54.122058+0000","last_became_peered":"2024-03-04T07:37:54.122058+0000","last_unstale":"2024-03-04T07:37:56.152629+0000","last_undegraded":"2024-03-04T07:37:56.152629+0000","last_fullsized":"2024-03-04T07:37:56.152629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:02.516175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300596+0000","last_change":"2024-03-04T07:37:55.136155+0000","last_active":"2024-03-04T07:37:56.300596+0000","last_peered":"2024-03-04T07:37:56.300596+0000","last_clean":"2024-03-04T07:37:56.300596+0000","last_became_active":"2024-03-04T07:37:55.135796+0000","last_became_peered":"2024-03-04T07:37:55.135796+0000","last_unstale":"2024-03-04T07:37:56.300596+0000","last_undegraded":"2024-03-04T07:37:56.300596+0000","last_fullsized":"2024-03-04T07:37:56.300596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:37.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152415+0000","last_change":"2024-03-04T07:37:54.118780+0000","last_active":"2024-03-04T07:37:56.152415+0000","last_peered":"2024-03-04T07:37:56.152415+0000","last_clean":"2024-03-04T07:37:56.152415+0000","last_became_active":"2024-03-04T07:37:54.118448+0000","last_became_peered":"2024-03-04T07:37:54.118448+0000","last_unstale":"2024-03-04T07:37:56.152415+0000","last_undegraded":"2024-03-04T07:37:56.152415+0000","last_fullsized":"2024-03-04T07:37:56.152415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:25.547046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301590+0000","last_change":"2024-03-04T07:37:55.120991+0000","last_active":"2024-03-04T07:37:56.301590+0000","last_peered":"2024-03-04T07:37:56.301590+0000","last_clean":"2024-03-04T07:37:56.301590+0000","last_became_active":"2024-03-04T07:37:55.120626+0000","last_became_peered":"2024-03-04T07:37:55.120626+0000","last_unstale":"2024-03-04T07:37:56.301590+0000","last_undegraded":"2024-03-04T07:37:56.301590+0000","last_fullsized":"2024-03-04T07:37:56.301590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:55:41.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301257+0000","last_change":"2024-03-04T07:37:54.122241+0000","last_active":"2024-03-04T07:37:56.301257+0000","last_peered":"2024-03-04T07:37:56.301257+0000","last_clean":"2024-03-04T07:37:56.301257+0000","last_became_active":"2024-03-04T07:37:54.120990+0000","last_became_peered":"2024-03-04T07:37:54.120990+0000","last_unstale":"2024-03-04T07:37:56.301257+0000","last_undegraded":"2024-03-04T07:37:56.301257+0000","last_fullsized":"2024-03-04T07:37:56.301257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:03.233754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152226+0000","last_change":"2024-03-04T07:37:55.121692+0000","last_active":"2024-03-04T07:37:56.152226+0000","last_peered":"2024-03-04T07:37:56.152226+0000","last_clean":"2024-03-04T07:37:56.152226+0000","last_became_active":"2024-03-04T07:37:55.121345+0000","last_became_peered":"2024-03-04T07:37:55.121345+0000","last_unstale":"2024-03-04T07:37:56.152226+0000","last_undegraded":"2024-03-04T07:37:56.152226+0000","last_fullsized":"2024-03-04T07:37:56.152226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:52:42.045265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.095944+0000","last_change":"2024-03-04T07:37:54.133489+0000","last_active":"2024-03-04T07:37:56.095944+0000","last_peered":"2024-03-04T07:37:56.095944+0000","last_clean":"2024-03-04T07:37:56.095944+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:37:56.095944+0000","last_undegraded":"2024-03-04T07:37:56.095944+0000","last_fullsized":"2024-03-04T07:37:56.095944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:49:03.600907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301458+0000","last_change":"2024-03-04T07:37:55.128278+0000","last_active":"2024-03-04T07:37:56.301458+0000","last_peered":"2024-03-04T07:37:56.301458+0000","last_clean":"2024-03-04T07:37:56.301458+0000","last_became_active":"2024-03-04T07:37:55.127913+0000","last_became_peered":"2024-03-04T07:37:55.127913+0000","last_unstale":"2024-03-04T07:37:56.301458+0000","last_undegraded":"2024-03-04T07:37:56.301458+0000","last_fullsized":"2024-03-04T07:37:56.301458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:54:18.623855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302158+0000","last_change":"2024-03-04T07:37:54.132195+0000","last_active":"2024-03-04T07:37:56.302158+0000","last_peered":"2024-03-04T07:37:56.302158+0000","last_clean":"2024-03-04T07:37:56.302158+0000","last_became_active":"2024-03-04T07:37:54.131830+0000","last_became_peered":"2024-03-04T07:37:54.131830+0000","last_unstale":"2024-03-04T07:37:56.302158+0000","last_undegraded":"2024-03-04T07:37:56.302158+0000","last_fullsized":"2024-03-04T07:37:56.302158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.744342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153043+0000","last_change":"2024-03-04T07:37:55.138281+0000","last_active":"2024-03-04T07:37:56.153043+0000","last_peered":"2024-03-04T07:37:56.153043+0000","last_clean":"2024-03-04T07:37:56.153043+0000","last_became_active":"2024-03-04T07:37:55.137919+0000","last_became_peered":"2024-03-04T07:37:55.137919+0000","last_unstale":"2024-03-04T07:37:56.153043+0000","last_undegraded":"2024-03-04T07:37:56.153043+0000","last_fullsized":"2024-03-04T07:37:56.153043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:53:06.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300702+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:37:56.300702+0000","last_peered":"2024-03-04T07:37:56.300702+0000","last_clean":"2024-03-04T07:37:56.300702+0000","last_became_active":"2024-03-04T07:37:54.121503+0000","last_became_peered":"2024-03-04T07:37:54.121503+0000","last_unstale":"2024-03-04T07:37:56.300702+0000","last_undegraded":"2024-03-04T07:37:56.300702+0000","last_fullsized":"2024-03-04T07:37:56.300702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:03:07.531246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096196+0000","last_change":"2024-03-04T07:37:55.126924+0000","last_active":"2024-03-04T07:37:56.096196+0000","last_peered":"2024-03-04T07:37:56.096196+0000","last_clean":"2024-03-04T07:37:56.096196+0000","last_became_active":"2024-03-04T07:37:55.126266+0000","last_became_peered":"2024-03-04T07:37:55.126266+0000","last_unstale":"2024-03-04T07:37:56.096196+0000","last_undegraded":"2024-03-04T07:37:56.096196+0000","last_fullsized":"2024-03-04T07:37:56.096196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:01.603430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096500+0000","last_change":"2024-03-04T07:37:54.119864+0000","last_active":"2024-03-04T07:37:56.096500+0000","last_peered":"2024-03-04T07:37:56.096500+0000","last_clean":"2024-03-04T07:37:56.096500+0000","last_became_active":"2024-03-04T07:37:54.119515+0000","last_became_peered":"2024-03-04T07:37:54.119515+0000","last_unstale":"2024-03-04T07:37:56.096500+0000","last_undegraded":"2024-03-04T07:37:56.096500+0000","last_fullsized":"2024-03-04T07:37:56.096500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:10.422458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300769+0000","last_change":"2024-03-04T07:37:55.120135+0000","last_active":"2024-03-04T07:37:56.300769+0000","last_peered":"2024-03-04T07:37:56.300769+0000","last_clean":"2024-03-04T07:37:56.300769+0000","last_became_active":"2024-03-04T07:37:55.119670+0000","last_became_peered":"2024-03-04T07:37:55.119670+0000","last_unstale":"2024-03-04T07:37:56.300769+0000","last_undegraded":"2024-03-04T07:37:56.300769+0000","last_fullsized":"2024-03-04T07:37:56.300769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:09.688447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337257+0000","last_change":"2024-03-04T07:37:54.120222+0000","last_active":"2024-03-04T07:37:56.337257+0000","last_peered":"2024-03-04T07:37:56.337257+0000","last_clean":"2024-03-04T07:37:56.337257+0000","last_became_active":"2024-03-04T07:37:54.119802+0000","last_became_peered":"2024-03-04T07:37:54.119802+0000","last_unstale":"2024-03-04T07:37:56.337257+0000","last_undegraded":"2024-03-04T07:37:56.337257+0000","last_fullsized":"2024-03-04T07:37:56.337257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:12:40.811870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302038+0000","last_change":"2024-03-04T07:37:55.136371+0000","last_active":"2024-03-04T07:37:56.302038+0000","last_peered":"2024-03-04T07:37:56.302038+0000","last_clean":"2024-03-04T07:37:56.302038+0000","last_became_active":"2024-03-04T07:37:55.132587+0000","last_became_peered":"2024-03-04T07:37:55.132587+0000","last_unstale":"2024-03-04T07:37:56.302038+0000","last_undegraded":"2024-03-04T07:37:56.302038+0000","last_fullsized":"2024-03-04T07:37:56.302038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:24.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301599+0000","last_change":"2024-03-04T07:37:54.123041+0000","last_active":"2024-03-04T07:37:56.301599+0000","last_peered":"2024-03-04T07:37:56.301599+0000","last_clean":"2024-03-04T07:37:56.301599+0000","last_became_active":"2024-03-04T07:37:54.122723+0000","last_became_peered":"2024-03-04T07:37:54.122723+0000","last_unstale":"2024-03-04T07:37:56.301599+0000","last_undegraded":"2024-03-04T07:37:56.301599+0000","last_fullsized":"2024-03-04T07:37:56.301599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:06:30.845376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300896+0000","last_change":"2024-03-04T07:37:55.115387+0000","last_active":"2024-03-04T07:37:56.300896+0000","last_peered":"2024-03-04T07:37:56.300896+0000","last_clean":"2024-03-04T07:37:56.300896+0000","last_became_active":"2024-03-04T07:37:55.114947+0000","last_became_peered":"2024-03-04T07:37:55.114947+0000","last_unstale":"2024-03-04T07:37:56.300896+0000","last_undegraded":"2024-03-04T07:37:56.300896+0000","last_fullsized":"2024-03-04T07:37:56.300896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:09:11.840989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"84'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096303+0000","last_change":"2024-03-04T07:37:54.117767+0000","last_active":"2024-03-04T07:37:56.096303+0000","last_peered":"2024-03-04T07:37:56.096303+0000","last_clean":"2024-03-04T07:37:56.096303+0000","last_became_active":"2024-03-04T07:37:54.117371+0000","last_became_peered":"2024-03-04T07:37:54.117371+0000","last_unstale":"2024-03-04T07:37:56.096303+0000","last_undegraded":"2024-03-04T07:37:56.096303+0000","last_fullsized":"2024-03-04T07:37:56.096303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:08:32.164636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152426+0000","last_change":"2024-03-04T07:37:55.112707+0000","last_active":"2024-03-04T07:37:56.152426+0000","last_peered":"2024-03-04T07:37:56.152426+0000","last_clean":"2024-03-04T07:37:56.152426+0000","last_became_active":"2024-03-04T07:37:55.112327+0000","last_became_peered":"2024-03-04T07:37:55.112327+0000","last_unstale":"2024-03-04T07:37:56.152426+0000","last_undegraded":"2024-03-04T07:37:56.152426+0000","last_fullsized":"2024-03-04T07:37:56.152426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:03.901687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096229+0000","last_change":"2024-03-04T07:37:54.125286+0000","last_active":"2024-03-04T07:37:56.096229+0000","last_peered":"2024-03-04T07:37:56.096229+0000","last_clean":"2024-03-04T07:37:56.096229+0000","last_became_active":"2024-03-04T07:37:54.124966+0000","last_became_peered":"2024-03-04T07:37:54.124966+0000","last_unstale":"2024-03-04T07:37:56.096229+0000","last_undegraded":"2024-03-04T07:37:56.096229+0000","last_fullsized":"2024-03-04T07:37:56.096229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:19.880681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302449+0000","last_change":"2024-03-04T07:37:55.137952+0000","last_active":"2024-03-04T07:37:56.302449+0000","last_peered":"2024-03-04T07:37:56.302449+0000","last_clean":"2024-03-04T07:37:56.302449+0000","last_became_active":"2024-03-04T07:37:55.137583+0000","last_became_peered":"2024-03-04T07:37:55.137583+0000","last_unstale":"2024-03-04T07:37:56.302449+0000","last_undegraded":"2024-03-04T07:37:56.302449+0000","last_fullsized":"2024-03-04T07:37:56.302449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:30.145453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301263+0000","last_change":"2024-03-04T07:37:54.134733+0000","last_active":"2024-03-04T07:37:56.301263+0000","last_peered":"2024-03-04T07:37:56.301263+0000","last_clean":"2024-03-04T07:37:56.301263+0000","last_became_active":"2024-03-04T07:37:54.134400+0000","last_became_peered":"2024-03-04T07:37:54.134400+0000","last_unstale":"2024-03-04T07:37:56.301263+0000","last_undegraded":"2024-03-04T07:37:56.301263+0000","last_fullsized":"2024-03-04T07:37:56.301263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:24.192479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337082+0000","last_change":"2024-03-04T07:37:55.124991+0000","last_active":"2024-03-04T07:37:56.337082+0000","last_peered":"2024-03-04T07:37:56.337082+0000","last_clean":"2024-03-04T07:37:56.337082+0000","last_became_active":"2024-03-04T07:37:55.124621+0000","last_became_peered":"2024-03-04T07:37:55.124621+0000","last_unstale":"2024-03-04T07:37:56.337082+0000","last_undegraded":"2024-03-04T07:37:56.337082+0000","last_fullsized":"2024-03-04T07:37:56.337082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:05.252145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300536+0000","last_change":"2024-03-04T07:37:54.118743+0000","last_active":"2024-03-04T07:37:56.300536+0000","last_peered":"2024-03-04T07:37:56.300536+0000","last_clean":"2024-03-04T07:37:56.300536+0000","last_became_active":"2024-03-04T07:37:54.118386+0000","last_became_peered":"2024-03-04T07:37:54.118386+0000","last_unstale":"2024-03-04T07:37:56.300536+0000","last_undegraded":"2024-03-04T07:37:56.300536+0000","last_fullsized":"2024-03-04T07:37:56.300536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:27.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336766+0000","last_change":"2024-03-04T07:37:55.115431+0000","last_active":"2024-03-04T07:37:56.336766+0000","last_peered":"2024-03-04T07:37:56.336766+0000","last_clean":"2024-03-04T07:37:56.336766+0000","last_became_active":"2024-03-04T07:37:55.115064+0000","last_became_peered":"2024-03-04T07:37:55.115064+0000","last_unstale":"2024-03-04T07:37:56.336766+0000","last_undegraded":"2024-03-04T07:37:56.336766+0000","last_fullsized":"2024-03-04T07:37:56.336766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:46:57.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152041+0000","last_change":"2024-03-04T07:37:54.118352+0000","last_active":"2024-03-04T07:37:56.152041+0000","last_peered":"2024-03-04T07:37:56.152041+0000","last_clean":"2024-03-04T07:37:56.152041+0000","last_became_active":"2024-03-04T07:37:54.117981+0000","last_became_peered":"2024-03-04T07:37:54.117981+0000","last_unstale":"2024-03-04T07:37:56.152041+0000","last_undegraded":"2024-03-04T07:37:56.152041+0000","last_fullsized":"2024-03-04T07:37:56.152041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:18:54.949946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300862+0000","last_change":"2024-03-04T07:37:55.114882+0000","last_active":"2024-03-04T07:37:56.300862+0000","last_peered":"2024-03-04T07:37:56.300862+0000","last_clean":"2024-03-04T07:37:56.300862+0000","last_became_active":"2024-03-04T07:37:55.113139+0000","last_became_peered":"2024-03-04T07:37:55.113139+0000","last_unstale":"2024-03-04T07:37:56.300862+0000","last_undegraded":"2024-03-04T07:37:56.300862+0000","last_fullsized":"2024-03-04T07:37:56.300862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:28.133710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152593+0000","last_change":"2024-03-04T07:37:54.119350+0000","last_active":"2024-03-04T07:37:56.152593+0000","last_peered":"2024-03-04T07:37:56.152593+0000","last_clean":"2024-03-04T07:37:56.152593+0000","last_became_active":"2024-03-04T07:37:54.119017+0000","last_became_peered":"2024-03-04T07:37:54.119017+0000","last_unstale":"2024-03-04T07:37:56.152593+0000","last_undegraded":"2024-03-04T07:37:56.152593+0000","last_fullsized":"2024-03-04T07:37:56.152593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:00.503583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301290+0000","last_change":"2024-03-04T07:37:55.114242+0000","last_active":"2024-03-04T07:37:56.301290+0000","last_peered":"2024-03-04T07:37:56.301290+0000","last_clean":"2024-03-04T07:37:56.301290+0000","last_became_active":"2024-03-04T07:37:55.113852+0000","last_became_peered":"2024-03-04T07:37:55.113852+0000","last_unstale":"2024-03-04T07:37:56.301290+0000","last_undegraded":"2024-03-04T07:37:56.301290+0000","last_fullsized":"2024-03-04T07:37:56.301290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:02:30.804894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096019+0000","last_change":"2024-03-04T07:37:54.125548+0000","last_active":"2024-03-04T07:37:56.096019+0000","last_peered":"2024-03-04T07:37:56.096019+0000","last_clean":"2024-03-04T07:37:56.096019+0000","last_became_active":"2024-03-04T07:37:54.125127+0000","last_became_peered":"2024-03-04T07:37:54.125127+0000","last_unstale":"2024-03-04T07:37:56.096019+0000","last_undegraded":"2024-03-04T07:37:56.096019+0000","last_fullsized":"2024-03-04T07:37:56.096019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:26.256247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096432+0000","last_change":"2024-03-04T07:37:55.128542+0000","last_active":"2024-03-04T07:37:56.096432+0000","last_peered":"2024-03-04T07:37:56.096432+0000","last_clean":"2024-03-04T07:37:56.096432+0000","last_became_active":"2024-03-04T07:37:55.128239+0000","last_became_peered":"2024-03-04T07:37:55.128239+0000","last_unstale":"2024-03-04T07:37:56.096432+0000","last_undegraded":"2024-03-04T07:37:56.096432+0000","last_fullsized":"2024-03-04T07:37:56.096432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:17.078095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336884+0000","last_change":"2024-03-04T07:37:54.133672+0000","last_active":"2024-03-04T07:37:56.336884+0000","last_peered":"2024-03-04T07:37:56.336884+0000","last_clean":"2024-03-04T07:37:56.336884+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:37:56.336884+0000","last_undegraded":"2024-03-04T07:37:56.336884+0000","last_fullsized":"2024-03-04T07:37:56.336884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:52.345158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096226+0000","last_change":"2024-03-04T07:37:55.137673+0000","last_active":"2024-03-04T07:37:56.096226+0000","last_peered":"2024-03-04T07:37:56.096226+0000","last_clean":"2024-03-04T07:37:56.096226+0000","last_became_active":"2024-03-04T07:37:55.137317+0000","last_became_peered":"2024-03-04T07:37:55.137317+0000","last_unstale":"2024-03-04T07:37:56.096226+0000","last_undegraded":"2024-03-04T07:37:56.096226+0000","last_fullsized":"2024-03-04T07:37:56.096226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:52.683889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301040+0000","last_change":"2024-03-04T07:37:55.112643+0000","last_active":"2024-03-04T07:37:56.301040+0000","last_peered":"2024-03-04T07:37:56.301040+0000","last_clean":"2024-03-04T07:37:56.301040+0000","last_became_active":"2024-03-04T07:37:55.112223+0000","last_became_peered":"2024-03-04T07:37:55.112223+0000","last_unstale":"2024-03-04T07:37:56.301040+0000","last_undegraded":"2024-03-04T07:37:56.301040+0000","last_fullsized":"2024-03-04T07:37:56.301040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:02:49.701226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301533+0000","last_change":"2024-03-04T07:37:54.139335+0000","last_active":"2024-03-04T07:37:56.301533+0000","last_peered":"2024-03-04T07:37:56.301533+0000","last_clean":"2024-03-04T07:37:56.301533+0000","last_became_active":"2024-03-04T07:37:54.138969+0000","last_became_peered":"2024-03-04T07:37:54.138969+0000","last_unstale":"2024-03-04T07:37:56.301533+0000","last_undegraded":"2024-03-04T07:37:56.301533+0000","last_fullsized":"2024-03-04T07:37:56.301533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:41:15.204738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154758+0000","last_change":"2024-03-04T07:37:55.133575+0000","last_active":"2024-03-04T07:37:56.154758+0000","last_peered":"2024-03-04T07:37:56.154758+0000","last_clean":"2024-03-04T07:37:56.154758+0000","last_became_active":"2024-03-04T07:37:55.133220+0000","last_became_peered":"2024-03-04T07:37:55.133220+0000","last_unstale":"2024-03-04T07:37:56.154758+0000","last_undegraded":"2024-03-04T07:37:56.154758+0000","last_fullsized":"2024-03-04T07:37:56.154758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:21:12.835718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300803+0000","last_change":"2024-03-04T07:37:54.119906+0000","last_active":"2024-03-04T07:37:56.300803+0000","last_peered":"2024-03-04T07:37:56.300803+0000","last_clean":"2024-03-04T07:37:56.300803+0000","last_became_active":"2024-03-04T07:37:54.119592+0000","last_became_peered":"2024-03-04T07:37:54.119592+0000","last_unstale":"2024-03-04T07:37:56.300803+0000","last_undegraded":"2024-03-04T07:37:56.300803+0000","last_fullsized":"2024-03-04T07:37:56.300803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:29.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":338,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336764+0000","last_change":"2024-03-04T07:36:35.278528+0000","last_active":"2024-03-04T07:37:56.336764+0000","last_peered":"2024-03-04T07:37:56.336764+0000","last_clean":"2024-03-04T07:37:56.336764+0000","last_became_active":"2024-03-04T07:36:35.277953+0000","last_became_peered":"2024-03-04T07:36:35.277953+0000","last_unstale":"2024-03-04T07:37:56.336764+0000","last_undegraded":"2024-03-04T07:37:56.336764+0000","last_fullsized":"2024-03-04T07:37:56.336764+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_clean_scrub_stamp":"2024-03-04T07:32:42.283525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:58:47.223527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301624+0000","last_change":"2024-03-04T07:37:55.124204+0000","last_active":"2024-03-04T07:37:56.301624+0000","last_peered":"2024-03-04T07:37:56.301624+0000","last_clean":"2024-03-04T07:37:56.301624+0000","last_became_active":"2024-03-04T07:37:55.123863+0000","last_became_peered":"2024-03-04T07:37:55.123863+0000","last_unstale":"2024-03-04T07:37:56.301624+0000","last_undegraded":"2024-03-04T07:37:56.301624+0000","last_fullsized":"2024-03-04T07:37:56.301624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:33.830367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336915+0000","last_change":"2024-03-04T07:37:54.121538+0000","last_active":"2024-03-04T07:37:56.336915+0000","last_peered":"2024-03-04T07:37:56.336915+0000","last_clean":"2024-03-04T07:37:56.336915+0000","last_became_active":"2024-03-04T07:37:54.120939+0000","last_became_peered":"2024-03-04T07:37:54.120939+0000","last_unstale":"2024-03-04T07:37:56.336915+0000","last_undegraded":"2024-03-04T07:37:56.336915+0000","last_fullsized":"2024-03-04T07:37:56.336915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:29.597247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152443+0000","last_change":"2024-03-04T07:37:54.128180+0000","last_active":"2024-03-04T07:37:56.152443+0000","last_peered":"2024-03-04T07:37:56.152443+0000","last_clean":"2024-03-04T07:37:56.152443+0000","last_became_active":"2024-03-04T07:37:54.127800+0000","last_became_peered":"2024-03-04T07:37:54.127800+0000","last_unstale":"2024-03-04T07:37:56.152443+0000","last_undegraded":"2024-03-04T07:37:56.152443+0000","last_fullsized":"2024-03-04T07:37:56.152443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:54.046873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302961+0000","last_change":"2024-03-04T07:37:55.133482+0000","last_active":"2024-03-04T07:37:56.302961+0000","last_peered":"2024-03-04T07:37:56.302961+0000","last_clean":"2024-03-04T07:37:56.302961+0000","last_became_active":"2024-03-04T07:37:55.133137+0000","last_became_peered":"2024-03-04T07:37:55.133137+0000","last_unstale":"2024-03-04T07:37:56.302961+0000","last_undegraded":"2024-03-04T07:37:56.302961+0000","last_fullsized":"2024-03-04T07:37:56.302961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:53.509564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096540+0000","last_change":"2024-03-04T07:37:54.134294+0000","last_active":"2024-03-04T07:37:56.096540+0000","last_peered":"2024-03-04T07:37:56.096540+0000","last_clean":"2024-03-04T07:37:56.096540+0000","last_became_active":"2024-03-04T07:37:54.133677+0000","last_became_peered":"2024-03-04T07:37:54.133677+0000","last_unstale":"2024-03-04T07:37:56.096540+0000","last_undegraded":"2024-03-04T07:37:56.096540+0000","last_fullsized":"2024-03-04T07:37:56.096540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:57:54.082419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154045+0000","last_change":"2024-03-04T07:37:55.138305+0000","last_active":"2024-03-04T07:37:56.154045+0000","last_peered":"2024-03-04T07:37:56.154045+0000","last_clean":"2024-03-04T07:37:56.154045+0000","last_became_active":"2024-03-04T07:37:55.129937+0000","last_became_peered":"2024-03-04T07:37:55.129937+0000","last_unstale":"2024-03-04T07:37:56.154045+0000","last_undegraded":"2024-03-04T07:37:56.154045+0000","last_fullsized":"2024-03-04T07:37:56.154045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:45.136539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300446+0000","last_change":"2024-03-04T07:37:54.133817+0000","last_active":"2024-03-04T07:37:56.300446+0000","last_peered":"2024-03-04T07:37:56.300446+0000","last_clean":"2024-03-04T07:37:56.300446+0000","last_became_active":"2024-03-04T07:37:54.133471+0000","last_became_peered":"2024-03-04T07:37:54.133471+0000","last_unstale":"2024-03-04T07:37:56.300446+0000","last_undegraded":"2024-03-04T07:37:56.300446+0000","last_fullsized":"2024-03-04T07:37:56.300446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:00:51.483859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302030+0000","last_change":"2024-03-04T07:37:55.144639+0000","last_active":"2024-03-04T07:37:56.302030+0000","last_peered":"2024-03-04T07:37:56.302030+0000","last_clean":"2024-03-04T07:37:56.302030+0000","last_became_active":"2024-03-04T07:37:55.144194+0000","last_became_peered":"2024-03-04T07:37:55.144194+0000","last_unstale":"2024-03-04T07:37:56.302030+0000","last_undegraded":"2024-03-04T07:37:56.302030+0000","last_fullsized":"2024-03-04T07:37:56.302030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:42.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300192+0000","last_change":"2024-03-04T07:37:54.119871+0000","last_active":"2024-03-04T07:37:56.300192+0000","last_peered":"2024-03-04T07:37:56.300192+0000","last_clean":"2024-03-04T07:37:56.300192+0000","last_became_active":"2024-03-04T07:37:54.119432+0000","last_became_peered":"2024-03-04T07:37:54.119432+0000","last_unstale":"2024-03-04T07:37:56.300192+0000","last_undegraded":"2024-03-04T07:37:56.300192+0000","last_fullsized":"2024-03-04T07:37:56.300192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:01:33.604940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152131+0000","last_change":"2024-03-04T07:37:55.115145+0000","last_active":"2024-03-04T07:37:56.152131+0000","last_peered":"2024-03-04T07:37:56.152131+0000","last_clean":"2024-03-04T07:37:56.152131+0000","last_became_active":"2024-03-04T07:37:55.114791+0000","last_became_peered":"2024-03-04T07:37:55.114791+0000","last_unstale":"2024-03-04T07:37:56.152131+0000","last_undegraded":"2024-03-04T07:37:56.152131+0000","last_fullsized":"2024-03-04T07:37:56.152131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:19.685977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153141+0000","last_change":"2024-03-04T07:37:54.141630+0000","last_active":"2024-03-04T07:37:56.153141+0000","last_peered":"2024-03-04T07:37:56.153141+0000","last_clean":"2024-03-04T07:37:56.153141+0000","last_became_active":"2024-03-04T07:37:54.139820+0000","last_became_peered":"2024-03-04T07:37:54.139820+0000","last_unstale":"2024-03-04T07:37:56.153141+0000","last_undegraded":"2024-03-04T07:37:56.153141+0000","last_fullsized":"2024-03-04T07:37:56.153141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:22.419437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300959+0000","last_change":"2024-03-04T07:37:55.133270+0000","last_active":"2024-03-04T07:37:56.300959+0000","last_peered":"2024-03-04T07:37:56.300959+0000","last_clean":"2024-03-04T07:37:56.300959+0000","last_became_active":"2024-03-04T07:37:55.132909+0000","last_became_peered":"2024-03-04T07:37:55.132909+0000","last_unstale":"2024-03-04T07:37:56.300959+0000","last_undegraded":"2024-03-04T07:37:56.300959+0000","last_fullsized":"2024-03-04T07:37:56.300959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:21.026141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.095914+0000","last_change":"2024-03-04T07:37:54.135988+0000","last_active":"2024-03-04T07:37:56.095914+0000","last_peered":"2024-03-04T07:37:56.095914+0000","last_clean":"2024-03-04T07:37:56.095914+0000","last_became_active":"2024-03-04T07:37:54.135627+0000","last_became_peered":"2024-03-04T07:37:54.135627+0000","last_unstale":"2024-03-04T07:37:56.095914+0000","last_undegraded":"2024-03-04T07:37:56.095914+0000","last_fullsized":"2024-03-04T07:37:56.095914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:06.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153423+0000","last_change":"2024-03-04T07:37:55.144551+0000","last_active":"2024-03-04T07:37:56.153423+0000","last_peered":"2024-03-04T07:37:56.153423+0000","last_clean":"2024-03-04T07:37:56.153423+0000","last_became_active":"2024-03-04T07:37:55.143882+0000","last_became_peered":"2024-03-04T07:37:55.143882+0000","last_unstale":"2024-03-04T07:37:56.153423+0000","last_undegraded":"2024-03-04T07:37:56.153423+0000","last_fullsized":"2024-03-04T07:37:56.153423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:07.254941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152278+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:37:56.152278+0000","last_peered":"2024-03-04T07:37:56.152278+0000","last_clean":"2024-03-04T07:37:56.152278+0000","last_became_active":"2024-03-04T07:37:54.121642+0000","last_became_peered":"2024-03-04T07:37:54.121642+0000","last_unstale":"2024-03-04T07:37:56.152278+0000","last_undegraded":"2024-03-04T07:37:56.152278+0000","last_fullsized":"2024-03-04T07:37:56.152278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:06.576421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337037+0000","last_change":"2024-03-04T07:37:55.133870+0000","last_active":"2024-03-04T07:37:56.337037+0000","last_peered":"2024-03-04T07:37:56.337037+0000","last_clean":"2024-03-04T07:37:56.337037+0000","last_became_active":"2024-03-04T07:37:55.133542+0000","last_became_peered":"2024-03-04T07:37:55.133542+0000","last_unstale":"2024-03-04T07:37:56.337037+0000","last_undegraded":"2024-03-04T07:37:56.337037+0000","last_fullsized":"2024-03-04T07:37:56.337037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:17:44.373426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096337+0000","last_change":"2024-03-04T07:37:54.126326+0000","last_active":"2024-03-04T07:37:56.096337+0000","last_peered":"2024-03-04T07:37:56.096337+0000","last_clean":"2024-03-04T07:37:56.096337+0000","last_became_active":"2024-03-04T07:37:54.125989+0000","last_became_peered":"2024-03-04T07:37:54.125989+0000","last_unstale":"2024-03-04T07:37:56.096337+0000","last_undegraded":"2024-03-04T07:37:56.096337+0000","last_fullsized":"2024-03-04T07:37:56.096337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:41.108528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337302+0000","last_change":"2024-03-04T07:37:55.120052+0000","last_active":"2024-03-04T07:37:56.337302+0000","last_peered":"2024-03-04T07:37:56.337302+0000","last_clean":"2024-03-04T07:37:56.337302+0000","last_became_active":"2024-03-04T07:37:55.119708+0000","last_became_peered":"2024-03-04T07:37:55.119708+0000","last_unstale":"2024-03-04T07:37:56.337302+0000","last_undegraded":"2024-03-04T07:37:56.337302+0000","last_fullsized":"2024-03-04T07:37:56.337302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:25:17.679416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301525+0000","last_change":"2024-03-04T07:37:54.134018+0000","last_active":"2024-03-04T07:37:56.301525+0000","last_peered":"2024-03-04T07:37:56.301525+0000","last_clean":"2024-03-04T07:37:56.301525+0000","last_became_active":"2024-03-04T07:37:54.133673+0000","last_became_peered":"2024-03-04T07:37:54.133673+0000","last_unstale":"2024-03-04T07:37:56.301525+0000","last_undegraded":"2024-03-04T07:37:56.301525+0000","last_fullsized":"2024-03-04T07:37:56.301525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:49.438338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337018+0000","last_change":"2024-03-04T07:37:55.124052+0000","last_active":"2024-03-04T07:37:56.337018+0000","last_peered":"2024-03-04T07:37:56.337018+0000","last_clean":"2024-03-04T07:37:56.337018+0000","last_became_active":"2024-03-04T07:37:55.123667+0000","last_became_peered":"2024-03-04T07:37:55.123667+0000","last_unstale":"2024-03-04T07:37:56.337018+0000","last_undegraded":"2024-03-04T07:37:56.337018+0000","last_fullsized":"2024-03-04T07:37:56.337018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:25.491236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302932+0000","last_change":"2024-03-04T07:37:54.140806+0000","last_active":"2024-03-04T07:37:56.302932+0000","last_peered":"2024-03-04T07:37:56.302932+0000","last_clean":"2024-03-04T07:37:56.302932+0000","last_became_active":"2024-03-04T07:37:54.140419+0000","last_became_peered":"2024-03-04T07:37:54.140419+0000","last_unstale":"2024-03-04T07:37:56.302932+0000","last_undegraded":"2024-03-04T07:37:56.302932+0000","last_fullsized":"2024-03-04T07:37:56.302932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:13:59.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337338+0000","last_change":"2024-03-04T07:37:55.113658+0000","last_active":"2024-03-04T07:37:56.337338+0000","last_peered":"2024-03-04T07:37:56.337338+0000","last_clean":"2024-03-04T07:37:56.337338+0000","last_became_active":"2024-03-04T07:37:55.113300+0000","last_became_peered":"2024-03-04T07:37:55.113300+0000","last_unstale":"2024-03-04T07:37:56.337338+0000","last_undegraded":"2024-03-04T07:37:56.337338+0000","last_fullsized":"2024-03-04T07:37:56.337338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:43:56.907387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301151+0000","last_change":"2024-03-04T07:37:55.117769+0000","last_active":"2024-03-04T07:37:56.301151+0000","last_peered":"2024-03-04T07:37:56.301151+0000","last_clean":"2024-03-04T07:37:56.301151+0000","last_became_active":"2024-03-04T07:37:55.117408+0000","last_became_peered":"2024-03-04T07:37:55.117408+0000","last_unstale":"2024-03-04T07:37:56.301151+0000","last_undegraded":"2024-03-04T07:37:56.301151+0000","last_fullsized":"2024-03-04T07:37:56.301151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:22.143108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096050+0000","last_change":"2024-03-04T07:37:55.115581+0000","last_active":"2024-03-04T07:37:56.096050+0000","last_peered":"2024-03-04T07:37:56.096050+0000","last_clean":"2024-03-04T07:37:56.096050+0000","last_became_active":"2024-03-04T07:37:55.113387+0000","last_became_peered":"2024-03-04T07:37:55.113387+0000","last_unstale":"2024-03-04T07:37:56.096050+0000","last_undegraded":"2024-03-04T07:37:56.096050+0000","last_fullsized":"2024-03-04T07:37:56.096050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:59.273650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096021+0000","last_change":"2024-03-04T07:37:55.135175+0000","last_active":"2024-03-04T07:37:56.096021+0000","last_peered":"2024-03-04T07:37:56.096021+0000","last_clean":"2024-03-04T07:37:56.096021+0000","last_became_active":"2024-03-04T07:37:55.134787+0000","last_became_peered":"2024-03-04T07:37:55.134787+0000","last_unstale":"2024-03-04T07:37:56.096021+0000","last_undegraded":"2024-03-04T07:37:56.096021+0000","last_fullsized":"2024-03-04T07:37:56.096021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:04.227886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301953+0000","last_change":"2024-03-04T07:37:55.138146+0000","last_active":"2024-03-04T07:37:56.301953+0000","last_peered":"2024-03-04T07:37:56.301953+0000","last_clean":"2024-03-04T07:37:56.301953+0000","last_became_active":"2024-03-04T07:37:55.137806+0000","last_became_peered":"2024-03-04T07:37:55.137806+0000","last_unstale":"2024-03-04T07:37:56.301953+0000","last_undegraded":"2024-03-04T07:37:56.301953+0000","last_fullsized":"2024-03-04T07:37:56.301953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:10.596224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302792+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:37:56.302792+0000","last_peered":"2024-03-04T07:37:56.302792+0000","last_clean":"2024-03-04T07:37:56.302792+0000","last_became_active":"2024-03-04T07:37:55.140670+0000","last_became_peered":"2024-03-04T07:37:55.140670+0000","last_unstale":"2024-03-04T07:37:56.302792+0000","last_undegraded":"2024-03-04T07:37:56.302792+0000","last_fullsized":"2024-03-04T07:37:56.302792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:29:54.439417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301249+0000","last_change":"2024-03-04T07:37:55.115015+0000","last_active":"2024-03-04T07:37:56.301249+0000","last_peered":"2024-03-04T07:37:56.301249+0000","last_clean":"2024-03-04T07:37:56.301249+0000","last_became_active":"2024-03-04T07:37:55.114668+0000","last_became_peered":"2024-03-04T07:37:55.114668+0000","last_unstale":"2024-03-04T07:37:56.301249+0000","last_undegraded":"2024-03-04T07:37:56.301249+0000","last_fullsized":"2024-03-04T07:37:56.301249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:29:03.309234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096475+0000","last_change":"2024-03-04T07:37:55.119514+0000","last_active":"2024-03-04T07:37:56.096475+0000","last_peered":"2024-03-04T07:37:56.096475+0000","last_clean":"2024-03-04T07:37:56.096475+0000","last_became_active":"2024-03-04T07:37:55.119166+0000","last_became_peered":"2024-03-04T07:37:55.119166+0000","last_unstale":"2024-03-04T07:37:56.096475+0000","last_undegraded":"2024-03-04T07:37:56.096475+0000","last_fullsized":"2024-03-04T07:37:56.096475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:33.468311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153879+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:37:56.153879+0000","last_peered":"2024-03-04T07:37:56.153879+0000","last_clean":"2024-03-04T07:37:56.153879+0000","last_became_active":"2024-03-04T07:37:55.140723+0000","last_became_peered":"2024-03-04T07:37:55.140723+0000","last_unstale":"2024-03-04T07:37:56.153879+0000","last_undegraded":"2024-03-04T07:37:56.153879+0000","last_fullsized":"2024-03-04T07:37:56.153879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:29.751190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300822+0000","last_change":"2024-03-04T07:37:55.127627+0000","last_active":"2024-03-04T07:37:56.300822+0000","last_peered":"2024-03-04T07:37:56.300822+0000","last_clean":"2024-03-04T07:37:56.300822+0000","last_became_active":"2024-03-04T07:37:55.127217+0000","last_became_peered":"2024-03-04T07:37:55.127217+0000","last_unstale":"2024-03-04T07:37:56.300822+0000","last_undegraded":"2024-03-04T07:37:56.300822+0000","last_fullsized":"2024-03-04T07:37:56.300822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:47.826970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153510+0000","last_change":"2024-03-04T07:37:55.129445+0000","last_active":"2024-03-04T07:37:56.153510+0000","last_peered":"2024-03-04T07:37:56.153510+0000","last_clean":"2024-03-04T07:37:56.153510+0000","last_became_active":"2024-03-04T07:37:55.129013+0000","last_became_peered":"2024-03-04T07:37:55.129013+0000","last_unstale":"2024-03-04T07:37:56.153510+0000","last_undegraded":"2024-03-04T07:37:56.153510+0000","last_fullsized":"2024-03-04T07:37:56.153510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:15.578436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300404+0000","last_change":"2024-03-04T07:37:55.126397+0000","last_active":"2024-03-04T07:37:56.300404+0000","last_peered":"2024-03-04T07:37:56.300404+0000","last_clean":"2024-03-04T07:37:56.300404+0000","last_became_active":"2024-03-04T07:37:55.126092+0000","last_became_peered":"2024-03-04T07:37:55.126092+0000","last_unstale":"2024-03-04T07:37:56.300404+0000","last_undegraded":"2024-03-04T07:37:56.300404+0000","last_fullsized":"2024-03-04T07:37:56.300404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:47.968250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302789+0000","last_change":"2024-03-04T07:37:55.140265+0000","last_active":"2024-03-04T07:37:56.302789+0000","last_peered":"2024-03-04T07:37:56.302789+0000","last_clean":"2024-03-04T07:37:56.302789+0000","last_became_active":"2024-03-04T07:37:55.139768+0000","last_became_peered":"2024-03-04T07:37:55.139768+0000","last_unstale":"2024-03-04T07:37:56.302789+0000","last_undegraded":"2024-03-04T07:37:56.302789+0000","last_fullsized":"2024-03-04T07:37:56.302789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:52.924188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301086+0000","last_change":"2024-03-04T07:37:55.118929+0000","last_active":"2024-03-04T07:37:56.301086+0000","last_peered":"2024-03-04T07:37:56.301086+0000","last_clean":"2024-03-04T07:37:56.301086+0000","last_became_active":"2024-03-04T07:37:55.118549+0000","last_became_peered":"2024-03-04T07:37:55.118549+0000","last_unstale":"2024-03-04T07:37:56.301086+0000","last_undegraded":"2024-03-04T07:37:56.301086+0000","last_fullsized":"2024-03-04T07:37:56.301086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:15.599080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244954,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":1030695,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":921600,"data_stored":439125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332132,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":966656,"data_stored":1029471,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":917504,"data_stored":431597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":921600,"data_stored":438625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":974848,"data_stored":1029071,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T07:38:02.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T07:38:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 -- 172.21.15.116:0/1314482960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 msgr2=0x7f90e410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 --2- 172.21.15.116:0/1314482960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e410ba60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009ca0 tx=0x7f90dc02c100 comp rx=0 tx=0).stop 2024-03-04T07:38:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 -- 172.21.15.116:0/1314482960 shutdown_connections 2024-03-04T07:38:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 --2- 172.21.15.116:0/1314482960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e410ba60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 --2- 172.21.15.116:0/1314482960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4104810 0x7f90e4104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 --2- 172.21.15.116:0/1314482960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4103620 0x7f90e4103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 -- 172.21.15.116:0/1314482960 >> 172.21.15.116:0/1314482960 conn(0x7f90e40febb0 msgr2=0x7f90e4100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 -- 172.21.15.116:0/1314482960 shutdown_connections 2024-03-04T07:38:03.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.247+0000 7f90e8a00640 1 -- 172.21.15.116:0/1314482960 wait complete. 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 Processor -- start 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 -- start start 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4103620 0x7f90e41998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4104810 0x7f90e4199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e41a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e410f2b0 con 0x7f90e4104810 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e410f130 con 0x7f90e41051d0 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90e8a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e410f430 con 0x7f90e4103620 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.248+0000 7f90dbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4104810 0x7f90e4199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e41a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e41a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60388/0 (socket says 172.21.15.116:60388) 2024-03-04T07:38:03.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 -- 172.21.15.116:0/468185107 learned_addr learned my addr 172.21.15.116:0/468185107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90dbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4104810 0x7f90e4199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37056/0 (socket says 172.21.15.116:37056) 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e2800640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4103620 0x7f90e41998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 -- 172.21.15.116:0/468185107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4103620 msgr2=0x7f90e41998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4103620 0x7f90e41998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 -- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4104810 msgr2=0x7f90e4199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4104810 0x7f90e4199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 -- 172.21.15.116:0/468185107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e41a14e0 con 0x7f90e41051d0 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e2800640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4103620 0x7f90e41998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90dbe00640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4104810 0x7f90e4199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e3200640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e41a0e70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f90dc02c640 tx=0x7f90dc035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90db400640 1 -- 172.21.15.116:0/468185107 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc039040 con 0x7f90e41051d0 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90db400640 1 -- 172.21.15.116:0/468185107 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90dc044070 con 0x7f90e41051d0 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e41a1770 con 0x7f90e41051d0 2024-03-04T07:38:03.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.249+0000 7f90db400640 1 -- 172.21.15.116:0/468185107 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc004460 con 0x7f90e41051d0 2024-03-04T07:38:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.250+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e41a1cb0 con 0x7f90e41051d0 2024-03-04T07:38:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.251+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e410a940 con 0x7f90e41051d0 2024-03-04T07:38:03.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.252+0000 7f90db400640 1 -- 172.21.15.116:0/468185107 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f90dc004600 con 0x7f90e41051d0 2024-03-04T07:38:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.254+0000 7f90db400640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f90c80a1e90 0x7f90c80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.254+0000 7f90db400640 1 -- 172.21.15.116:0/468185107 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f90dc0f0360 con 0x7f90e41051d0 2024-03-04T07:38:03.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.254+0000 7f90e2800640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f90c80a1e90 0x7f90c80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.254+0000 7f90e2800640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f90c80a1e90 0x7f90c80a4350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f90e4104670 tx=0x7f90d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:03.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.258+0000 7f90db400640 1 -- 172.21.15.116:0/468185107 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f90dc0bd870 con 0x7f90e41051d0 2024-03-04T07:38:03.257 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:03 smithi116 ceph-mon[32150]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-04T07:38:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:03 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1774254336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:03.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.451+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90e4061a10 con 0x7f90c80a1e90 2024-03-04T07:38:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.475+0000 7f90db400640 1 -- 172.21.15.116:0/468185107 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258812 (secure 0 0 0) 0x7f90e4061a10 con 0x7f90c80a1e90 2024-03-04T07:38:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:38:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T07:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f90c80a1e90 msgr2=0x7f90c80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f90c80a1e90 0x7f90c80a4350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f90e4104670 tx=0x7f90d4005f90 comp rx=0 tx=0).stop 2024-03-04T07:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 msgr2=0x7f90e41a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e41a0e70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f90dc02c640 tx=0x7f90dc035d10 comp rx=0 tx=0).stop 2024-03-04T07:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 shutdown_connections 2024-03-04T07:38:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90e41051d0 0x7f90e41a0e70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90e4104810 0x7f90e4199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f90c80a1e90 0x7f90c80a4350 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 --2- 172.21.15.116:0/468185107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90e4103620 0x7f90e41998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 >> 172.21.15.116:0/468185107 conn(0x7f90e40febb0 msgr2=0x7f90e4107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 shutdown_connections 2024-03-04T07:38:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:03.481+0000 7f90e8a00640 1 -- 172.21.15.116:0/468185107 wait complete. 2024-03-04T07:38:03.500 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-03-04T07:38:01.466562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":131,"num_read_kb":114,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360168,"kb_used_data":10932,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581656,"statfs":{"total":1151940427776,"available":1151571615744,"internally_reserved":0,"allocated":11194368,"data_stored":7032016,"data_compressed":107094,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.361333"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301431+0000","last_change":"2024-03-04T07:37:55.137540+0000","last_active":"2024-03-04T07:37:56.301431+0000","last_peered":"2024-03-04T07:37:56.301431+0000","last_clean":"2024-03-04T07:37:56.301431+0000","last_became_active":"2024-03-04T07:37:55.137183+0000","last_became_peered":"2024-03-04T07:37:55.137183+0000","last_unstale":"2024-03-04T07:37:56.301431+0000","last_undegraded":"2024-03-04T07:37:56.301431+0000","last_fullsized":"2024-03-04T07:37:56.301431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:15.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301829+0000","last_change":"2024-03-04T07:37:55.136049+0000","last_active":"2024-03-04T07:37:56.301829+0000","last_peered":"2024-03-04T07:37:56.301829+0000","last_clean":"2024-03-04T07:37:56.301829+0000","last_became_active":"2024-03-04T07:37:55.132153+0000","last_became_peered":"2024-03-04T07:37:55.132153+0000","last_unstale":"2024-03-04T07:37:56.301829+0000","last_undegraded":"2024-03-04T07:37:56.301829+0000","last_fullsized":"2024-03-04T07:37:56.301829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:52.426892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302297+0000","last_change":"2024-03-04T07:37:55.133571+0000","last_active":"2024-03-04T07:37:56.302297+0000","last_peered":"2024-03-04T07:37:56.302297+0000","last_clean":"2024-03-04T07:37:56.302297+0000","last_became_active":"2024-03-04T07:37:55.133214+0000","last_became_peered":"2024-03-04T07:37:55.133214+0000","last_unstale":"2024-03-04T07:37:56.302297+0000","last_undegraded":"2024-03-04T07:37:56.302297+0000","last_fullsized":"2024-03-04T07:37:56.302297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:17.924826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300967+0000","last_change":"2024-03-04T07:37:55.118146+0000","last_active":"2024-03-04T07:37:56.300967+0000","last_peered":"2024-03-04T07:37:56.300967+0000","last_clean":"2024-03-04T07:37:56.300967+0000","last_became_active":"2024-03-04T07:37:55.117687+0000","last_became_peered":"2024-03-04T07:37:55.117687+0000","last_unstale":"2024-03-04T07:37:56.300967+0000","last_undegraded":"2024-03-04T07:37:56.300967+0000","last_fullsized":"2024-03-04T07:37:56.300967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:34:35.343400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152180+0000","last_change":"2024-03-04T07:37:55.135943+0000","last_active":"2024-03-04T07:37:56.152180+0000","last_peered":"2024-03-04T07:37:56.152180+0000","last_clean":"2024-03-04T07:37:56.152180+0000","last_became_active":"2024-03-04T07:37:55.135549+0000","last_became_peered":"2024-03-04T07:37:55.135549+0000","last_unstale":"2024-03-04T07:37:56.152180+0000","last_undegraded":"2024-03-04T07:37:56.152180+0000","last_fullsized":"2024-03-04T07:37:56.152180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:35:53.953771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096012+0000","last_change":"2024-03-04T07:37:55.116949+0000","last_active":"2024-03-04T07:37:56.096012+0000","last_peered":"2024-03-04T07:37:56.096012+0000","last_clean":"2024-03-04T07:37:56.096012+0000","last_became_active":"2024-03-04T07:37:55.116620+0000","last_became_peered":"2024-03-04T07:37:55.116620+0000","last_unstale":"2024-03-04T07:37:56.096012+0000","last_undegraded":"2024-03-04T07:37:56.096012+0000","last_fullsized":"2024-03-04T07:37:56.096012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:39.655217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336783+0000","last_change":"2024-03-04T07:37:55.100975+0000","last_active":"2024-03-04T07:37:56.336783+0000","last_peered":"2024-03-04T07:37:56.336783+0000","last_clean":"2024-03-04T07:37:56.336783+0000","last_became_active":"2024-03-04T07:37:55.100636+0000","last_became_peered":"2024-03-04T07:37:55.100636+0000","last_unstale":"2024-03-04T07:37:56.336783+0000","last_undegraded":"2024-03-04T07:37:56.336783+0000","last_fullsized":"2024-03-04T07:37:56.336783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:29.593763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152558+0000","last_change":"2024-03-04T07:37:55.131895+0000","last_active":"2024-03-04T07:37:56.152558+0000","last_peered":"2024-03-04T07:37:56.152558+0000","last_clean":"2024-03-04T07:37:56.152558+0000","last_became_active":"2024-03-04T07:37:55.129682+0000","last_became_peered":"2024-03-04T07:37:55.129682+0000","last_unstale":"2024-03-04T07:37:56.152558+0000","last_undegraded":"2024-03-04T07:37:56.152558+0000","last_fullsized":"2024-03-04T07:37:56.152558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:54.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300914+0000","last_change":"2024-03-04T07:37:55.114475+0000","last_active":"2024-03-04T07:37:56.300914+0000","last_peered":"2024-03-04T07:37:56.300914+0000","last_clean":"2024-03-04T07:37:56.300914+0000","last_became_active":"2024-03-04T07:37:55.114068+0000","last_became_peered":"2024-03-04T07:37:55.114068+0000","last_unstale":"2024-03-04T07:37:56.300914+0000","last_undegraded":"2024-03-04T07:37:56.300914+0000","last_fullsized":"2024-03-04T07:37:56.300914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:21.466143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096584+0000","last_change":"2024-03-04T07:37:55.115660+0000","last_active":"2024-03-04T07:37:56.096584+0000","last_peered":"2024-03-04T07:37:56.096584+0000","last_clean":"2024-03-04T07:37:56.096584+0000","last_became_active":"2024-03-04T07:37:55.115324+0000","last_became_peered":"2024-03-04T07:37:55.115324+0000","last_unstale":"2024-03-04T07:37:56.096584+0000","last_undegraded":"2024-03-04T07:37:56.096584+0000","last_fullsized":"2024-03-04T07:37:56.096584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:42.119728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337102+0000","last_change":"2024-03-04T07:37:55.119834+0000","last_active":"2024-03-04T07:37:56.337102+0000","last_peered":"2024-03-04T07:37:56.337102+0000","last_clean":"2024-03-04T07:37:56.337102+0000","last_became_active":"2024-03-04T07:37:55.119467+0000","last_became_peered":"2024-03-04T07:37:55.119467+0000","last_unstale":"2024-03-04T07:37:56.337102+0000","last_undegraded":"2024-03-04T07:37:56.337102+0000","last_fullsized":"2024-03-04T07:37:56.337102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:49:19.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300908+0000","last_change":"2024-03-04T07:37:55.118154+0000","last_active":"2024-03-04T07:37:56.300908+0000","last_peered":"2024-03-04T07:37:56.300908+0000","last_clean":"2024-03-04T07:37:56.300908+0000","last_became_active":"2024-03-04T07:37:55.117809+0000","last_became_peered":"2024-03-04T07:37:55.117809+0000","last_unstale":"2024-03-04T07:37:56.300908+0000","last_undegraded":"2024-03-04T07:37:56.300908+0000","last_fullsized":"2024-03-04T07:37:56.300908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:50.195946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302928+0000","last_change":"2024-03-04T07:37:55.123325+0000","last_active":"2024-03-04T07:37:56.302928+0000","last_peered":"2024-03-04T07:37:56.302928+0000","last_clean":"2024-03-04T07:37:56.302928+0000","last_became_active":"2024-03-04T07:37:55.122848+0000","last_became_peered":"2024-03-04T07:37:55.122848+0000","last_unstale":"2024-03-04T07:37:56.302928+0000","last_undegraded":"2024-03-04T07:37:56.302928+0000","last_fullsized":"2024-03-04T07:37:56.302928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:18.842011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302409+0000","last_change":"2024-03-04T07:37:55.132447+0000","last_active":"2024-03-04T07:37:56.302409+0000","last_peered":"2024-03-04T07:37:56.302409+0000","last_clean":"2024-03-04T07:37:56.302409+0000","last_became_active":"2024-03-04T07:37:55.131512+0000","last_became_peered":"2024-03-04T07:37:55.131512+0000","last_unstale":"2024-03-04T07:37:56.302409+0000","last_undegraded":"2024-03-04T07:37:56.302409+0000","last_fullsized":"2024-03-04T07:37:56.302409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:23.960029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301583+0000","last_change":"2024-03-04T07:37:55.131715+0000","last_active":"2024-03-04T07:37:56.301583+0000","last_peered":"2024-03-04T07:37:56.301583+0000","last_clean":"2024-03-04T07:37:56.301583+0000","last_became_active":"2024-03-04T07:37:55.131326+0000","last_became_peered":"2024-03-04T07:37:55.131326+0000","last_unstale":"2024-03-04T07:37:56.301583+0000","last_undegraded":"2024-03-04T07:37:56.301583+0000","last_fullsized":"2024-03-04T07:37:56.301583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:56.231578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300272+0000","last_change":"2024-03-04T07:37:55.137395+0000","last_active":"2024-03-04T07:37:56.300272+0000","last_peered":"2024-03-04T07:37:56.300272+0000","last_clean":"2024-03-04T07:37:56.300272+0000","last_became_active":"2024-03-04T07:37:55.137054+0000","last_became_peered":"2024-03-04T07:37:55.137054+0000","last_unstale":"2024-03-04T07:37:56.300272+0000","last_undegraded":"2024-03-04T07:37:56.300272+0000","last_fullsized":"2024-03-04T07:37:56.300272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:19.222481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301716+0000","last_change":"2024-03-04T07:37:55.137414+0000","last_active":"2024-03-04T07:37:56.301716+0000","last_peered":"2024-03-04T07:37:56.301716+0000","last_clean":"2024-03-04T07:37:56.301716+0000","last_became_active":"2024-03-04T07:37:55.137157+0000","last_became_peered":"2024-03-04T07:37:55.137157+0000","last_unstale":"2024-03-04T07:37:56.301716+0000","last_undegraded":"2024-03-04T07:37:56.301716+0000","last_fullsized":"2024-03-04T07:37:56.301716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:36.593491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096514+0000","last_change":"2024-03-04T07:37:55.121960+0000","last_active":"2024-03-04T07:37:56.096514+0000","last_peered":"2024-03-04T07:37:56.096514+0000","last_clean":"2024-03-04T07:37:56.096514+0000","last_became_active":"2024-03-04T07:37:55.115543+0000","last_became_peered":"2024-03-04T07:37:55.115543+0000","last_unstale":"2024-03-04T07:37:56.096514+0000","last_undegraded":"2024-03-04T07:37:56.096514+0000","last_fullsized":"2024-03-04T07:37:56.096514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:36.320475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096321+0000","last_change":"2024-03-04T07:37:55.128908+0000","last_active":"2024-03-04T07:37:56.096321+0000","last_peered":"2024-03-04T07:37:56.096321+0000","last_clean":"2024-03-04T07:37:56.096321+0000","last_became_active":"2024-03-04T07:37:55.128500+0000","last_became_peered":"2024-03-04T07:37:55.128500+0000","last_unstale":"2024-03-04T07:37:56.096321+0000","last_undegraded":"2024-03-04T07:37:56.096321+0000","last_fullsized":"2024-03-04T07:37:56.096321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:10.598424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096219+0000","last_change":"2024-03-04T07:37:54.121610+0000","last_active":"2024-03-04T07:37:56.096219+0000","last_peered":"2024-03-04T07:37:56.096219+0000","last_clean":"2024-03-04T07:37:56.096219+0000","last_became_active":"2024-03-04T07:37:54.121150+0000","last_became_peered":"2024-03-04T07:37:54.121150+0000","last_unstale":"2024-03-04T07:37:56.096219+0000","last_undegraded":"2024-03-04T07:37:56.096219+0000","last_fullsized":"2024-03-04T07:37:56.096219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:05:03.965626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154034+0000","last_change":"2024-03-04T07:37:55.138904+0000","last_active":"2024-03-04T07:37:56.154034+0000","last_peered":"2024-03-04T07:37:56.154034+0000","last_clean":"2024-03-04T07:37:56.154034+0000","last_became_active":"2024-03-04T07:37:55.138376+0000","last_became_peered":"2024-03-04T07:37:55.138376+0000","last_unstale":"2024-03-04T07:37:56.154034+0000","last_undegraded":"2024-03-04T07:37:56.154034+0000","last_fullsized":"2024-03-04T07:37:56.154034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:28.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154617+0000","last_change":"2024-03-04T07:37:54.139218+0000","last_active":"2024-03-04T07:37:56.154617+0000","last_peered":"2024-03-04T07:37:56.154617+0000","last_clean":"2024-03-04T07:37:56.154617+0000","last_became_active":"2024-03-04T07:37:54.138893+0000","last_became_peered":"2024-03-04T07:37:54.138893+0000","last_unstale":"2024-03-04T07:37:56.154617+0000","last_undegraded":"2024-03-04T07:37:56.154617+0000","last_fullsized":"2024-03-04T07:37:56.154617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:01.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302247+0000","last_change":"2024-03-04T07:37:55.135953+0000","last_active":"2024-03-04T07:37:56.302247+0000","last_peered":"2024-03-04T07:37:56.302247+0000","last_clean":"2024-03-04T07:37:56.302247+0000","last_became_active":"2024-03-04T07:37:55.132880+0000","last_became_peered":"2024-03-04T07:37:55.132880+0000","last_unstale":"2024-03-04T07:37:56.302247+0000","last_undegraded":"2024-03-04T07:37:56.302247+0000","last_fullsized":"2024-03-04T07:37:56.302247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:03.848428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152629+0000","last_change":"2024-03-04T07:37:54.122493+0000","last_active":"2024-03-04T07:37:56.152629+0000","last_peered":"2024-03-04T07:37:56.152629+0000","last_clean":"2024-03-04T07:37:56.152629+0000","last_became_active":"2024-03-04T07:37:54.122058+0000","last_became_peered":"2024-03-04T07:37:54.122058+0000","last_unstale":"2024-03-04T07:37:56.152629+0000","last_undegraded":"2024-03-04T07:37:56.152629+0000","last_fullsized":"2024-03-04T07:37:56.152629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:02.516175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300596+0000","last_change":"2024-03-04T07:37:55.136155+0000","last_active":"2024-03-04T07:37:56.300596+0000","last_peered":"2024-03-04T07:37:56.300596+0000","last_clean":"2024-03-04T07:37:56.300596+0000","last_became_active":"2024-03-04T07:37:55.135796+0000","last_became_peered":"2024-03-04T07:37:55.135796+0000","last_unstale":"2024-03-04T07:37:56.300596+0000","last_undegraded":"2024-03-04T07:37:56.300596+0000","last_fullsized":"2024-03-04T07:37:56.300596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:37.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152415+0000","last_change":"2024-03-04T07:37:54.118780+0000","last_active":"2024-03-04T07:37:56.152415+0000","last_peered":"2024-03-04T07:37:56.152415+0000","last_clean":"2024-03-04T07:37:56.152415+0000","last_became_active":"2024-03-04T07:37:54.118448+0000","last_became_peered":"2024-03-04T07:37:54.118448+0000","last_unstale":"2024-03-04T07:37:56.152415+0000","last_undegraded":"2024-03-04T07:37:56.152415+0000","last_fullsized":"2024-03-04T07:37:56.152415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:25.547046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301590+0000","last_change":"2024-03-04T07:37:55.120991+0000","last_active":"2024-03-04T07:37:56.301590+0000","last_peered":"2024-03-04T07:37:56.301590+0000","last_clean":"2024-03-04T07:37:56.301590+0000","last_became_active":"2024-03-04T07:37:55.120626+0000","last_became_peered":"2024-03-04T07:37:55.120626+0000","last_unstale":"2024-03-04T07:37:56.301590+0000","last_undegraded":"2024-03-04T07:37:56.301590+0000","last_fullsized":"2024-03-04T07:37:56.301590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:55:41.749205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301257+0000","last_change":"2024-03-04T07:37:54.122241+0000","last_active":"2024-03-04T07:37:56.301257+0000","last_peered":"2024-03-04T07:37:56.301257+0000","last_clean":"2024-03-04T07:37:56.301257+0000","last_became_active":"2024-03-04T07:37:54.120990+0000","last_became_peered":"2024-03-04T07:37:54.120990+0000","last_unstale":"2024-03-04T07:37:56.301257+0000","last_undegraded":"2024-03-04T07:37:56.301257+0000","last_fullsized":"2024-03-04T07:37:56.301257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:03.233754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152226+0000","last_change":"2024-03-04T07:37:55.121692+0000","last_active":"2024-03-04T07:37:56.152226+0000","last_peered":"2024-03-04T07:37:56.152226+0000","last_clean":"2024-03-04T07:37:56.152226+0000","last_became_active":"2024-03-04T07:37:55.121345+0000","last_became_peered":"2024-03-04T07:37:55.121345+0000","last_unstale":"2024-03-04T07:37:56.152226+0000","last_undegraded":"2024-03-04T07:37:56.152226+0000","last_fullsized":"2024-03-04T07:37:56.152226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:52:42.045265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.095944+0000","last_change":"2024-03-04T07:37:54.133489+0000","last_active":"2024-03-04T07:37:56.095944+0000","last_peered":"2024-03-04T07:37:56.095944+0000","last_clean":"2024-03-04T07:37:56.095944+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:37:56.095944+0000","last_undegraded":"2024-03-04T07:37:56.095944+0000","last_fullsized":"2024-03-04T07:37:56.095944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:49:03.600907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301458+0000","last_change":"2024-03-04T07:37:55.128278+0000","last_active":"2024-03-04T07:37:56.301458+0000","last_peered":"2024-03-04T07:37:56.301458+0000","last_clean":"2024-03-04T07:37:56.301458+0000","last_became_active":"2024-03-04T07:37:55.127913+0000","last_became_peered":"2024-03-04T07:37:55.127913+0000","last_unstale":"2024-03-04T07:37:56.301458+0000","last_undegraded":"2024-03-04T07:37:56.301458+0000","last_fullsized":"2024-03-04T07:37:56.301458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:54:18.623855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302158+0000","last_change":"2024-03-04T07:37:54.132195+0000","last_active":"2024-03-04T07:37:56.302158+0000","last_peered":"2024-03-04T07:37:56.302158+0000","last_clean":"2024-03-04T07:37:56.302158+0000","last_became_active":"2024-03-04T07:37:54.131830+0000","last_became_peered":"2024-03-04T07:37:54.131830+0000","last_unstale":"2024-03-04T07:37:56.302158+0000","last_undegraded":"2024-03-04T07:37:56.302158+0000","last_fullsized":"2024-03-04T07:37:56.302158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.744342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153043+0000","last_change":"2024-03-04T07:37:55.138281+0000","last_active":"2024-03-04T07:37:56.153043+0000","last_peered":"2024-03-04T07:37:56.153043+0000","last_clean":"2024-03-04T07:37:56.153043+0000","last_became_active":"2024-03-04T07:37:55.137919+0000","last_became_peered":"2024-03-04T07:37:55.137919+0000","last_unstale":"2024-03-04T07:37:56.153043+0000","last_undegraded":"2024-03-04T07:37:56.153043+0000","last_fullsized":"2024-03-04T07:37:56.153043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:53:06.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300702+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:37:56.300702+0000","last_peered":"2024-03-04T07:37:56.300702+0000","last_clean":"2024-03-04T07:37:56.300702+0000","last_became_active":"2024-03-04T07:37:54.121503+0000","last_became_peered":"2024-03-04T07:37:54.121503+0000","last_unstale":"2024-03-04T07:37:56.300702+0000","last_undegraded":"2024-03-04T07:37:56.300702+0000","last_fullsized":"2024-03-04T07:37:56.300702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:03:07.531246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096196+0000","last_change":"2024-03-04T07:37:55.126924+0000","last_active":"2024-03-04T07:37:56.096196+0000","last_peered":"2024-03-04T07:37:56.096196+0000","last_clean":"2024-03-04T07:37:56.096196+0000","last_became_active":"2024-03-04T07:37:55.126266+0000","last_became_peered":"2024-03-04T07:37:55.126266+0000","last_unstale":"2024-03-04T07:37:56.096196+0000","last_undegraded":"2024-03-04T07:37:56.096196+0000","last_fullsized":"2024-03-04T07:37:56.096196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:01.603430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096500+0000","last_change":"2024-03-04T07:37:54.119864+0000","last_active":"2024-03-04T07:37:56.096500+0000","last_peered":"2024-03-04T07:37:56.096500+0000","last_clean":"2024-03-04T07:37:56.096500+0000","last_became_active":"2024-03-04T07:37:54.119515+0000","last_became_peered":"2024-03-04T07:37:54.119515+0000","last_unstale":"2024-03-04T07:37:56.096500+0000","last_undegraded":"2024-03-04T07:37:56.096500+0000","last_fullsized":"2024-03-04T07:37:56.096500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:10.422458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300769+0000","last_change":"2024-03-04T07:37:55.120135+0000","last_active":"2024-03-04T07:37:56.300769+0000","last_peered":"2024-03-04T07:37:56.300769+0000","last_clean":"2024-03-04T07:37:56.300769+0000","last_became_active":"2024-03-04T07:37:55.119670+0000","last_became_peered":"2024-03-04T07:37:55.119670+0000","last_unstale":"2024-03-04T07:37:56.300769+0000","last_undegraded":"2024-03-04T07:37:56.300769+0000","last_fullsized":"2024-03-04T07:37:56.300769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:09.688447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337257+0000","last_change":"2024-03-04T07:37:54.120222+0000","last_active":"2024-03-04T07:37:56.337257+0000","last_peered":"2024-03-04T07:37:56.337257+0000","last_clean":"2024-03-04T07:37:56.337257+0000","last_became_active":"2024-03-04T07:37:54.119802+0000","last_became_peered":"2024-03-04T07:37:54.119802+0000","last_unstale":"2024-03-04T07:37:56.337257+0000","last_undegraded":"2024-03-04T07:37:56.337257+0000","last_fullsized":"2024-03-04T07:37:56.337257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:12:40.811870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302038+0000","last_change":"2024-03-04T07:37:55.136371+0000","last_active":"2024-03-04T07:37:56.302038+0000","last_peered":"2024-03-04T07:37:56.302038+0000","last_clean":"2024-03-04T07:37:56.302038+0000","last_became_active":"2024-03-04T07:37:55.132587+0000","last_became_peered":"2024-03-04T07:37:55.132587+0000","last_unstale":"2024-03-04T07:37:56.302038+0000","last_undegraded":"2024-03-04T07:37:56.302038+0000","last_fullsized":"2024-03-04T07:37:56.302038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:24.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301599+0000","last_change":"2024-03-04T07:37:54.123041+0000","last_active":"2024-03-04T07:37:56.301599+0000","last_peered":"2024-03-04T07:37:56.301599+0000","last_clean":"2024-03-04T07:37:56.301599+0000","last_became_active":"2024-03-04T07:37:54.122723+0000","last_became_peered":"2024-03-04T07:37:54.122723+0000","last_unstale":"2024-03-04T07:37:56.301599+0000","last_undegraded":"2024-03-04T07:37:56.301599+0000","last_fullsized":"2024-03-04T07:37:56.301599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:06:30.845376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300896+0000","last_change":"2024-03-04T07:37:55.115387+0000","last_active":"2024-03-04T07:37:56.300896+0000","last_peered":"2024-03-04T07:37:56.300896+0000","last_clean":"2024-03-04T07:37:56.300896+0000","last_became_active":"2024-03-04T07:37:55.114947+0000","last_became_peered":"2024-03-04T07:37:55.114947+0000","last_unstale":"2024-03-04T07:37:56.300896+0000","last_undegraded":"2024-03-04T07:37:56.300896+0000","last_fullsized":"2024-03-04T07:37:56.300896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:09:11.840989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"84'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096303+0000","last_change":"2024-03-04T07:37:54.117767+0000","last_active":"2024-03-04T07:37:56.096303+0000","last_peered":"2024-03-04T07:37:56.096303+0000","last_clean":"2024-03-04T07:37:56.096303+0000","last_became_active":"2024-03-04T07:37:54.117371+0000","last_became_peered":"2024-03-04T07:37:54.117371+0000","last_unstale":"2024-03-04T07:37:56.096303+0000","last_undegraded":"2024-03-04T07:37:56.096303+0000","last_fullsized":"2024-03-04T07:37:56.096303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:08:32.164636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152426+0000","last_change":"2024-03-04T07:37:55.112707+0000","last_active":"2024-03-04T07:37:56.152426+0000","last_peered":"2024-03-04T07:37:56.152426+0000","last_clean":"2024-03-04T07:37:56.152426+0000","last_became_active":"2024-03-04T07:37:55.112327+0000","last_became_peered":"2024-03-04T07:37:55.112327+0000","last_unstale":"2024-03-04T07:37:56.152426+0000","last_undegraded":"2024-03-04T07:37:56.152426+0000","last_fullsized":"2024-03-04T07:37:56.152426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:03.901687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096229+0000","last_change":"2024-03-04T07:37:54.125286+0000","last_active":"2024-03-04T07:37:56.096229+0000","last_peered":"2024-03-04T07:37:56.096229+0000","last_clean":"2024-03-04T07:37:56.096229+0000","last_became_active":"2024-03-04T07:37:54.124966+0000","last_became_peered":"2024-03-04T07:37:54.124966+0000","last_unstale":"2024-03-04T07:37:56.096229+0000","last_undegraded":"2024-03-04T07:37:56.096229+0000","last_fullsized":"2024-03-04T07:37:56.096229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:19.880681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302449+0000","last_change":"2024-03-04T07:37:55.137952+0000","last_active":"2024-03-04T07:37:56.302449+0000","last_peered":"2024-03-04T07:37:56.302449+0000","last_clean":"2024-03-04T07:37:56.302449+0000","last_became_active":"2024-03-04T07:37:55.137583+0000","last_became_peered":"2024-03-04T07:37:55.137583+0000","last_unstale":"2024-03-04T07:37:56.302449+0000","last_undegraded":"2024-03-04T07:37:56.302449+0000","last_fullsized":"2024-03-04T07:37:56.302449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:30.145453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301263+0000","last_change":"2024-03-04T07:37:54.134733+0000","last_active":"2024-03-04T07:37:56.301263+0000","last_peered":"2024-03-04T07:37:56.301263+0000","last_clean":"2024-03-04T07:37:56.301263+0000","last_became_active":"2024-03-04T07:37:54.134400+0000","last_became_peered":"2024-03-04T07:37:54.134400+0000","last_unstale":"2024-03-04T07:37:56.301263+0000","last_undegraded":"2024-03-04T07:37:56.301263+0000","last_fullsized":"2024-03-04T07:37:56.301263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:24.192479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337082+0000","last_change":"2024-03-04T07:37:55.124991+0000","last_active":"2024-03-04T07:37:56.337082+0000","last_peered":"2024-03-04T07:37:56.337082+0000","last_clean":"2024-03-04T07:37:56.337082+0000","last_became_active":"2024-03-04T07:37:55.124621+0000","last_became_peered":"2024-03-04T07:37:55.124621+0000","last_unstale":"2024-03-04T07:37:56.337082+0000","last_undegraded":"2024-03-04T07:37:56.337082+0000","last_fullsized":"2024-03-04T07:37:56.337082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:05.252145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300536+0000","last_change":"2024-03-04T07:37:54.118743+0000","last_active":"2024-03-04T07:37:56.300536+0000","last_peered":"2024-03-04T07:37:56.300536+0000","last_clean":"2024-03-04T07:37:56.300536+0000","last_became_active":"2024-03-04T07:37:54.118386+0000","last_became_peered":"2024-03-04T07:37:54.118386+0000","last_unstale":"2024-03-04T07:37:56.300536+0000","last_undegraded":"2024-03-04T07:37:56.300536+0000","last_fullsized":"2024-03-04T07:37:56.300536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:27.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336766+0000","last_change":"2024-03-04T07:37:55.115431+0000","last_active":"2024-03-04T07:37:56.336766+0000","last_peered":"2024-03-04T07:37:56.336766+0000","last_clean":"2024-03-04T07:37:56.336766+0000","last_became_active":"2024-03-04T07:37:55.115064+0000","last_became_peered":"2024-03-04T07:37:55.115064+0000","last_unstale":"2024-03-04T07:37:56.336766+0000","last_undegraded":"2024-03-04T07:37:56.336766+0000","last_fullsized":"2024-03-04T07:37:56.336766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:46:57.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152041+0000","last_change":"2024-03-04T07:37:54.118352+0000","last_active":"2024-03-04T07:37:56.152041+0000","last_peered":"2024-03-04T07:37:56.152041+0000","last_clean":"2024-03-04T07:37:56.152041+0000","last_became_active":"2024-03-04T07:37:54.117981+0000","last_became_peered":"2024-03-04T07:37:54.117981+0000","last_unstale":"2024-03-04T07:37:56.152041+0000","last_undegraded":"2024-03-04T07:37:56.152041+0000","last_fullsized":"2024-03-04T07:37:56.152041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:18:54.949946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300862+0000","last_change":"2024-03-04T07:37:55.114882+0000","last_active":"2024-03-04T07:37:56.300862+0000","last_peered":"2024-03-04T07:37:56.300862+0000","last_clean":"2024-03-04T07:37:56.300862+0000","last_became_active":"2024-03-04T07:37:55.113139+0000","last_became_peered":"2024-03-04T07:37:55.113139+0000","last_unstale":"2024-03-04T07:37:56.300862+0000","last_undegraded":"2024-03-04T07:37:56.300862+0000","last_fullsized":"2024-03-04T07:37:56.300862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:28.133710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152593+0000","last_change":"2024-03-04T07:37:54.119350+0000","last_active":"2024-03-04T07:37:56.152593+0000","last_peered":"2024-03-04T07:37:56.152593+0000","last_clean":"2024-03-04T07:37:56.152593+0000","last_became_active":"2024-03-04T07:37:54.119017+0000","last_became_peered":"2024-03-04T07:37:54.119017+0000","last_unstale":"2024-03-04T07:37:56.152593+0000","last_undegraded":"2024-03-04T07:37:56.152593+0000","last_fullsized":"2024-03-04T07:37:56.152593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:00.503583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301290+0000","last_change":"2024-03-04T07:37:55.114242+0000","last_active":"2024-03-04T07:37:56.301290+0000","last_peered":"2024-03-04T07:37:56.301290+0000","last_clean":"2024-03-04T07:37:56.301290+0000","last_became_active":"2024-03-04T07:37:55.113852+0000","last_became_peered":"2024-03-04T07:37:55.113852+0000","last_unstale":"2024-03-04T07:37:56.301290+0000","last_undegraded":"2024-03-04T07:37:56.301290+0000","last_fullsized":"2024-03-04T07:37:56.301290+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:02:30.804894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096019+0000","last_change":"2024-03-04T07:37:54.125548+0000","last_active":"2024-03-04T07:37:56.096019+0000","last_peered":"2024-03-04T07:37:56.096019+0000","last_clean":"2024-03-04T07:37:56.096019+0000","last_became_active":"2024-03-04T07:37:54.125127+0000","last_became_peered":"2024-03-04T07:37:54.125127+0000","last_unstale":"2024-03-04T07:37:56.096019+0000","last_undegraded":"2024-03-04T07:37:56.096019+0000","last_fullsized":"2024-03-04T07:37:56.096019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:26.256247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096432+0000","last_change":"2024-03-04T07:37:55.128542+0000","last_active":"2024-03-04T07:37:56.096432+0000","last_peered":"2024-03-04T07:37:56.096432+0000","last_clean":"2024-03-04T07:37:56.096432+0000","last_became_active":"2024-03-04T07:37:55.128239+0000","last_became_peered":"2024-03-04T07:37:55.128239+0000","last_unstale":"2024-03-04T07:37:56.096432+0000","last_undegraded":"2024-03-04T07:37:56.096432+0000","last_fullsized":"2024-03-04T07:37:56.096432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:17.078095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336884+0000","last_change":"2024-03-04T07:37:54.133672+0000","last_active":"2024-03-04T07:37:56.336884+0000","last_peered":"2024-03-04T07:37:56.336884+0000","last_clean":"2024-03-04T07:37:56.336884+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:37:56.336884+0000","last_undegraded":"2024-03-04T07:37:56.336884+0000","last_fullsized":"2024-03-04T07:37:56.336884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:52.345158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096226+0000","last_change":"2024-03-04T07:37:55.137673+0000","last_active":"2024-03-04T07:37:56.096226+0000","last_peered":"2024-03-04T07:37:56.096226+0000","last_clean":"2024-03-04T07:37:56.096226+0000","last_became_active":"2024-03-04T07:37:55.137317+0000","last_became_peered":"2024-03-04T07:37:55.137317+0000","last_unstale":"2024-03-04T07:37:56.096226+0000","last_undegraded":"2024-03-04T07:37:56.096226+0000","last_fullsized":"2024-03-04T07:37:56.096226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:52.683889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301040+0000","last_change":"2024-03-04T07:37:55.112643+0000","last_active":"2024-03-04T07:37:56.301040+0000","last_peered":"2024-03-04T07:37:56.301040+0000","last_clean":"2024-03-04T07:37:56.301040+0000","last_became_active":"2024-03-04T07:37:55.112223+0000","last_became_peered":"2024-03-04T07:37:55.112223+0000","last_unstale":"2024-03-04T07:37:56.301040+0000","last_undegraded":"2024-03-04T07:37:56.301040+0000","last_fullsized":"2024-03-04T07:37:56.301040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:02:49.701226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301533+0000","last_change":"2024-03-04T07:37:54.139335+0000","last_active":"2024-03-04T07:37:56.301533+0000","last_peered":"2024-03-04T07:37:56.301533+0000","last_clean":"2024-03-04T07:37:56.301533+0000","last_became_active":"2024-03-04T07:37:54.138969+0000","last_became_peered":"2024-03-04T07:37:54.138969+0000","last_unstale":"2024-03-04T07:37:56.301533+0000","last_undegraded":"2024-03-04T07:37:56.301533+0000","last_fullsized":"2024-03-04T07:37:56.301533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:41:15.204738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154758+0000","last_change":"2024-03-04T07:37:55.133575+0000","last_active":"2024-03-04T07:37:56.154758+0000","last_peered":"2024-03-04T07:37:56.154758+0000","last_clean":"2024-03-04T07:37:56.154758+0000","last_became_active":"2024-03-04T07:37:55.133220+0000","last_became_peered":"2024-03-04T07:37:55.133220+0000","last_unstale":"2024-03-04T07:37:56.154758+0000","last_undegraded":"2024-03-04T07:37:56.154758+0000","last_fullsized":"2024-03-04T07:37:56.154758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:21:12.835718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300803+0000","last_change":"2024-03-04T07:37:54.119906+0000","last_active":"2024-03-04T07:37:56.300803+0000","last_peered":"2024-03-04T07:37:56.300803+0000","last_clean":"2024-03-04T07:37:56.300803+0000","last_became_active":"2024-03-04T07:37:54.119592+0000","last_became_peered":"2024-03-04T07:37:54.119592+0000","last_unstale":"2024-03-04T07:37:56.300803+0000","last_undegraded":"2024-03-04T07:37:56.300803+0000","last_fullsized":"2024-03-04T07:37:56.300803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:29.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":338,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336764+0000","last_change":"2024-03-04T07:36:35.278528+0000","last_active":"2024-03-04T07:37:56.336764+0000","last_peered":"2024-03-04T07:37:56.336764+0000","last_clean":"2024-03-04T07:37:56.336764+0000","last_became_active":"2024-03-04T07:36:35.277953+0000","last_became_peered":"2024-03-04T07:36:35.277953+0000","last_unstale":"2024-03-04T07:37:56.336764+0000","last_undegraded":"2024-03-04T07:37:56.336764+0000","last_fullsized":"2024-03-04T07:37:56.336764+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_clean_scrub_stamp":"2024-03-04T07:32:42.283525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:58:47.223527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301624+0000","last_change":"2024-03-04T07:37:55.124204+0000","last_active":"2024-03-04T07:37:56.301624+0000","last_peered":"2024-03-04T07:37:56.301624+0000","last_clean":"2024-03-04T07:37:56.301624+0000","last_became_active":"2024-03-04T07:37:55.123863+0000","last_became_peered":"2024-03-04T07:37:55.123863+0000","last_unstale":"2024-03-04T07:37:56.301624+0000","last_undegraded":"2024-03-04T07:37:56.301624+0000","last_fullsized":"2024-03-04T07:37:56.301624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:33.830367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.336915+0000","last_change":"2024-03-04T07:37:54.121538+0000","last_active":"2024-03-04T07:37:56.336915+0000","last_peered":"2024-03-04T07:37:56.336915+0000","last_clean":"2024-03-04T07:37:56.336915+0000","last_became_active":"2024-03-04T07:37:54.120939+0000","last_became_peered":"2024-03-04T07:37:54.120939+0000","last_unstale":"2024-03-04T07:37:56.336915+0000","last_undegraded":"2024-03-04T07:37:56.336915+0000","last_fullsized":"2024-03-04T07:37:56.336915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:29.597247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152443+0000","last_change":"2024-03-04T07:37:54.128180+0000","last_active":"2024-03-04T07:37:56.152443+0000","last_peered":"2024-03-04T07:37:56.152443+0000","last_clean":"2024-03-04T07:37:56.152443+0000","last_became_active":"2024-03-04T07:37:54.127800+0000","last_became_peered":"2024-03-04T07:37:54.127800+0000","last_unstale":"2024-03-04T07:37:56.152443+0000","last_undegraded":"2024-03-04T07:37:56.152443+0000","last_fullsized":"2024-03-04T07:37:56.152443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:54.046873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302961+0000","last_change":"2024-03-04T07:37:55.133482+0000","last_active":"2024-03-04T07:37:56.302961+0000","last_peered":"2024-03-04T07:37:56.302961+0000","last_clean":"2024-03-04T07:37:56.302961+0000","last_became_active":"2024-03-04T07:37:55.133137+0000","last_became_peered":"2024-03-04T07:37:55.133137+0000","last_unstale":"2024-03-04T07:37:56.302961+0000","last_undegraded":"2024-03-04T07:37:56.302961+0000","last_fullsized":"2024-03-04T07:37:56.302961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:53.509564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096540+0000","last_change":"2024-03-04T07:37:54.134294+0000","last_active":"2024-03-04T07:37:56.096540+0000","last_peered":"2024-03-04T07:37:56.096540+0000","last_clean":"2024-03-04T07:37:56.096540+0000","last_became_active":"2024-03-04T07:37:54.133677+0000","last_became_peered":"2024-03-04T07:37:54.133677+0000","last_unstale":"2024-03-04T07:37:56.096540+0000","last_undegraded":"2024-03-04T07:37:56.096540+0000","last_fullsized":"2024-03-04T07:37:56.096540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:57:54.082419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.154045+0000","last_change":"2024-03-04T07:37:55.138305+0000","last_active":"2024-03-04T07:37:56.154045+0000","last_peered":"2024-03-04T07:37:56.154045+0000","last_clean":"2024-03-04T07:37:56.154045+0000","last_became_active":"2024-03-04T07:37:55.129937+0000","last_became_peered":"2024-03-04T07:37:55.129937+0000","last_unstale":"2024-03-04T07:37:56.154045+0000","last_undegraded":"2024-03-04T07:37:56.154045+0000","last_fullsized":"2024-03-04T07:37:56.154045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:45.136539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300446+0000","last_change":"2024-03-04T07:37:54.133817+0000","last_active":"2024-03-04T07:37:56.300446+0000","last_peered":"2024-03-04T07:37:56.300446+0000","last_clean":"2024-03-04T07:37:56.300446+0000","last_became_active":"2024-03-04T07:37:54.133471+0000","last_became_peered":"2024-03-04T07:37:54.133471+0000","last_unstale":"2024-03-04T07:37:56.300446+0000","last_undegraded":"2024-03-04T07:37:56.300446+0000","last_fullsized":"2024-03-04T07:37:56.300446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:00:51.483859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302030+0000","last_change":"2024-03-04T07:37:55.144639+0000","last_active":"2024-03-04T07:37:56.302030+0000","last_peered":"2024-03-04T07:37:56.302030+0000","last_clean":"2024-03-04T07:37:56.302030+0000","last_became_active":"2024-03-04T07:37:55.144194+0000","last_became_peered":"2024-03-04T07:37:55.144194+0000","last_unstale":"2024-03-04T07:37:56.302030+0000","last_undegraded":"2024-03-04T07:37:56.302030+0000","last_fullsized":"2024-03-04T07:37:56.302030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:42.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300192+0000","last_change":"2024-03-04T07:37:54.119871+0000","last_active":"2024-03-04T07:37:56.300192+0000","last_peered":"2024-03-04T07:37:56.300192+0000","last_clean":"2024-03-04T07:37:56.300192+0000","last_became_active":"2024-03-04T07:37:54.119432+0000","last_became_peered":"2024-03-04T07:37:54.119432+0000","last_unstale":"2024-03-04T07:37:56.300192+0000","last_undegraded":"2024-03-04T07:37:56.300192+0000","last_fullsized":"2024-03-04T07:37:56.300192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:01:33.604940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152131+0000","last_change":"2024-03-04T07:37:55.115145+0000","last_active":"2024-03-04T07:37:56.152131+0000","last_peered":"2024-03-04T07:37:56.152131+0000","last_clean":"2024-03-04T07:37:56.152131+0000","last_became_active":"2024-03-04T07:37:55.114791+0000","last_became_peered":"2024-03-04T07:37:55.114791+0000","last_unstale":"2024-03-04T07:37:56.152131+0000","last_undegraded":"2024-03-04T07:37:56.152131+0000","last_fullsized":"2024-03-04T07:37:56.152131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:19.685977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153141+0000","last_change":"2024-03-04T07:37:54.141630+0000","last_active":"2024-03-04T07:37:56.153141+0000","last_peered":"2024-03-04T07:37:56.153141+0000","last_clean":"2024-03-04T07:37:56.153141+0000","last_became_active":"2024-03-04T07:37:54.139820+0000","last_became_peered":"2024-03-04T07:37:54.139820+0000","last_unstale":"2024-03-04T07:37:56.153141+0000","last_undegraded":"2024-03-04T07:37:56.153141+0000","last_fullsized":"2024-03-04T07:37:56.153141+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:22.419437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300959+0000","last_change":"2024-03-04T07:37:55.133270+0000","last_active":"2024-03-04T07:37:56.300959+0000","last_peered":"2024-03-04T07:37:56.300959+0000","last_clean":"2024-03-04T07:37:56.300959+0000","last_became_active":"2024-03-04T07:37:55.132909+0000","last_became_peered":"2024-03-04T07:37:55.132909+0000","last_unstale":"2024-03-04T07:37:56.300959+0000","last_undegraded":"2024-03-04T07:37:56.300959+0000","last_fullsized":"2024-03-04T07:37:56.300959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:21.026141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.095914+0000","last_change":"2024-03-04T07:37:54.135988+0000","last_active":"2024-03-04T07:37:56.095914+0000","last_peered":"2024-03-04T07:37:56.095914+0000","last_clean":"2024-03-04T07:37:56.095914+0000","last_became_active":"2024-03-04T07:37:54.135627+0000","last_became_peered":"2024-03-04T07:37:54.135627+0000","last_unstale":"2024-03-04T07:37:56.095914+0000","last_undegraded":"2024-03-04T07:37:56.095914+0000","last_fullsized":"2024-03-04T07:37:56.095914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:06.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153423+0000","last_change":"2024-03-04T07:37:55.144551+0000","last_active":"2024-03-04T07:37:56.153423+0000","last_peered":"2024-03-04T07:37:56.153423+0000","last_clean":"2024-03-04T07:37:56.153423+0000","last_became_active":"2024-03-04T07:37:55.143882+0000","last_became_peered":"2024-03-04T07:37:55.143882+0000","last_unstale":"2024-03-04T07:37:56.153423+0000","last_undegraded":"2024-03-04T07:37:56.153423+0000","last_fullsized":"2024-03-04T07:37:56.153423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:07.254941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.152278+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:37:56.152278+0000","last_peered":"2024-03-04T07:37:56.152278+0000","last_clean":"2024-03-04T07:37:56.152278+0000","last_became_active":"2024-03-04T07:37:54.121642+0000","last_became_peered":"2024-03-04T07:37:54.121642+0000","last_unstale":"2024-03-04T07:37:56.152278+0000","last_undegraded":"2024-03-04T07:37:56.152278+0000","last_fullsized":"2024-03-04T07:37:56.152278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:06.576421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337037+0000","last_change":"2024-03-04T07:37:55.133870+0000","last_active":"2024-03-04T07:37:56.337037+0000","last_peered":"2024-03-04T07:37:56.337037+0000","last_clean":"2024-03-04T07:37:56.337037+0000","last_became_active":"2024-03-04T07:37:55.133542+0000","last_became_peered":"2024-03-04T07:37:55.133542+0000","last_unstale":"2024-03-04T07:37:56.337037+0000","last_undegraded":"2024-03-04T07:37:56.337037+0000","last_fullsized":"2024-03-04T07:37:56.337037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:17:44.373426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096337+0000","last_change":"2024-03-04T07:37:54.126326+0000","last_active":"2024-03-04T07:37:56.096337+0000","last_peered":"2024-03-04T07:37:56.096337+0000","last_clean":"2024-03-04T07:37:56.096337+0000","last_became_active":"2024-03-04T07:37:54.125989+0000","last_became_peered":"2024-03-04T07:37:54.125989+0000","last_unstale":"2024-03-04T07:37:56.096337+0000","last_undegraded":"2024-03-04T07:37:56.096337+0000","last_fullsized":"2024-03-04T07:37:56.096337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:41.108528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337302+0000","last_change":"2024-03-04T07:37:55.120052+0000","last_active":"2024-03-04T07:37:56.337302+0000","last_peered":"2024-03-04T07:37:56.337302+0000","last_clean":"2024-03-04T07:37:56.337302+0000","last_became_active":"2024-03-04T07:37:55.119708+0000","last_became_peered":"2024-03-04T07:37:55.119708+0000","last_unstale":"2024-03-04T07:37:56.337302+0000","last_undegraded":"2024-03-04T07:37:56.337302+0000","last_fullsized":"2024-03-04T07:37:56.337302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:25:17.679416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301525+0000","last_change":"2024-03-04T07:37:54.134018+0000","last_active":"2024-03-04T07:37:56.301525+0000","last_peered":"2024-03-04T07:37:56.301525+0000","last_clean":"2024-03-04T07:37:56.301525+0000","last_became_active":"2024-03-04T07:37:54.133673+0000","last_became_peered":"2024-03-04T07:37:54.133673+0000","last_unstale":"2024-03-04T07:37:56.301525+0000","last_undegraded":"2024-03-04T07:37:56.301525+0000","last_fullsized":"2024-03-04T07:37:56.301525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:49.438338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337018+0000","last_change":"2024-03-04T07:37:55.124052+0000","last_active":"2024-03-04T07:37:56.337018+0000","last_peered":"2024-03-04T07:37:56.337018+0000","last_clean":"2024-03-04T07:37:56.337018+0000","last_became_active":"2024-03-04T07:37:55.123667+0000","last_became_peered":"2024-03-04T07:37:55.123667+0000","last_unstale":"2024-03-04T07:37:56.337018+0000","last_undegraded":"2024-03-04T07:37:56.337018+0000","last_fullsized":"2024-03-04T07:37:56.337018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:35:25.491236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302932+0000","last_change":"2024-03-04T07:37:54.140806+0000","last_active":"2024-03-04T07:37:56.302932+0000","last_peered":"2024-03-04T07:37:56.302932+0000","last_clean":"2024-03-04T07:37:56.302932+0000","last_became_active":"2024-03-04T07:37:54.140419+0000","last_became_peered":"2024-03-04T07:37:54.140419+0000","last_unstale":"2024-03-04T07:37:56.302932+0000","last_undegraded":"2024-03-04T07:37:56.302932+0000","last_fullsized":"2024-03-04T07:37:56.302932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:13:59.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.337338+0000","last_change":"2024-03-04T07:37:55.113658+0000","last_active":"2024-03-04T07:37:56.337338+0000","last_peered":"2024-03-04T07:37:56.337338+0000","last_clean":"2024-03-04T07:37:56.337338+0000","last_became_active":"2024-03-04T07:37:55.113300+0000","last_became_peered":"2024-03-04T07:37:55.113300+0000","last_unstale":"2024-03-04T07:37:56.337338+0000","last_undegraded":"2024-03-04T07:37:56.337338+0000","last_fullsized":"2024-03-04T07:37:56.337338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:43:56.907387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301151+0000","last_change":"2024-03-04T07:37:55.117769+0000","last_active":"2024-03-04T07:37:56.301151+0000","last_peered":"2024-03-04T07:37:56.301151+0000","last_clean":"2024-03-04T07:37:56.301151+0000","last_became_active":"2024-03-04T07:37:55.117408+0000","last_became_peered":"2024-03-04T07:37:55.117408+0000","last_unstale":"2024-03-04T07:37:56.301151+0000","last_undegraded":"2024-03-04T07:37:56.301151+0000","last_fullsized":"2024-03-04T07:37:56.301151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:22.143108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096050+0000","last_change":"2024-03-04T07:37:55.115581+0000","last_active":"2024-03-04T07:37:56.096050+0000","last_peered":"2024-03-04T07:37:56.096050+0000","last_clean":"2024-03-04T07:37:56.096050+0000","last_became_active":"2024-03-04T07:37:55.113387+0000","last_became_peered":"2024-03-04T07:37:55.113387+0000","last_unstale":"2024-03-04T07:37:56.096050+0000","last_undegraded":"2024-03-04T07:37:56.096050+0000","last_fullsized":"2024-03-04T07:37:56.096050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:59.273650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096021+0000","last_change":"2024-03-04T07:37:55.135175+0000","last_active":"2024-03-04T07:37:56.096021+0000","last_peered":"2024-03-04T07:37:56.096021+0000","last_clean":"2024-03-04T07:37:56.096021+0000","last_became_active":"2024-03-04T07:37:55.134787+0000","last_became_peered":"2024-03-04T07:37:55.134787+0000","last_unstale":"2024-03-04T07:37:56.096021+0000","last_undegraded":"2024-03-04T07:37:56.096021+0000","last_fullsized":"2024-03-04T07:37:56.096021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:04.227886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301953+0000","last_change":"2024-03-04T07:37:55.138146+0000","last_active":"2024-03-04T07:37:56.301953+0000","last_peered":"2024-03-04T07:37:56.301953+0000","last_clean":"2024-03-04T07:37:56.301953+0000","last_became_active":"2024-03-04T07:37:55.137806+0000","last_became_peered":"2024-03-04T07:37:55.137806+0000","last_unstale":"2024-03-04T07:37:56.301953+0000","last_undegraded":"2024-03-04T07:37:56.301953+0000","last_fullsized":"2024-03-04T07:37:56.301953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:10.596224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302792+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:37:56.302792+0000","last_peered":"2024-03-04T07:37:56.302792+0000","last_clean":"2024-03-04T07:37:56.302792+0000","last_became_active":"2024-03-04T07:37:55.140670+0000","last_became_peered":"2024-03-04T07:37:55.140670+0000","last_unstale":"2024-03-04T07:37:56.302792+0000","last_undegraded":"2024-03-04T07:37:56.302792+0000","last_fullsized":"2024-03-04T07:37:56.302792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:29:54.439417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301249+0000","last_change":"2024-03-04T07:37:55.115015+0000","last_active":"2024-03-04T07:37:56.301249+0000","last_peered":"2024-03-04T07:37:56.301249+0000","last_clean":"2024-03-04T07:37:56.301249+0000","last_became_active":"2024-03-04T07:37:55.114668+0000","last_became_peered":"2024-03-04T07:37:55.114668+0000","last_unstale":"2024-03-04T07:37:56.301249+0000","last_undegraded":"2024-03-04T07:37:56.301249+0000","last_fullsized":"2024-03-04T07:37:56.301249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:29:03.309234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.096475+0000","last_change":"2024-03-04T07:37:55.119514+0000","last_active":"2024-03-04T07:37:56.096475+0000","last_peered":"2024-03-04T07:37:56.096475+0000","last_clean":"2024-03-04T07:37:56.096475+0000","last_became_active":"2024-03-04T07:37:55.119166+0000","last_became_peered":"2024-03-04T07:37:55.119166+0000","last_unstale":"2024-03-04T07:37:56.096475+0000","last_undegraded":"2024-03-04T07:37:56.096475+0000","last_fullsized":"2024-03-04T07:37:56.096475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:33.468311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153879+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:37:56.153879+0000","last_peered":"2024-03-04T07:37:56.153879+0000","last_clean":"2024-03-04T07:37:56.153879+0000","last_became_active":"2024-03-04T07:37:55.140723+0000","last_became_peered":"2024-03-04T07:37:55.140723+0000","last_unstale":"2024-03-04T07:37:56.153879+0000","last_undegraded":"2024-03-04T07:37:56.153879+0000","last_fullsized":"2024-03-04T07:37:56.153879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:29.751190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300822+0000","last_change":"2024-03-04T07:37:55.127627+0000","last_active":"2024-03-04T07:37:56.300822+0000","last_peered":"2024-03-04T07:37:56.300822+0000","last_clean":"2024-03-04T07:37:56.300822+0000","last_became_active":"2024-03-04T07:37:55.127217+0000","last_became_peered":"2024-03-04T07:37:55.127217+0000","last_unstale":"2024-03-04T07:37:56.300822+0000","last_undegraded":"2024-03-04T07:37:56.300822+0000","last_fullsized":"2024-03-04T07:37:56.300822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:47.826970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.153510+0000","last_change":"2024-03-04T07:37:55.129445+0000","last_active":"2024-03-04T07:37:56.153510+0000","last_peered":"2024-03-04T07:37:56.153510+0000","last_clean":"2024-03-04T07:37:56.153510+0000","last_became_active":"2024-03-04T07:37:55.129013+0000","last_became_peered":"2024-03-04T07:37:55.129013+0000","last_unstale":"2024-03-04T07:37:56.153510+0000","last_undegraded":"2024-03-04T07:37:56.153510+0000","last_fullsized":"2024-03-04T07:37:56.153510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:15.578436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.300404+0000","last_change":"2024-03-04T07:37:55.126397+0000","last_active":"2024-03-04T07:37:56.300404+0000","last_peered":"2024-03-04T07:37:56.300404+0000","last_clean":"2024-03-04T07:37:56.300404+0000","last_became_active":"2024-03-04T07:37:55.126092+0000","last_became_peered":"2024-03-04T07:37:55.126092+0000","last_unstale":"2024-03-04T07:37:56.300404+0000","last_undegraded":"2024-03-04T07:37:56.300404+0000","last_fullsized":"2024-03-04T07:37:56.300404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:47.968250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.302789+0000","last_change":"2024-03-04T07:37:55.140265+0000","last_active":"2024-03-04T07:37:56.302789+0000","last_peered":"2024-03-04T07:37:56.302789+0000","last_clean":"2024-03-04T07:37:56.302789+0000","last_became_active":"2024-03-04T07:37:55.139768+0000","last_became_peered":"2024-03-04T07:37:55.139768+0000","last_unstale":"2024-03-04T07:37:56.302789+0000","last_undegraded":"2024-03-04T07:37:56.302789+0000","last_fullsized":"2024-03-04T07:37:56.302789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:52.924188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-04T07:37:56.301086+0000","last_change":"2024-03-04T07:37:55.118929+0000","last_active":"2024-03-04T07:37:56.301086+0000","last_peered":"2024-03-04T07:37:56.301086+0000","last_clean":"2024-03-04T07:37:56.301086+0000","last_became_active":"2024-03-04T07:37:55.118549+0000","last_became_peered":"2024-03-04T07:37:55.118549+0000","last_unstale":"2024-03-04T07:37:56.301086+0000","last_undegraded":"2024-03-04T07:37:56.301086+0000","last_fullsized":"2024-03-04T07:37:56.301086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:15.599080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244954,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":1030695,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":921600,"data_stored":439125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332132,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":966656,"data_stored":1029471,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":917504,"data_stored":431597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":921600,"data_stored":438625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":974848,"data_stored":1029071,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T07:38:03.504 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T07:38:03.505 INFO:tasks.cephadm:Setting up client nodes... 2024-03-04T07:38:03.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T07:38:03.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:03 smithi152 ceph-mon[37154]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-04T07:38:03.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:03 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1774254336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:03.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:03 smithi133 ceph-mon[36505]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-04T07:38:03.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:03 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1774254336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:04.008 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:04.503 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:04 smithi116 ceph-mon[32150]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:38:04.503 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:04 smithi116 ceph-mon[32150]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:38:04.503 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:04 smithi116 ceph-mon[32150]: pgmap v318: 97 pgs: 97 active+clean; 615 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-04T07:38:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:04 smithi152 ceph-mon[37154]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:38:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:04 smithi152 ceph-mon[37154]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:38:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:04 smithi152 ceph-mon[37154]: pgmap v318: 97 pgs: 97 active+clean; 615 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-04T07:38:04.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:04 smithi133 ceph-mon[36505]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:38:04.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:04 smithi133 ceph-mon[36505]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:38:04.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:04 smithi133 ceph-mon[36505]: pgmap v318: 97 pgs: 97 active+clean; 615 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-04T07:38:06.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.364+0000 7ffb79c00700 1 -- 172.21.15.116:0/2823983313 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74109fe0 msgr2=0x7ffb7410c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:06.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.364+0000 7ffb79c00700 1 --2- 172.21.15.116:0/2823983313 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74109fe0 0x7ffb7410c470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009f90 tx=0x7ffb6c00b330 comp rx=0 tx=0).stop 2024-03-04T07:38:06.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.365+0000 7ffb79c00700 1 -- 172.21.15.116:0/2823983313 shutdown_connections 2024-03-04T07:38:06.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.365+0000 7ffb79c00700 1 --2- 172.21.15.116:0/2823983313 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74109fe0 0x7ffb7410c470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.365+0000 7ffb79c00700 1 --2- 172.21.15.116:0/2823983313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb740760f0 0x7ffb74074550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.365+0000 7ffb79c00700 1 --2- 172.21.15.116:0/2823983313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb74074b20 0x7ffb74074fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.365+0000 7ffb79c00700 1 -- 172.21.15.116:0/2823983313 >> 172.21.15.116:0/2823983313 conn(0x7ffb740fd6d0 msgr2=0x7ffb740ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:06.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.365+0000 7ffb79c00700 1 -- 172.21.15.116:0/2823983313 shutdown_connections 2024-03-04T07:38:06.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.365+0000 7ffb79c00700 1 -- 172.21.15.116:0/2823983313 wait complete. 2024-03-04T07:38:06.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 Processor -- start 2024-03-04T07:38:06.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 -- start start 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74074b20 0x7ffb7419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb740760f0 0x7ffb7419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb74109fe0 0x7ffb741a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7410ff60 con 0x7ffb740760f0 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7410fde0 con 0x7ffb74074b20 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.366+0000 7ffb79c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb741100e0 con 0x7ffb74109fe0 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74074b20 0x7ffb7419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74074b20 0x7ffb7419a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34962/0 (socket says 172.21.15.116:34962) 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 -- 172.21.15.116:0/6608268 learned_addr learned my addr 172.21.15.116:0/6608268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb6a000700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb740760f0 0x7ffb7419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 -- 172.21.15.116:0/6608268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb74109fe0 msgr2=0x7ffb741a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:06.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb74109fe0 0x7ffb741a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 -- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb740760f0 msgr2=0x7ffb7419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb740760f0 0x7ffb7419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 -- 172.21.15.116:0/6608268 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb741a20b0 con 0x7ffb74074b20 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.367+0000 7ffb72a00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74074b20 0x7ffb7419a450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400e9f0 tx=0x7ffb6400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.368+0000 7ffb6be00700 1 -- 172.21.15.116:0/6608268 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6400cbd0 con 0x7ffb74074b20 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.368+0000 7ffb6be00700 1 -- 172.21.15.116:0/6608268 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb6400cd70 con 0x7ffb74074b20 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.368+0000 7ffb6be00700 1 -- 172.21.15.116:0/6608268 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64010430 con 0x7ffb74074b20 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.368+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb741a23a0 con 0x7ffb74074b20 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.368+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb741a2890 con 0x7ffb74074b20 2024-03-04T07:38:06.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.369+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb740505d0 con 0x7ffb74074b20 2024-03-04T07:38:06.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.371+0000 7ffb6be00700 1 -- 172.21.15.116:0/6608268 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7ffb64004750 con 0x7ffb74074b20 2024-03-04T07:38:06.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.372+0000 7ffb6be00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffb580a1fd0 0x7ffb580a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:06.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.372+0000 7ffb6be00700 1 -- 172.21.15.116:0/6608268 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ffb640ce070 con 0x7ffb74074b20 2024-03-04T07:38:06.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.372+0000 7ffb6a000700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffb580a1fd0 0x7ffb580a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:06.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.373+0000 7ffb6a000700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffb580a1fd0 0x7ffb580a4490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009750 tx=0x7ffb5c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:06.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.374+0000 7ffb6be00700 1 -- 172.21.15.116:0/6608268 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffb6409b3f0 con 0x7ffb74074b20 2024-03-04T07:38:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.683+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ffb7402da50 con 0x7ffb74074b20 2024-03-04T07:38:06.684 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:06 smithi116 ceph-mon[32150]: pgmap v319: 97 pgs: 97 active+clean; 615 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-04T07:38:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.689+0000 7ffb6be00700 1 -- 172.21.15.116:0/6608268 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7ffb6400fe10 con 0x7ffb74074b20 2024-03-04T07:38:06.690 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-03-04T07:38:06.690 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBeeuVlYxbhKBAAPzEyUmtBU2JCHWwnxjew0g== 2024-03-04T07:38:06.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.691+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffb580a1fd0 msgr2=0x7ffb580a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:06.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffb580a1fd0 0x7ffb580a4490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009750 tx=0x7ffb5c009310 comp rx=0 tx=0).stop 2024-03-04T07:38:06.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74074b20 msgr2=0x7ffb7419a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:06.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74074b20 0x7ffb7419a450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400e9f0 tx=0x7ffb6400ed00 comp rx=0 tx=0).stop 2024-03-04T07:38:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 shutdown_connections 2024-03-04T07:38:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffb580a1fd0 0x7ffb580a4490 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb74109fe0 0x7ffb741a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffb74074b20 0x7ffb7419a450 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 --2- 172.21.15.116:0/6608268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb740760f0 0x7ffb7419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 >> 172.21.15.116:0/6608268 conn(0x7ffb740fd6d0 msgr2=0x7ffb74105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 shutdown_connections 2024-03-04T07:38:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:06.692+0000 7ffb79c00700 1 -- 172.21.15.116:0/6608268 wait complete. 2024-03-04T07:38:06.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:06 smithi133 ceph-mon[36505]: pgmap v319: 97 pgs: 97 active+clean; 615 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-04T07:38:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:06 smithi152 ceph-mon[37154]: pgmap v319: 97 pgs: 97 active+clean; 615 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-04T07:38:07.559 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:38:07.559 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-04T07:38:07.559 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T07:38:07.719 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T07:38:07.719 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-04T07:38:07.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-04T07:38:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:07 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/6608268' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T07:38:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:07 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T07:38:07.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:07 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T07:38:07.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:07 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/6608268' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T07:38:07.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:07 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T07:38:07.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:07 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T07:38:08.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:07 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/6608268' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T07:38:08.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:07 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T07:38:08.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:07 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T07:38:08.193 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:08 smithi116 ceph-mon[32150]: pgmap v320: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2024-03-04T07:38:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:08 smithi152 ceph-mon[37154]: pgmap v320: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2024-03-04T07:38:09.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:08 smithi133 ceph-mon[36505]: pgmap v320: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2024-03-04T07:38:10.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.536+0000 7f228ae00700 1 -- 172.21.15.116:0/719635555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284069d70 msgr2=0x7f228406a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.536+0000 7f228ae00700 1 --2- 172.21.15.116:0/719635555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284069d70 0x7f228406a1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2270009990 tx=0x7f2270009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.537+0000 7f228ae00700 1 -- 172.21.15.116:0/719635555 shutdown_connections 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.537+0000 7f228ae00700 1 --2- 172.21.15.116:0/719635555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f228406a710 0x7f228410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.537+0000 7f228ae00700 1 --2- 172.21.15.116:0/719635555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2284108110 0x7f22841084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.537+0000 7f228ae00700 1 --2- 172.21.15.116:0/719635555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284069d70 0x7f228406a1d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.537+0000 7f228ae00700 1 -- 172.21.15.116:0/719635555 >> 172.21.15.116:0/719635555 conn(0x7f2284077ef0 msgr2=0x7f2284078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.537+0000 7f228ae00700 1 -- 172.21.15.116:0/719635555 shutdown_connections 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.537+0000 7f228ae00700 1 -- 172.21.15.116:0/719635555 wait complete. 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 Processor -- start 2024-03-04T07:38:10.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 -- start start 2024-03-04T07:38:10.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2284069d70 0x7f22841a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f228406a710 0x7f22841a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284108110 0x7f228419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22841123c0 con 0x7f2284069d70 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2284112240 con 0x7f2284108110 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.538+0000 7f228ae00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2284112540 con 0x7f228406a710 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2283e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2284069d70 0x7f22841a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2288e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284108110 0x7f228419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2283e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2284069d70 0x7f22841a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42158/0 (socket says 172.21.15.116:42158) 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2288e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284108110 0x7f228419a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34976/0 (socket says 172.21.15.116:34976) 2024-03-04T07:38:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2283e00700 1 -- 172.21.15.116:0/3952763843 learned_addr learned my addr 172.21.15.116:0/3952763843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2283400700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f228406a710 0x7f22841a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2288e00700 1 -- 172.21.15.116:0/3952763843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f228406a710 msgr2=0x7f22841a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2288e00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f228406a710 0x7f22841a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2288e00700 1 -- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2284069d70 msgr2=0x7f22841a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2288e00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2284069d70 0x7f22841a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2288e00700 1 -- 172.21.15.116:0/3952763843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228419afe0 con 0x7f2284108110 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.539+0000 7f2283e00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2284069d70 0x7f22841a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.540+0000 7f2288e00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284108110 0x7f228419a880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f227c00efc0 tx=0x7f227c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.540+0000 7f2280c00700 1 -- 172.21.15.116:0/3952763843 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0092e0 con 0x7f2284108110 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.540+0000 7f2280c00700 1 -- 172.21.15.116:0/3952763843 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f227c004bb0 con 0x7f2284108110 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.540+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228419b270 con 0x7f2284108110 2024-03-04T07:38:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.540+0000 7f2280c00700 1 -- 172.21.15.116:0/3952763843 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c004280 con 0x7f2284108110 2024-03-04T07:38:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.540+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22841a7590 con 0x7f2284108110 2024-03-04T07:38:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.541+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2284068d80 con 0x7f2284108110 2024-03-04T07:38:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.543+0000 7f2280c00700 1 -- 172.21.15.116:0/3952763843 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f227c01b030 con 0x7f2284108110 2024-03-04T07:38:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.544+0000 7f2280c00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22680a1f20 0x7f22680a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.544+0000 7f2280c00700 1 -- 172.21.15.116:0/3952763843 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f227c0cd930 con 0x7f2284108110 2024-03-04T07:38:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.544+0000 7f2283e00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22680a1f20 0x7f22680a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:10.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.545+0000 7f2283e00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22680a1f20 0x7f22680a43e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f228419b9c0 tx=0x7f2274009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:10.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.548+0000 7f2280c00700 1 -- 172.21.15.116:0/3952763843 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f227c022020 con 0x7f2284108110 2024-03-04T07:38:10.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.830+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f22841a7760 con 0x7f2284108110 2024-03-04T07:38:10.831 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:10 smithi116 ceph-mon[32150]: pgmap v321: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-03-04T07:38:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.835+0000 7f2280c00700 1 -- 172.21.15.116:0/3952763843 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+261472 (secure 0 0 0) 0x7f227c096030 con 0x7f2284108110 2024-03-04T07:38:10.840 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:38:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.841+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22680a1f20 msgr2=0x7f22680a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.841+0000 7f228ae00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22680a1f20 0x7f22680a43e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f228419b9c0 tx=0x7f2274009260 comp rx=0 tx=0).stop 2024-03-04T07:38:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.841+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284108110 msgr2=0x7f228419a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:10.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.841+0000 7f228ae00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284108110 0x7f228419a880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f227c00efc0 tx=0x7f227c00c3a0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.841+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 shutdown_connections 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.841+0000 7f228ae00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2284069d70 0x7f22841a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.841+0000 7f228ae00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22680a1f20 0x7f22680a43e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.842+0000 7f228ae00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2284108110 0x7f228419a880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.842+0000 7f228ae00700 1 --2- 172.21.15.116:0/3952763843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f228406a710 0x7f22841a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.842+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 >> 172.21.15.116:0/3952763843 conn(0x7f2284077ef0 msgr2=0x7f228410d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.842+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 shutdown_connections 2024-03-04T07:38:10.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:10.842+0000 7f228ae00700 1 -- 172.21.15.116:0/3952763843 wait complete. 2024-03-04T07:38:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:10 smithi152 ceph-mon[37154]: pgmap v321: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-03-04T07:38:11.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:10 smithi133 ceph-mon[36505]: pgmap v321: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-03-04T07:38:11.878 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":142890918},{"type":"v1","addr":"172.21.15.116:6801","nonce":142890918}]},"active_addr":"172.21.15.116:6801/142890918","active_change":"2024-03-04T07:29:57.331964+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.116:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1106698012}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":3861295184}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":9312237}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":855763577}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2815568038}]}]} 2024-03-04T07:38:11.885 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-04T07:38:11.885 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T07:38:11.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T07:38:12.000 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3952763843' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T07:38:12.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:11 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3952763843' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T07:38:12.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:11 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3952763843' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T07:38:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:12.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:12 smithi116 ceph-mon[32150]: pgmap v322: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-03-04T07:38:13.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:12 smithi152 ceph-mon[37154]: pgmap v322: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-03-04T07:38:13.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:12 smithi133 ceph-mon[36505]: pgmap v322: 97 pgs: 97 active+clean; 620 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-03-04T07:38:14.731 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:14 smithi116 ceph-mon[32150]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-03-04T07:38:14.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 -- 172.21.15.116:0/248963350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 msgr2=0x7f771806a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:14.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 --2- 172.21.15.116:0/248963350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771806a1d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f77000099c0 tx=0x7f7700009cd0 comp rx=0 tx=0).stop 2024-03-04T07:38:14.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 -- 172.21.15.116:0/248963350 shutdown_connections 2024-03-04T07:38:14.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 --2- 172.21.15.116:0/248963350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f771806a710 0x7f771810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:14.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 --2- 172.21.15.116:0/248963350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7718108110 0x7f77181084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:14.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 --2- 172.21.15.116:0/248963350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771806a1d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:14.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 -- 172.21.15.116:0/248963350 >> 172.21.15.116:0/248963350 conn(0x7f7718077ef0 msgr2=0x7f7718078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 -- 172.21.15.116:0/248963350 shutdown_connections 2024-03-04T07:38:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.763+0000 7f771e000700 1 -- 172.21.15.116:0/248963350 wait complete. 2024-03-04T07:38:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.764+0000 7f771e000700 1 Processor -- start 2024-03-04T07:38:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.764+0000 7f771e000700 1 -- start start 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f771e000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f771e000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f771806a710 0x7f771819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f771e000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7718108110 0x7f7718198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f771e000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77181123f0 con 0x7f7718069d70 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f771e000700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7718112270 con 0x7f7718108110 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f771e000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7718112570 con 0x7f771806a710 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f7717400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f7717400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42166/0 (socket says 172.21.15.116:42166) 2024-03-04T07:38:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f7717400700 1 -- 172.21.15.116:0/3153871017 learned_addr learned my addr 172.21.15.116:0/3153871017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f7717e00700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7718108110 0x7f7718198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.765+0000 7f770ea00700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f771806a710 0x7f771819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7717e00700 1 -- 172.21.15.116:0/3153871017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f771806a710 msgr2=0x7f771819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7717e00700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f771806a710 0x7f771819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7717e00700 1 -- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 msgr2=0x7f771819e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7717e00700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771819e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7717e00700 1 -- 172.21.15.116:0/3153871017 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718198f70 con 0x7f7718108110 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7717400700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f770ea00700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f771806a710 0x7f771819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7717e00700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7718108110 0x7f7718198690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f771000e970 tx=0x7f771000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7714c00700 1 -- 172.21.15.116:0/3153871017 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710004220 con 0x7f7718108110 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7718199260 con 0x7f7718108110 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7714c00700 1 -- 172.21.15.116:0/3153871017 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7710004d10 con 0x7f7718108110 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f7714c00700 1 -- 172.21.15.116:0/3153871017 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710010700 con 0x7f7718108110 2024-03-04T07:38:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.766+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77181a5540 con 0x7f7718108110 2024-03-04T07:38:14.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.767+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77181037c0 con 0x7f7718108110 2024-03-04T07:38:14.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.769+0000 7f7714c00700 1 -- 172.21.15.116:0/3153871017 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f7710004750 con 0x7f7718108110 2024-03-04T07:38:14.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.770+0000 7f7714c00700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f76fc0a1ea0 0x7f76fc0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:14.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.771+0000 7f7714c00700 1 -- 172.21.15.116:0/3153871017 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f77100cdbb0 con 0x7f7718108110 2024-03-04T07:38:14.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.771+0000 7f7717400700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f76fc0a1ea0 0x7f76fc0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:14.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.771+0000 7f7717400700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f76fc0a1ea0 0x7f76fc0a4360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f770800c530 tx=0x7f770800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:14.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.776+0000 7f7714c00700 1 -- 172.21.15.116:0/3153871017 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f771009af30 con 0x7f7718108110 2024-03-04T07:38:15.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:14.999+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f771802d9b0 con 0x7f7718108110 2024-03-04T07:38:15.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.001+0000 7f7714c00700 1 -- 172.21.15.116:0/3153871017 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22302 (secure 0 0 0) 0x7f771009a830 con 0x7f7718108110 2024-03-04T07:38:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:38:15.003 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":85,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:37:56.091002+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:38:15.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f76fc0a1ea0 msgr2=0x7f76fc0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f76fc0a1ea0 0x7f76fc0a4360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f770800c530 tx=0x7f770800b040 comp rx=0 tx=0).stop 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7718108110 msgr2=0x7f7718198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7718108110 0x7f7718198690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f771000e970 tx=0x7f771000ec80 comp rx=0 tx=0).stop 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 shutdown_connections 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7718069d70 0x7f771819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f76fc0a1ea0 0x7f76fc0a4360 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7718108110 0x7f7718198690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:15.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 --2- 172.21.15.116:0/3153871017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f771806a710 0x7f771819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:15.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.005+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 >> 172.21.15.116:0/3153871017 conn(0x7f7718077ef0 msgr2=0x7f77180ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:15.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.006+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 shutdown_connections 2024-03-04T07:38:15.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:15.006+0000 7f771e000700 1 -- 172.21.15.116:0/3153871017 wait complete. 2024-03-04T07:38:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:14 smithi152 ceph-mon[37154]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-03-04T07:38:15.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:14 smithi133 ceph-mon[36505]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-03-04T07:38:16.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3153871017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:16.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3153871017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:16.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3153871017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:16.213 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T07:38:16.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T07:38:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:16 smithi116 ceph-mon[32150]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2024-03-04T07:38:17.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:16 smithi152 ceph-mon[37154]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2024-03-04T07:38:17.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:16 smithi133 ceph-mon[36505]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.4 KiB/s wr, 3 op/s 2024-03-04T07:38:18.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:18 smithi116 ceph-mon[32150]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-03-04T07:38:19.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 -- 172.21.15.116:0/3276228153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 msgr2=0x7fb334106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:19.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 --2- 172.21.15.116:0/3276228153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 0x7fb334106340 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb3240099c0 tx=0x7fb324009cd0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 -- 172.21.15.116:0/3276228153 shutdown_connections 2024-03-04T07:38:19.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 --2- 172.21.15.116:0/3276228153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb334106880 0x7fb334111260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 --2- 172.21.15.116:0/3276228153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3341045d0 0x7fb3341049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 --2- 172.21.15.116:0/3276228153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 0x7fb334106340 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 -- 172.21.15.116:0/3276228153 >> 172.21.15.116:0/3276228153 conn(0x7fb3340ffd60 msgr2=0x7fb334102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:19.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 -- 172.21.15.116:0/3276228153 shutdown_connections 2024-03-04T07:38:19.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.008+0000 7fb33c800700 1 -- 172.21.15.116:0/3276228153 wait complete. 2024-03-04T07:38:19.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.009+0000 7fb33c800700 1 Processor -- start 2024-03-04T07:38:19.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33c800700 1 -- start start 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33c800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3341045d0 0x7fb3341a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33c800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 0x7fb3341a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33c800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb334106880 0x7fb33419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33c800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3341146d0 con 0x7fb334105ec0 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33c800700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb334114550 con 0x7fb3341045d0 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33c800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb334114850 con 0x7fb334106880 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb339e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3341045d0 0x7fb3341a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb339400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 0x7fb3341a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb339e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3341045d0 0x7fb3341a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49718/0 (socket says 172.21.15.116:49718) 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb339e00700 1 -- 172.21.15.116:0/2238361852 learned_addr learned my addr 172.21.15.116:0/2238361852 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.010+0000 7fb33a800700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb334106880 0x7fb33419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:19.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb339e00700 1 -- 172.21.15.116:0/2238361852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb334106880 msgr2=0x7fb33419a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb339e00700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb334106880 0x7fb33419a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb339e00700 1 -- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 msgr2=0x7fb3341a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb339e00700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 0x7fb3341a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb339e00700 1 -- 172.21.15.116:0/2238361852 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33419b050 con 0x7fb3341045d0 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb339e00700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3341045d0 0x7fb3341a0730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d980 tx=0x7fb32800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb32ea00700 1 -- 172.21.15.116:0/2238361852 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328016070 con 0x7fb3341045d0 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.011+0000 7fb32ea00700 1 -- 172.21.15.116:0/2238361852 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb328004d10 con 0x7fb3341045d0 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.012+0000 7fb32ea00700 1 -- 172.21.15.116:0/2238361852 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3280054e0 con 0x7fb3341045d0 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.012+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33419b340 con 0x7fb3341045d0 2024-03-04T07:38:19.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.012+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3341a7600 con 0x7fb3341045d0 2024-03-04T07:38:19.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.013+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb334068d80 con 0x7fb3341045d0 2024-03-04T07:38:19.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.015+0000 7fb32ea00700 1 -- 172.21.15.116:0/2238361852 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fb328044a60 con 0x7fb3341045d0 2024-03-04T07:38:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.016+0000 7fb32ea00700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb31c0a1f20 0x7fb31c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.016+0000 7fb32ea00700 1 -- 172.21.15.116:0/2238361852 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb3280cde50 con 0x7fb3341045d0 2024-03-04T07:38:19.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.016+0000 7fb339400700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb31c0a1f20 0x7fb31c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:19.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.017+0000 7fb339400700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb31c0a1f20 0x7fb31c0a43e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb334102340 tx=0x7fb324005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:19.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.018+0000 7fb32ea00700 1 -- 172.21.15.116:0/2238361852 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb32801f020 con 0x7fb3341045d0 2024-03-04T07:38:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:18 smithi152 ceph-mon[37154]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-03-04T07:38:19.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:18 smithi133 ceph-mon[36505]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-03-04T07:38:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.245+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb33402d9b0 con 0x7fb3341045d0 2024-03-04T07:38:19.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.247+0000 7fb32ea00700 1 -- 172.21.15.116:0/2238361852 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22302 (secure 0 0 0) 0x7fb32809b1d0 con 0x7fb3341045d0 2024-03-04T07:38:19.249 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:38:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":85,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:37:56.091002+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:38:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.250+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb31c0a1f20 msgr2=0x7fb31c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.250+0000 7fb33c800700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb31c0a1f20 0x7fb31c0a43e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb334102340 tx=0x7fb324005fb0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3341045d0 msgr2=0x7fb3341a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3341045d0 0x7fb3341a0730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d980 tx=0x7fb32800dc90 comp rx=0 tx=0).stop 2024-03-04T07:38:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 shutdown_connections 2024-03-04T07:38:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb31c0a1f20 0x7fb31c0a43e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb334106880 0x7fb33419a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3341045d0 0x7fb3341a0730 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 --2- 172.21.15.116:0/2238361852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb334105ec0 0x7fb3341a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:19.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 >> 172.21.15.116:0/2238361852 conn(0x7fb3340ffd60 msgr2=0x7fb3341002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:19.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 shutdown_connections 2024-03-04T07:38:19.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:19.251+0000 7fb33c800700 1 -- 172.21.15.116:0/2238361852 wait complete. 2024-03-04T07:38:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:19 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2238361852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:19 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2238361852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:20.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:19 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2238361852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:38:20.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T07:38:20.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T07:38:20.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T07:38:20.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-04T07:38:20.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-04T07:38:20.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-04T07:38:20.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-04T07:38:20.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-04T07:38:20.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-04T07:38:20.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-04T07:38:20.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-04T07:38:20.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-04T07:38:21.007 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:20 smithi116 ceph-mon[32150]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T07:38:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:20 smithi152 ceph-mon[37154]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T07:38:21.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:20 smithi133 ceph-mon[36505]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T07:38:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.325 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.358 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:21.388 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:22.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:22 smithi116 ceph-mon[32150]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T07:38:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:22 smithi152 ceph-mon[37154]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T07:38:23.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:22 smithi133 ceph-mon[36505]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-04T07:38:24.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:24 smithi116 ceph-mon[32150]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-04T07:38:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:24 smithi152 ceph-mon[37154]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-04T07:38:25.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:24 smithi133 ceph-mon[36505]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-04T07:38:26.961 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:26 smithi116 ceph-mon[32150]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:26 smithi152 ceph-mon[37154]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:27.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:26 smithi133 ceph-mon[36505]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:28.823 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:28 smithi116 ceph-mon[32150]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:38:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:28 smithi152 ceph-mon[37154]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:38:29.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:28 smithi133 ceph-mon[36505]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:38:30.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:30 smithi116 ceph-mon[32150]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:30 smithi152 ceph-mon[37154]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:30 smithi133 ceph-mon[36505]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 -- 172.21.15.116:0/293604643 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b781095f0 msgr2=0x7f5b781111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 --2- 172.21.15.116:0/293604643 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b781095f0 0x7f5b781111c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000b0a0 tx=0x7f5b7000b3b0 comp rx=0 tx=0).stop 2024-03-04T07:38:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 -- 172.21.15.116:0/293604643 shutdown_connections 2024-03-04T07:38:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 --2- 172.21.15.116:0/293604643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b78104510 0x7f5b781048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 --2- 172.21.15.116:0/293604643 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b781095f0 0x7f5b781111c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 --2- 172.21.15.116:0/293604643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b78104ec0 0x7f5b78108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 -- 172.21.15.116:0/293604643 >> 172.21.15.116:0/293604643 conn(0x7f5b780ffda0 msgr2=0x7f5b781021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 -- 172.21.15.116:0/293604643 shutdown_connections 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.860+0000 7f5b7de00700 1 -- 172.21.15.116:0/293604643 wait complete. 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.861+0000 7f5b7de00700 1 Processor -- start 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.861+0000 7f5b7de00700 1 -- start start 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.861+0000 7f5b7de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b78104510 0x7f5b781a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.861+0000 7f5b7de00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b78104ec0 0x7f5b781a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b7de00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b781095f0 0x7f5b7819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b7de00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b78114650 con 0x7f5b78104ec0 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b7de00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b781144d0 con 0x7f5b781095f0 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b7de00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b781147d0 con 0x7f5b78104510 2024-03-04T07:38:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b78104ec0 0x7f5b781a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b78104ec0 0x7f5b781a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50710/0 (socket says 172.21.15.116:50710) 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 -- 172.21.15.116:0/2252932656 learned_addr learned my addr 172.21.15.116:0/2252932656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b77400700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b781095f0 0x7f5b7819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76a00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b78104510 0x7f5b781a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 -- 172.21.15.116:0/2252932656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b78104510 msgr2=0x7f5b781a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b78104510 0x7f5b781a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 -- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b781095f0 msgr2=0x7f5b7819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b781095f0 0x7f5b7819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 -- 172.21.15.116:0/2252932656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7819b070 con 0x7f5b78104ec0 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.862+0000 7f5b76000700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b78104ec0 0x7f5b781a0c40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9f0 tx=0x7f5b6400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.863+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b64004a30 con 0x7f5b78104ec0 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.863+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b6400cca0 con 0x7f5b78104ec0 2024-03-04T07:38:31.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.863+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7819b360 con 0x7f5b78104ec0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.863+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b640056c0 con 0x7f5b78104ec0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.863+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b781a75d0 con 0x7f5b78104ec0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.864+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5b780627d0 con 0x7f5b78104ec0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.866+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f5b6400c7c0 con 0x7f5b78104ec0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.867+0000 7f5b7ca00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b5c0a1f20 0x7f5b5c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.867+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5b640ce300 con 0x7f5b78104ec0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.867+0000 7f5b76a00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b5c0a1f20 0x7f5b5c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.868+0000 7f5b7ca00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] conn(0x7f5b5c0ab700 0x7f5b5c0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.868+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 --> [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b5c0ae1d0 con 0x7f5b5c0ab700 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.868+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b640ce6b0 con 0x7f5b78104ec0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.868+0000 7f5b77400700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] conn(0x7f5b5c0ab700 0x7f5b5c0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.868+0000 7f5b76a00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b5c0a1f20 0x7f5b5c0a43e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b78109370 tx=0x7f5b68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.868+0000 7f5b77400700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] conn(0x7f5b5c0ab700 0x7f5b5c0adb20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:31.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.869+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== osd.7 v2:172.21.15.133:6816/3238142404 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5b5c0ae1d0 con 0x7f5b5c0ab700 2024-03-04T07:38:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.887+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 --> [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5b78062c50 con 0x7f5b5c0ab700 2024-03-04T07:38:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.887+0000 7f5b7ca00700 1 -- 172.21.15.116:0/2252932656 <== osd.7 v2:172.21.15.133:6816/3238142404 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5b78062c50 con 0x7f5b5c0ab700 2024-03-04T07:38:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] conn(0x7f5b5c0ab700 msgr2=0x7f5b5c0adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:31.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] conn(0x7f5b5c0ab700 0x7f5b5c0adb20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b5c0a1f20 msgr2=0x7f5b5c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b5c0a1f20 0x7f5b5c0a43e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b78109370 tx=0x7f5b68009260 comp rx=0 tx=0).stop 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b78104ec0 msgr2=0x7f5b781a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 --2- 172.21.15.116:0/2252932656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b78104ec0 0x7f5b781a0c40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9f0 tx=0x7f5b6400ed00 comp rx=0 tx=0).stop 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b77400700 1 -- 172.21.15.116:0/2252932656 reap_dead start 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 shutdown_connections 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 >> 172.21.15.116:0/2252932656 conn(0x7f5b780ffda0 msgr2=0x7f5b78100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 shutdown_connections 2024-03-04T07:38:31.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:30.888+0000 7f5b7de00700 1 -- 172.21.15.116:0/2252932656 wait complete. 2024-03-04T07:38:32.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.486+0000 7fde63e00700 1 -- 172.21.15.116:0/3051161238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c102d20 msgr2=0x7fde5c106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.486+0000 7fde63e00700 1 --2- 172.21.15.116:0/3051161238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c102d20 0x7fde5c106d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde48009990 tx=0x7fde48009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.486+0000 7fde63e00700 1 -- 172.21.15.116:0/3051161238 shutdown_connections 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.486+0000 7fde63e00700 1 --2- 172.21.15.116:0/3051161238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102370 0x7fde5c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.486+0000 7fde63e00700 1 --2- 172.21.15.116:0/3051161238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c107440 0x7fde5c10f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.486+0000 7fde63e00700 1 --2- 172.21.15.116:0/3051161238 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c102d20 0x7fde5c106d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.486+0000 7fde63e00700 1 -- 172.21.15.116:0/3051161238 >> 172.21.15.116:0/3051161238 conn(0x7fde5c079220 msgr2=0x7fde5c079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.487+0000 7fde63e00700 1 -- 172.21.15.116:0/3051161238 shutdown_connections 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.487+0000 7fde63e00700 1 -- 172.21.15.116:0/3051161238 wait complete. 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.487+0000 7fde63e00700 1 Processor -- start 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde63e00700 1 -- start start 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde63e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c102370 0x7fde5c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde63e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102d20 0x7fde5c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde63e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c107440 0x7fde5c19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde63e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c112480 con 0x7fde5c102d20 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde63e00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c112300 con 0x7fde5c107440 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde63e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c112600 con 0x7fde5c102370 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde61e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c107440 0x7fde5c19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde60a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102d20 0x7fde5c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde61e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c107440 0x7fde5c19a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40940/0 (socket says 172.21.15.116:40940) 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde61e00700 1 -- 172.21.15.116:0/2760730195 learned_addr learned my addr 172.21.15.116:0/2760730195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.488+0000 7fde61400700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c102370 0x7fde5c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde61e00700 1 -- 172.21.15.116:0/2760730195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c102370 msgr2=0x7fde5c1a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde61e00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde5c102370 0x7fde5c1a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde61e00700 1 -- 172.21.15.116:0/2760730195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102d20 msgr2=0x7fde5c1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde61e00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde5c102d20 0x7fde5c1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde61e00700 1 -- 172.21.15.116:0/2760730195 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c19b130 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde61e00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c107440 0x7fde5c19a9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fde5400efc0 tx=0x7fde5400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde540092e0 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde54004bb0 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54004020 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c19b3c0 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.489+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c1a7690 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.490+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde5c100920 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.492+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fde5401b030 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.493+0000 7fde5a000700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440a1f20 0x7fde440a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.494+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fde540cdfe0 con 0x7fde5c107440 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.494+0000 7fde61400700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440a1f20 0x7fde440a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.494+0000 7fde5a000700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] conn(0x7fde440ab700 0x7fde440adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.494+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 --> [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde5401fe50 con 0x7fde440ab700 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.494+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde540ce410 con 0x7fde5c107440 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.494+0000 7fde60a00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] conn(0x7fde440ab700 0x7fde440adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.494+0000 7fde61400700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440a1f20 0x7fde440a43e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fde5c107120 tx=0x7fde50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.495+0000 7fde60a00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] conn(0x7fde440ab700 0x7fde440adb20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.496+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== osd.4 v2:172.21.15.133:6808/3217315189 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fde5401fe50 con 0x7fde440ab700 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.516+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 --> [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde5c103840 con 0x7fde440ab700 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.516+0000 7fde5a000700 1 -- 172.21.15.116:0/2760730195 <== osd.4 v2:172.21.15.133:6808/3217315189 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fde5c103840 con 0x7fde440ab700 2024-03-04T07:38:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] conn(0x7fde440ab700 msgr2=0x7fde440adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] conn(0x7fde440ab700 0x7fde440adb20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440a1f20 msgr2=0x7fde440a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde440a1f20 0x7fde440a43e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fde5c107120 tx=0x7fde50009260 comp rx=0 tx=0).stop 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c107440 msgr2=0x7fde5c19a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 --2- 172.21.15.116:0/2760730195 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde5c107440 0x7fde5c19a9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fde5400efc0 tx=0x7fde5400c3a0 comp rx=0 tx=0).stop 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde61e00700 1 -- 172.21.15.116:0/2760730195 reap_dead start 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 shutdown_connections 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 >> 172.21.15.116:0/2760730195 conn(0x7fde5c079220 msgr2=0x7fde5c0fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 shutdown_connections 2024-03-04T07:38:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:31.517+0000 7fde63e00700 1 -- 172.21.15.116:0/2760730195 wait complete. 2024-03-04T07:38:33.075 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:32 smithi116 ceph-mon[32150]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:33.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:32 smithi133 ceph-mon[36505]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:32 smithi152 ceph-mon[37154]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-04T07:38:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 -- 172.21.15.116:0/886321067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 msgr2=0x7f18b4112a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 --2- 172.21.15.116:0/886321067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 0x7f18b4112a10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f189c0098f0 tx=0x7f189c009c00 comp rx=0 tx=0).stop 2024-03-04T07:38:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 -- 172.21.15.116:0/886321067 shutdown_connections 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 --2- 172.21.15.116:0/886321067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18b4112f50 0x7f18b4115340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 --2- 172.21.15.116:0/886321067 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f18b40765f0 0x7f18b40769d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 --2- 172.21.15.116:0/886321067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 0x7f18b4112a10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 -- 172.21.15.116:0/886321067 >> 172.21.15.116:0/886321067 conn(0x7f18b40ffda0 msgr2=0x7f18b41021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.773+0000 7f18b9a00700 1 -- 172.21.15.116:0/886321067 shutdown_connections 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.774+0000 7f18b9a00700 1 -- 172.21.15.116:0/886321067 wait complete. 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 Processor -- start 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 -- start start 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18b40765f0 0x7f18b419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 0x7f18b419f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f18b4112f50 0x7f18b41a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:33.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4118a10 con 0x7f18b4076f10 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4118890 con 0x7f18b4112f50 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18b9a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4118b90 con 0x7f18b40765f0 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.775+0000 7f18aa000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 0x7f18b419f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f18b4112f50 0x7f18b41a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18aa000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 0x7f18b419f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50692/0 (socket says 172.21.15.116:50692) 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18aa000700 1 -- 172.21.15.116:0/1333059111 learned_addr learned my addr 172.21.15.116:0/1333059111 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f18b4112f50 0x7f18b41a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40898/0 (socket says 172.21.15.116:40898) 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b2a00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18b40765f0 0x7f18b419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 -- 172.21.15.116:0/1333059111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18b40765f0 msgr2=0x7f18b419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18b40765f0 0x7f18b419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 -- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 msgr2=0x7f18b419f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 0x7f18b419f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 -- 172.21.15.116:0/1333059111 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b41a3ea0 con 0x7f18b4112f50 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18aa000700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18b4076f10 0x7f18b419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:33.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18b3400700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f18b4112f50 0x7f18b41a3760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18b406b8d0 tx=0x7f18ac00da00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac016070 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18ac010c10 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.776+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac003c20 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.777+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b41a4190 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.777+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b41aba10 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.777+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18b4062850 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.779+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f18ac004220 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.780+0000 7f18abe00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f18980a1f20 0x7f18980a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.781+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f18ac0cd4c0 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.781+0000 7f18b2a00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f18980a1f20 0x7f18980a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.781+0000 7f18abe00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] conn(0x7f18980ab700 0x7f18980adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.781+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 --> [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18ac0056d0 con 0x7f18980ab700 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.781+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18ac0d4050 con 0x7f18b4112f50 2024-03-04T07:38:33.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.781+0000 7f18aa000700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] conn(0x7f18980ab700 0x7f18980adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:33.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.781+0000 7f18b2a00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f18980a1f20 0x7f18980a43e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009e60 tx=0x7f18a400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:33.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.782+0000 7f18aa000700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] conn(0x7f18980ab700 0x7f18980adb20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:33.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.783+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== osd.0 v2:172.21.15.116:6802/3034556591 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f18ac0056d0 con 0x7f18980ab700 2024-03-04T07:38:33.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.801+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 --> [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18b4062cd0 con 0x7f18980ab700 2024-03-04T07:38:33.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.801+0000 7f18abe00700 1 -- 172.21.15.116:0/1333059111 <== osd.0 v2:172.21.15.116:6802/3034556591 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f18b4062cd0 con 0x7f18980ab700 2024-03-04T07:38:33.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] conn(0x7f18980ab700 msgr2=0x7f18980adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:33.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] conn(0x7f18980ab700 0x7f18980adb20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f18980a1f20 msgr2=0x7f18980a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f18980a1f20 0x7f18980a43e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009e60 tx=0x7f18a400b040 comp rx=0 tx=0).stop 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f18b4112f50 msgr2=0x7f18b41a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 --2- 172.21.15.116:0/1333059111 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f18b4112f50 0x7f18b41a3760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18b406b8d0 tx=0x7f18ac00da00 comp rx=0 tx=0).stop 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b3400700 1 -- 172.21.15.116:0/1333059111 reap_dead start 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 shutdown_connections 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 >> 172.21.15.116:0/1333059111 conn(0x7f18b40ffda0 msgr2=0x7f18b4102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 shutdown_connections 2024-03-04T07:38:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:27.802+0000 7f18b9a00700 1 -- 172.21.15.116:0/1333059111 wait complete. 2024-03-04T07:38:34.470 INFO:teuthology.orchestra.run.smithi116.stdout:141733920830 2024-03-04T07:38:34.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-04T07:38:35.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:34 smithi133 ceph-mon[36505]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:38:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:34 smithi116 ceph-mon[32150]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:38:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:34 smithi152 ceph-mon[37154]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:38:36.390 INFO:teuthology.orchestra.run.smithi116.stdout:219043332139 2024-03-04T07:38:36.390 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-04T07:38:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 -- 172.21.15.116:0/1277546376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 msgr2=0x7f598410f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 --2- 172.21.15.116:0/1277546376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 0x7f598410f010 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f597c00b0a0 tx=0x7f597c00b3b0 comp rx=0 tx=0).stop 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 -- 172.21.15.116:0/1277546376 shutdown_connections 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 --2- 172.21.15.116:0/1277546376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 0x7f598410f010 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 --2- 172.21.15.116:0/1277546376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5984102370 0x7f5984102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 --2- 172.21.15.116:0/1277546376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5984102d20 0x7f5984106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 -- 172.21.15.116:0/1277546376 >> 172.21.15.116:0/1277546376 conn(0x7f5984079220 msgr2=0x7f5984079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 -- 172.21.15.116:0/1277546376 shutdown_connections 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.602+0000 7f598aa00700 1 -- 172.21.15.116:0/1277546376 wait complete. 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 Processor -- start 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 -- start start 2024-03-04T07:38:36.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5984102370 0x7f59841a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5984102d20 0x7f59841a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 0x7f598419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984112460 con 0x7f5984102d20 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59841122e0 con 0x7f5984107440 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.604+0000 7f598aa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59841125e0 con 0x7f5984102370 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 0x7f598419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 0x7f598419a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57860/0 (socket says 172.21.15.116:57860) 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 -- 172.21.15.116:0/3795333343 learned_addr learned my addr 172.21.15.116:0/3795333343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5983400700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5984102d20 0x7f59841a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5983e00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5984102370 0x7f59841a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 -- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5984102370 msgr2=0x7f59841a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5984102370 0x7f59841a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:36.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 -- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5984102d20 msgr2=0x7f59841a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5984102d20 0x7f59841a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 -- 172.21.15.116:0/3795333343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598419afc0 con 0x7f5984107440 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5988a00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 0x7f598419a860 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f597c006070 tx=0x7f597c004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.605+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c021070 con 0x7f5984107440 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.606+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f597c003c00 con 0x7f5984107440 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.606+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c0046a0 con 0x7f5984107440 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.606+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598419b250 con 0x7f5984107440 2024-03-04T07:38:36.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.606+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59841a7570 con 0x7f5984107440 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.607+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5984100920 con 0x7f5984107440 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.608+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f597c007980 con 0x7f5984107440 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.610+0000 7f5980c00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f59680a1f20 0x7f59680a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.610+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f597c0ceef0 con 0x7f5984107440 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.610+0000 7f5983e00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f59680a1f20 0x7f59680a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.610+0000 7f5980c00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] conn(0x7f59680ab700 0x7f59680adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.610+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 --> [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f597c015ec0 con 0x7f59680ab700 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.610+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f597c0d1510 con 0x7f5984107440 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.611+0000 7f5983400700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] conn(0x7f59680ab700 0x7f59680adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.611+0000 7f5983e00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f59680a1f20 0x7f59680a43e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f598419b9a0 tx=0x7f5974009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.611+0000 7f5983400700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] conn(0x7f59680ab700 0x7f59680adb20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.612+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== osd.5 v2:172.21.15.152:6808/2293273292 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f597c015ec0 con 0x7f59680ab700 2024-03-04T07:38:36.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.630+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 --> [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5984103840 con 0x7f59680ab700 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.630+0000 7f5980c00700 1 -- 172.21.15.116:0/3795333343 <== osd.5 v2:172.21.15.152:6808/2293273292 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5984103840 con 0x7f59680ab700 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] conn(0x7f59680ab700 msgr2=0x7f59680adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] conn(0x7f59680ab700 0x7f59680adb20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f59680a1f20 msgr2=0x7f59680a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f59680a1f20 0x7f59680a43e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f598419b9a0 tx=0x7f5974009260 comp rx=0 tx=0).stop 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 msgr2=0x7f598419a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 --2- 172.21.15.116:0/3795333343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5984107440 0x7f598419a860 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f597c006070 tx=0x7f597c004d00 comp rx=0 tx=0).stop 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f5988a00700 1 -- 172.21.15.116:0/3795333343 reap_dead start 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 shutdown_connections 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 >> 172.21.15.116:0/3795333343 conn(0x7f5984079220 msgr2=0x7f59840fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 shutdown_connections 2024-03-04T07:38:36.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:36.631+0000 7f598aa00700 1 -- 172.21.15.116:0/3795333343 wait complete. 2024-03-04T07:38:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:36 smithi116 ceph-mon[32150]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:38:37.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:36 smithi133 ceph-mon[36505]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:38:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:36 smithi152 ceph-mon[37154]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:38:37.428 INFO:teuthology.orchestra.run.smithi116.stdout:34359738450 2024-03-04T07:38:37.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T07:38:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:38 smithi133 ceph-mon[36505]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:39.205 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:38 smithi116 ceph-mon[32150]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:38 smithi152 ceph-mon[37154]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:41.005 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:40 smithi116 ceph-mon[32150]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:41.073 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:40 smithi152 ceph-mon[37154]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:40 smithi133 ceph-mon[36505]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:41.488 INFO:teuthology.orchestra.run.smithi116.stdout:163208757305 2024-03-04T07:38:41.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-04T07:38:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:42 smithi116 ceph-mon[32150]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:43.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:42 smithi133 ceph-mon[36505]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.226+0000 7efcb9200700 1 -- 172.21.15.116:0/2546241629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb4074c10 msgr2=0x7efcb410e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.226+0000 7efcb9200700 1 --2- 172.21.15.116:0/2546241629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb4074c10 0x7efcb410e580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efca0009990 tx=0x7efca0009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.227+0000 7efcb9200700 1 -- 172.21.15.116:0/2546241629 shutdown_connections 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.227+0000 7efcb9200700 1 --2- 172.21.15.116:0/2546241629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb410eac0 0x7efcb4110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.227+0000 7efcb9200700 1 --2- 172.21.15.116:0/2546241629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb40742f0 0x7efcb40746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.227+0000 7efcb9200700 1 --2- 172.21.15.116:0/2546241629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb4074c10 0x7efcb410e580 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.227+0000 7efcb9200700 1 -- 172.21.15.116:0/2546241629 >> 172.21.15.116:0/2546241629 conn(0x7efcb4079220 msgr2=0x7efcb4079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:43.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.227+0000 7efcb9200700 1 -- 172.21.15.116:0/2546241629 shutdown_connections 2024-03-04T07:38:43.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.227+0000 7efcb9200700 1 -- 172.21.15.116:0/2546241629 wait complete. 2024-03-04T07:38:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 Processor -- start 2024-03-04T07:38:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 -- start start 2024-03-04T07:38:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb40742f0 0x7efcb41a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4074c10 0x7efcb41a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb410eac0 0x7efcb419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb41146d0 con 0x7efcb4074c10 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4114550 con 0x7efcb40742f0 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.228+0000 7efcb9200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4114850 con 0x7efcb410eac0 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb40742f0 0x7efcb41a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb40742f0 0x7efcb41a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57908/0 (socket says 172.21.15.116:57908) 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 -- 172.21.15.116:0/1907989297 learned_addr learned my addr 172.21.15.116:0/1907989297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1400700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4074c10 0x7efcb41a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb2800700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb410eac0 0x7efcb419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 -- 172.21.15.116:0/1907989297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb410eac0 msgr2=0x7efcb419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcb410eac0 0x7efcb419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 -- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4074c10 msgr2=0x7efcb41a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb4074c10 0x7efcb41a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 -- 172.21.15.116:0/1907989297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb419b0f0 con 0x7efcb40742f0 2024-03-04T07:38:43.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.229+0000 7efcb1e00700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb40742f0 0x7efcb41a0780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efca400e9c0 tx=0x7efca400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:43.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.230+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400cbd0 con 0x7efcb40742f0 2024-03-04T07:38:43.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.230+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca400cd70 con 0x7efcb40742f0 2024-03-04T07:38:43.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.230+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb419b3e0 con 0x7efcb40742f0 2024-03-04T07:38:43.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.230+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4010430 con 0x7efcb40742f0 2024-03-04T07:38:43.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.230+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb41a7650 con 0x7efcb40742f0 2024-03-04T07:38:43.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.231+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efcb40ffc50 con 0x7efcb40742f0 2024-03-04T07:38:43.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.233+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7efca4004750 con 0x7efcb40742f0 2024-03-04T07:38:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.234+0000 7efcaaa00700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc980a1f20 0x7efc980a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.234+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efca40ce210 con 0x7efcb40742f0 2024-03-04T07:38:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.234+0000 7efcb1400700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc980a1f20 0x7efc980a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.235+0000 7efcaaa00700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] conn(0x7efc980ab700 0x7efc980adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:43.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.235+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 --> [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efca4011e50 con 0x7efc980ab700 2024-03-04T07:38:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.235+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7efca40ce5c0 con 0x7efcb40742f0 2024-03-04T07:38:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.235+0000 7efcb2800700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] conn(0x7efc980ab700 0x7efc980adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.235+0000 7efcb1400700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc980a1f20 0x7efc980a43e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efcb4101bc0 tx=0x7efca0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:43.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.235+0000 7efcb2800700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] conn(0x7efc980ab700 0x7efc980adb20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:43.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.236+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== osd.9 v2:172.21.15.116:6826/3236031777 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efca4011e50 con 0x7efc980ab700 2024-03-04T07:38:43.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.254+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 --> [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efcb4075a20 con 0x7efc980ab700 2024-03-04T07:38:43.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.255+0000 7efcaaa00700 1 -- 172.21.15.116:0/1907989297 <== osd.9 v2:172.21.15.116:6826/3236031777 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efcb4075a20 con 0x7efc980ab700 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.255+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] conn(0x7efc980ab700 msgr2=0x7efc980adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.255+0000 7efcb9200700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] conn(0x7efc980ab700 0x7efc980adb20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc980a1f20 msgr2=0x7efc980a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efc980a1f20 0x7efc980a43e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efcb4101bc0 tx=0x7efca0005fd0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb40742f0 msgr2=0x7efcb41a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 --2- 172.21.15.116:0/1907989297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efcb40742f0 0x7efcb41a0780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efca400e9c0 tx=0x7efca400ecd0 comp rx=0 tx=0).stop 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb2800700 1 -- 172.21.15.116:0/1907989297 reap_dead start 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 shutdown_connections 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 >> 172.21.15.116:0/1907989297 conn(0x7efcb4079220 msgr2=0x7efcb40ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:43.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 shutdown_connections 2024-03-04T07:38:43.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:43.256+0000 7efcb9200700 1 -- 172.21.15.116:0/1907989297 wait complete. 2024-03-04T07:38:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:42 smithi152 ceph-mon[37154]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.809+0000 7f023e000700 1 -- 172.21.15.116:0/565857432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 msgr2=0x7f0238110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.809+0000 7f023e000700 1 --2- 172.21.15.116:0/565857432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 0x7f0238110d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f023000b0a0 tx=0x7f023000b3b0 comp rx=0 tx=0).stop 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.809+0000 7f023e000700 1 -- 172.21.15.116:0/565857432 shutdown_connections 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.810+0000 7f023e000700 1 --2- 172.21.15.116:0/565857432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 0x7f0238110d50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.810+0000 7f023e000700 1 --2- 172.21.15.116:0/565857432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02380744f0 0x7f02380748d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.810+0000 7f023e000700 1 --2- 172.21.15.116:0/565857432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238074e10 0x7f023810e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.810+0000 7f023e000700 1 -- 172.21.15.116:0/565857432 >> 172.21.15.116:0/565857432 conn(0x7f02380fdb70 msgr2=0x7f02380fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.810+0000 7f023e000700 1 -- 172.21.15.116:0/565857432 shutdown_connections 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.810+0000 7f023e000700 1 -- 172.21.15.116:0/565857432 wait complete. 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.811+0000 7f023e000700 1 Processor -- start 2024-03-04T07:38:44.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.811+0000 7f023e000700 1 -- start start 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f023e000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02380744f0 0x7f02381a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f023e000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238074e10 0x7f02381a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f023e000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 0x7f023819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f023e000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238114600 con 0x7f0238074e10 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f023e000700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238114480 con 0x7f023810e8e0 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f023e000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238114780 con 0x7f02380744f0 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f0237400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02380744f0 0x7f02381a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f0237e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 0x7f023819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f0236a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238074e10 0x7f02381a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f0237400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02380744f0 0x7f02381a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41864/0 (socket says 172.21.15.116:41864) 2024-03-04T07:38:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.812+0000 7f0237400700 1 -- 172.21.15.116:0/3102657520 learned_addr learned my addr 172.21.15.116:0/3102657520 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f0237e00700 1 -- 172.21.15.116:0/3102657520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02380744f0 msgr2=0x7f02381a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f0237e00700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02380744f0 0x7f02381a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f0237e00700 1 -- 172.21.15.116:0/3102657520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238074e10 msgr2=0x7f02381a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f0237e00700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238074e10 0x7f02381a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f0237e00700 1 -- 172.21.15.116:0/3102657520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023819afc0 con 0x7f023810e8e0 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f0237e00700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 0x7f023819a860 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f023000b070 tx=0x7f0230004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230021070 con 0x7f023810e8e0 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023819b250 con 0x7f023810e8e0 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0230007b80 con 0x7f023810e8e0 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02300052c0 con 0x7f023810e8e0 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.813+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02381a7570 con 0x7f023810e8e0 2024-03-04T07:38:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.814+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0238062850 con 0x7f023810e8e0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.816+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f0230024070 con 0x7f023810e8e0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.817+0000 7f022fe00700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f021c0a1ea0 0x7f021c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.818+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f02300cf0e0 con 0x7f023810e8e0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.818+0000 7f022fe00700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] conn(0x7f021c0ab680 0x7f021c0adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.818+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 --> [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f023001d8d0 con 0x7f021c0ab680 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.818+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02300d6050 con 0x7f023810e8e0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.818+0000 7f0237400700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f021c0a1ea0 0x7f021c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.818+0000 7f0236a00700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] conn(0x7f021c0ab680 0x7f021c0adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.819+0000 7f0237400700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f021c0a1ea0 0x7f021c0a4360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f023819b9a0 tx=0x7f0228007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.819+0000 7f0236a00700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] conn(0x7f021c0ab680 0x7f021c0adaa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.820+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== osd.10 v2:172.21.15.133:6824/2680230905 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f023001d8d0 con 0x7f021c0ab680 2024-03-04T07:38:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.837+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 --> [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0238062cd0 con 0x7f021c0ab680 2024-03-04T07:38:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f022fe00700 1 -- 172.21.15.116:0/3102657520 <== osd.10 v2:172.21.15.133:6824/2680230905 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0238062cd0 con 0x7f021c0ab680 2024-03-04T07:38:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] conn(0x7f021c0ab680 msgr2=0x7f021c0adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] conn(0x7f021c0ab680 0x7f021c0adaa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f021c0a1ea0 msgr2=0x7f021c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f021c0a1ea0 0x7f021c0a4360 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f023819b9a0 tx=0x7f0228007480 comp rx=0 tx=0).stop 2024-03-04T07:38:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 msgr2=0x7f023819a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 --2- 172.21.15.116:0/3102657520 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f023810e8e0 0x7f023819a860 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f023000b070 tx=0x7f0230004c80 comp rx=0 tx=0).stop 2024-03-04T07:38:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f0237e00700 1 -- 172.21.15.116:0/3102657520 reap_dead start 2024-03-04T07:38:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 shutdown_connections 2024-03-04T07:38:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 >> 172.21.15.116:0/3102657520 conn(0x7f02380fdb70 msgr2=0x7f02380fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 shutdown_connections 2024-03-04T07:38:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:41.838+0000 7f023e000700 1 -- 172.21.15.116:0/3102657520 wait complete. 2024-03-04T07:38:44.931 INFO:teuthology.orchestra.run.smithi116.stdout:309237645340 2024-03-04T07:38:44.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-04T07:38:45.060 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:44 smithi116 ceph-mon[32150]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:45.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:44 smithi133 ceph-mon[36505]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:44 smithi152 ceph-mon[37154]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 -- 172.21.15.116:0/2738433327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ea4102ad0 msgr2=0x7f8ea410ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 --2- 172.21.15.116:0/2738433327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ea4102ad0 0x7f8ea410ef10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00b0d0 tx=0x7f8e9c00b3e0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 -- 172.21.15.116:0/2738433327 shutdown_connections 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 --2- 172.21.15.116:0/2738433327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ea4102ad0 0x7f8ea410ef10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 --2- 172.21.15.116:0/2738433327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ea41080f0 0x7f8ea41084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 --2- 172.21.15.116:0/2738433327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ea4102110 0x7f8ea4102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 -- 172.21.15.116:0/2738433327 >> 172.21.15.116:0/2738433327 conn(0x7f8ea40791f0 msgr2=0x7f8ea4079600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 -- 172.21.15.116:0/2738433327 shutdown_connections 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.752+0000 7f8ea9800700 1 -- 172.21.15.116:0/2738433327 wait complete. 2024-03-04T07:38:45.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.753+0000 7f8ea9800700 1 Processor -- start 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.753+0000 7f8ea9800700 1 -- start start 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.753+0000 7f8ea9800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ea4102110 0x7f8ea41a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.753+0000 7f8ea9800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ea4102ad0 0x7f8ea41a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea9800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ea41080f0 0x7f8ea419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea9800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea41123c0 con 0x7f8ea4102110 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea9800700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4112240 con 0x7f8ea4102ad0 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea9800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4112540 con 0x7f8ea41080f0 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ea4102ad0 0x7f8ea41a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ea4102ad0 0x7f8ea41a0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57888/0 (socket says 172.21.15.116:57888) 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ea4102110 0x7f8ea41a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 -- 172.21.15.116:0/1769139667 learned_addr learned my addr 172.21.15.116:0/1769139667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea3400700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ea41080f0 0x7f8ea419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:45.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 -- 172.21.15.116:0/1769139667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ea41080f0 msgr2=0x7f8ea419a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ea41080f0 0x7f8ea419a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 -- 172.21.15.116:0/1769139667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ea4102110 msgr2=0x7f8ea41a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ea4102110 0x7f8ea41a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea2000700 1 -- 172.21.15.116:0/1769139667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea419af60 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.754+0000 7f8ea3400700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ea41080f0 0x7f8ea419a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.755+0000 7f8ea2000700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ea4102ad0 0x7f8ea41a0b40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000e9a0 tx=0x7f8e9000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.755+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90024070 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.755+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e9000cca0 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.755+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea419b250 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.755+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9001fd40 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.755+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea4108e30 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.756+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ea41014e0 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.758+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f8e90027070 con 0x7f8ea4102ad0 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.759+0000 7f8e9b400700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e880a1f20 0x7f8e880a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:45.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.759+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8e900d6150 con 0x7f8ea4102ad0 2024-03-04T07:38:45.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.759+0000 7f8ea2a00700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e880a1f20 0x7f8e880a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:45.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.760+0000 7f8e9b400700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] conn(0x7f8e880ab700 0x7f8e880adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:45.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.760+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 --> [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e9000e1a0 con 0x7f8e880ab700 2024-03-04T07:38:45.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.760+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e900d6500 con 0x7f8ea4102ad0 2024-03-04T07:38:45.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.760+0000 7f8ea3400700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] conn(0x7f8e880ab700 0x7f8e880adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:45.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.760+0000 7f8ea2a00700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e880a1f20 0x7f8e880a43e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009c60 tx=0x7f8e94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.760+0000 7f8ea3400700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] conn(0x7f8e880ab700 0x7f8e880adb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.761+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== osd.1 v2:172.21.15.133:6800/1626213243 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8e9000e1a0 con 0x7f8e880ab700 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.780+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 --> [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8ea41037a0 con 0x7f8e880ab700 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.780+0000 7f8e9b400700 1 -- 172.21.15.116:0/1769139667 <== osd.1 v2:172.21.15.133:6800/1626213243 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8ea41037a0 con 0x7f8e880ab700 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] conn(0x7f8e880ab700 msgr2=0x7f8e880adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] conn(0x7f8e880ab700 0x7f8e880adb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e880a1f20 msgr2=0x7f8e880a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e880a1f20 0x7f8e880a43e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009c60 tx=0x7f8e94009260 comp rx=0 tx=0).stop 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ea4102ad0 msgr2=0x7f8ea41a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 --2- 172.21.15.116:0/1769139667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8ea4102ad0 0x7f8ea41a0b40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000e9a0 tx=0x7f8e9000ecb0 comp rx=0 tx=0).stop 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea3400700 1 -- 172.21.15.116:0/1769139667 reap_dead start 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 shutdown_connections 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 >> 172.21.15.116:0/1769139667 conn(0x7f8ea40791f0 msgr2=0x7f8ea4100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 shutdown_connections 2024-03-04T07:38:45.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:42.781+0000 7f8ea9800700 1 -- 172.21.15.116:0/1769139667 wait complete. 2024-03-04T07:38:46.856 INFO:teuthology.orchestra.run.smithi116.stdout:55834574927 2024-03-04T07:38:46.857 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T07:38:47.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:46 smithi116 ceph-mon[32150]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:46 smithi133 ceph-mon[36505]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:46 smithi152 ceph-mon[37154]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:47.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.058+0000 7f27dd600700 1 -- 172.21.15.116:0/3188279393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 msgr2=0x7f27d81025f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:47.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.058+0000 7f27dd600700 1 --2- 172.21.15.116:0/3188279393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 0x7f27d81025f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009990 tx=0x7f27c8009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.059+0000 7f27dd600700 1 -- 172.21.15.116:0/3188279393 shutdown_connections 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.059+0000 7f27dd600700 1 --2- 172.21.15.116:0/3188279393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27d81072f0 0x7f27d810eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.059+0000 7f27dd600700 1 --2- 172.21.15.116:0/3188279393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 0x7f27d81025f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.059+0000 7f27dd600700 1 --2- 172.21.15.116:0/3188279393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27d8102bc0 0x7f27d8106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.059+0000 7f27dd600700 1 -- 172.21.15.116:0/3188279393 >> 172.21.15.116:0/3188279393 conn(0x7f27d80fdac0 msgr2=0x7f27d80ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.059+0000 7f27dd600700 1 -- 172.21.15.116:0/3188279393 shutdown_connections 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.059+0000 7f27dd600700 1 -- 172.21.15.116:0/3188279393 wait complete. 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.060+0000 7f27dd600700 1 Processor -- start 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.060+0000 7f27dd600700 1 -- start start 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dd600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 0x7f27d80743c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dd600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27d8102bc0 0x7f27d8074900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dd600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27d81072f0 0x7f27d8074e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dd600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d8112320 con 0x7f27d8102bc0 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dd600700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d81121a0 con 0x7f27d8102210 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dd600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d81124a0 con 0x7f27d81072f0 2024-03-04T07:38:47.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27d8102bc0 0x7f27d8074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 0x7f27d80743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 0x7f27d80743c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57938/0 (socket says 172.21.15.116:57938) 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27d8102bc0 0x7f27d8074900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42908/0 (socket says 172.21.15.116:42908) 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 -- 172.21.15.116:0/943812083 learned_addr learned my addr 172.21.15.116:0/943812083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dcc00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27d81072f0 0x7f27d8074e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 -- 172.21.15.116:0/943812083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27d81072f0 msgr2=0x7f27d8074e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27d81072f0 0x7f27d8074e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 -- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27d8102bc0 msgr2=0x7f27d8074900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27d8102bc0 0x7f27d8074900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27d7e00700 1 -- 172.21.15.116:0/943812083 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d81a6cb0 con 0x7f27d8102210 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.061+0000 7f27dcc00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27d81072f0 0x7f27d8074e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.062+0000 7f27d7e00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 0x7f27d80743c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f27c8000c00 tx=0x7f27c80057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:47.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.062+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8027070 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.062+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27c8022470 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.062+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d81a6f40 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.062+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8017650 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.062+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d81a7390 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.063+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f27d8062750 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.065+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f27c802a070 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.066+0000 7f27d4c00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27bc0a1fa0 0x7f27bc0a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.066+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f27c80cebf0 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.066+0000 7f27d7400700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27bc0a1fa0 0x7f27bc0a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.067+0000 7f27d4c00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] conn(0x7f27bc0ab780 0x7f27bc0adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.067+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 --> [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27bc0ae250 con 0x7f27bc0ab780 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.067+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27c80cefe0 con 0x7f27d8102210 2024-03-04T07:38:47.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.067+0000 7f27dcc00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] conn(0x7f27bc0ab780 0x7f27bc0adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.067+0000 7f27d7400700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27bc0a1fa0 0x7f27bc0a4460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f27d8108840 tx=0x7f27c40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.067+0000 7f27dcc00700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] conn(0x7f27bc0ab780 0x7f27bc0adba0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.068+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== osd.3 v2:172.21.15.116:6810/2894752093 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f27bc0ae250 con 0x7f27bc0ab780 2024-03-04T07:38:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.087+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 --> [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f27d8062bd0 con 0x7f27bc0ab780 2024-03-04T07:38:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.087+0000 7f27d4c00700 1 -- 172.21.15.116:0/943812083 <== osd.3 v2:172.21.15.116:6810/2894752093 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f27d8062bd0 con 0x7f27bc0ab780 2024-03-04T07:38:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] conn(0x7f27bc0ab780 msgr2=0x7f27bc0adba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] conn(0x7f27bc0ab780 0x7f27bc0adba0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27bc0a1fa0 msgr2=0x7f27bc0a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27bc0a1fa0 0x7f27bc0a4460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f27d8108840 tx=0x7f27c40073d0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 msgr2=0x7f27d80743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 --2- 172.21.15.116:0/943812083 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27d8102210 0x7f27d80743c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f27c8000c00 tx=0x7f27c80057c0 comp rx=0 tx=0).stop 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dcc00700 1 -- 172.21.15.116:0/943812083 reap_dead start 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 shutdown_connections 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 >> 172.21.15.116:0/943812083 conn(0x7f27d80fdac0 msgr2=0x7f27d80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 shutdown_connections 2024-03-04T07:38:47.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:44.088+0000 7f27dd600700 1 -- 172.21.15.116:0/943812083 wait complete. 2024-03-04T07:38:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.295+0000 7efd76a00700 1 -- 172.21.15.116:0/359287506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd70076f10 msgr2=0x7efd70112a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.295+0000 7efd76a00700 1 --2- 172.21.15.116:0/359287506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd70076f10 0x7efd70112a30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009990 tx=0x7efd5c009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.296+0000 7efd76a00700 1 -- 172.21.15.116:0/359287506 shutdown_connections 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.296+0000 7efd76a00700 1 --2- 172.21.15.116:0/359287506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd70112f70 0x7efd70115360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.296+0000 7efd76a00700 1 --2- 172.21.15.116:0/359287506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd700765f0 0x7efd700769d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.296+0000 7efd76a00700 1 --2- 172.21.15.116:0/359287506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd70076f10 0x7efd70112a30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.296+0000 7efd76a00700 1 -- 172.21.15.116:0/359287506 >> 172.21.15.116:0/359287506 conn(0x7efd700ffdc0 msgr2=0x7efd701021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.296+0000 7efd76a00700 1 -- 172.21.15.116:0/359287506 shutdown_connections 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.296+0000 7efd76a00700 1 -- 172.21.15.116:0/359287506 wait complete. 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 Processor -- start 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 -- start start 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd700765f0 0x7efd7019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd70076f10 0x7efd7019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd70112f70 0x7efd701a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70118ab0 con 0x7efd70076f10 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70118930 con 0x7efd70112f70 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.297+0000 7efd76a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70118c30 con 0x7efd700765f0 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd70076f10 0x7efd7019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd70076f10 0x7efd7019f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48028/0 (socket says 172.21.15.116:48028) 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 -- 172.21.15.116:0/672031619 learned_addr learned my addr 172.21.15.116:0/672031619 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6fe00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd700765f0 0x7efd7019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd74a00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd70112f70 0x7efd701a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 -- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd700765f0 msgr2=0x7efd7019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd700765f0 0x7efd7019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 -- 172.21.15.116:0/672031619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd70112f70 msgr2=0x7efd701a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efd70112f70 0x7efd701a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 -- 172.21.15.116:0/672031619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd701a3f10 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6fe00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd700765f0 0x7efd7019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.298+0000 7efd6f400700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd70076f10 0x7efd7019f440 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efd5c005910 tx=0x7efd5c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.299+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c027070 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.299+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd701a41a0 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.299+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd5c003d70 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.299+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c00f460 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.299+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd701aba80 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.300+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd70062850 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.302+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7efd5c02a070 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.303+0000 7efd6cc00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd540a1ea0 0x7efd540a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.303+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efd5c0cf3c0 con 0x7efd70076f10 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.303+0000 7efd6fe00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd540a1ea0 0x7efd540a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.304+0000 7efd6cc00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] conn(0x7efd540ab680 0x7efd540adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.304+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 --> [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd5c02be50 con 0x7efd540ab680 2024-03-04T07:38:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.304+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd5c0cf770 con 0x7efd70076f10 2024-03-04T07:38:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.304+0000 7efd6fe00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd540a1ea0 0x7efd540a4360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd60011490 tx=0x7efd60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.304+0000 7efd74a00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] conn(0x7efd540ab680 0x7efd540adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.305+0000 7efd74a00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] conn(0x7efd540ab680 0x7efd540adaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.306+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== osd.8 v2:172.21.15.152:6816/1470448295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efd5c02be50 con 0x7efd540ab680 2024-03-04T07:38:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.325+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 --> [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd70062c50 con 0x7efd540ab680 2024-03-04T07:38:48.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.326+0000 7efd6cc00700 1 -- 172.21.15.116:0/672031619 <== osd.8 v2:172.21.15.152:6816/1470448295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efd70062c50 con 0x7efd540ab680 2024-03-04T07:38:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.326+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] conn(0x7efd540ab680 msgr2=0x7efd540adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.326+0000 7efd76a00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] conn(0x7efd540ab680 0x7efd540adaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd540a1ea0 msgr2=0x7efd540a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efd540a1ea0 0x7efd540a4360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd60011490 tx=0x7efd60009260 comp rx=0 tx=0).stop 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd70076f10 msgr2=0x7efd7019f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 --2- 172.21.15.116:0/672031619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd70076f10 0x7efd7019f440 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efd5c005910 tx=0x7efd5c00bab0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd74a00700 1 -- 172.21.15.116:0/672031619 reap_dead start 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 shutdown_connections 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 >> 172.21.15.116:0/672031619 conn(0x7efd700ffdc0 msgr2=0x7efd701021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 shutdown_connections 2024-03-04T07:38:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.327+0000 7efd76a00700 1 -- 172.21.15.116:0/672031619 wait complete. 2024-03-04T07:38:48.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 -- 172.21.15.116:0/3710764285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1440742f0 msgr2=0x7fa1440746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 --2- 172.21.15.116:0/3710764285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1440742f0 0x7fa1440746d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa134009990 tx=0x7fa134009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 -- 172.21.15.116:0/3710764285 shutdown_connections 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 --2- 172.21.15.116:0/3710764285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa144113290 0x7fa144115680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 --2- 172.21.15.116:0/3710764285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1440742f0 0x7fa1440746d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 --2- 172.21.15.116:0/3710764285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa144074c10 0x7fa144112d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 -- 172.21.15.116:0/3710764285 >> 172.21.15.116:0/3710764285 conn(0x7fa144079220 msgr2=0x7fa144079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 -- 172.21.15.116:0/3710764285 shutdown_connections 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.075+0000 7fa14a200700 1 -- 172.21.15.116:0/3710764285 wait complete. 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.076+0000 7fa14a200700 1 Processor -- start 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa14a200700 1 -- start start 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa14a200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1440742f0 0x7fa14419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa14a200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa144074c10 0x7fa14419f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa14a200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa144113290 0x7fa1441a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa14a200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa144118af0 con 0x7fa144074c10 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa14a200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa144118970 con 0x7fa144113290 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa14a200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa144118c70 con 0x7fa1440742f0 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa143e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa144113290 0x7fa1441a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa143e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa144113290 0x7fa1441a37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56516/0 (socket says 172.21.15.116:56516) 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa143e00700 1 -- 172.21.15.116:0/3390388342 learned_addr learned my addr 172.21.15.116:0/3390388342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa142a00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa144074c10 0x7fa14419f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.077+0000 7fa143400700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1440742f0 0x7fa14419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa143e00700 1 -- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1440742f0 msgr2=0x7fa14419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa143e00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1440742f0 0x7fa14419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa143e00700 1 -- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa144074c10 msgr2=0x7fa14419f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa143e00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa144074c10 0x7fa14419f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa143e00700 1 -- 172.21.15.116:0/3390388342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1441a3f00 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa143400700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1440742f0 0x7fa14419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa142a00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa144074c10 0x7fa14419f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa143e00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa144113290 0x7fa1441a37c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00efc0 tx=0x7fa13c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0092e0 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa13c004bb0 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1441a4190 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c004280 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.078+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1441aba70 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.079+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa1441129c0 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.081+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fa13c01b030 con 0x7fa144113290 2024-03-04T07:38:48.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.082+0000 7fa13be00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1280a1f20 0x7fa1280a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.082+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa13c0cd840 con 0x7fa144113290 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.083+0000 7fa143400700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1280a1f20 0x7fa1280a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.083+0000 7fa13be00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] conn(0x7fa1280ab700 0x7fa1280adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.083+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 --> [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa13c013ec0 con 0x7fa1280ab700 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.083+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa13c0d4050 con 0x7fa144113290 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.083+0000 7fa142a00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] conn(0x7fa1280ab700 0x7fa1280adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.083+0000 7fa143400700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1280a1f20 0x7fa1280a43e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa13400b2e0 tx=0x7fa13401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.084+0000 7fa142a00700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] conn(0x7fa1280ab700 0x7fa1280adb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:48.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.085+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== osd.2 v2:172.21.15.152:6800/4177028869 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa13c013ec0 con 0x7fa1280ab700 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.103+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 --> [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa144075a20 con 0x7fa1280ab700 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.104+0000 7fa13be00700 1 -- 172.21.15.116:0/3390388342 <== osd.2 v2:172.21.15.152:6800/4177028869 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa144075a20 con 0x7fa1280ab700 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.104+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] conn(0x7fa1280ab700 msgr2=0x7fa1280adb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.104+0000 7fa14a200700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] conn(0x7fa1280ab700 0x7fa1280adb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1280a1f20 msgr2=0x7fa1280a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1280a1f20 0x7fa1280a43e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa13400b2e0 tx=0x7fa13401a040 comp rx=0 tx=0).stop 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa144113290 msgr2=0x7fa1441a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 --2- 172.21.15.116:0/3390388342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa144113290 0x7fa1441a37c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00efc0 tx=0x7fa13c00c3a0 comp rx=0 tx=0).stop 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa143e00700 1 -- 172.21.15.116:0/3390388342 reap_dead start 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 shutdown_connections 2024-03-04T07:38:48.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 >> 172.21.15.116:0/3390388342 conn(0x7fa144079220 msgr2=0x7fa144111fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:48.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 shutdown_connections 2024-03-04T07:38:48.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.105+0000 7fa14a200700 1 -- 172.21.15.116:0/3390388342 wait complete. 2024-03-04T07:38:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:48 smithi133 ceph-mon[36505]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:48 smithi116 ceph-mon[32150]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:48 smithi152 ceph-mon[37154]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 -- 172.21.15.116:0/582166529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 msgr2=0x7f07e806a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 --2- 172.21.15.116:0/582166529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 0x7f07e806a710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009990 tx=0x7f07d4009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 -- 172.21.15.116:0/582166529 shutdown_connections 2024-03-04T07:38:49.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 --2- 172.21.15.116:0/582166529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e8104ac0 0x7f07e8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 --2- 172.21.15.116:0/582166529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07e8106fb0 0x7f07e8069d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 --2- 172.21.15.116:0/582166529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 0x7f07e806a710 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 -- 172.21.15.116:0/582166529 >> 172.21.15.116:0/582166529 conn(0x7f07e8077ef0 msgr2=0x7f07e8078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 -- 172.21.15.116:0/582166529 shutdown_connections 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.168+0000 7f07eda00700 1 -- 172.21.15.116:0/582166529 wait complete. 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.169+0000 7f07eda00700 1 Processor -- start 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07eda00700 1 -- start start 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07eda00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 0x7f07e819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07eda00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e8104ac0 0x7f07e819f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07eda00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07e8106fb0 0x7f07e81a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07eda00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e81189b0 con 0x7f07e8104ac0 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07eda00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8118830 con 0x7f07e806a2b0 2024-03-04T07:38:49.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07eda00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8118b30 con 0x7f07e8106fb0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07e6a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 0x7f07e819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07e6a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 0x7f07e819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56528/0 (socket says 172.21.15.116:56528) 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07e6a00700 1 -- 172.21.15.116:0/3016249480 learned_addr learned my addr 172.21.15.116:0/3016249480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07e6000700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e8104ac0 0x7f07e819f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.170+0000 7f07e7400700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07e8106fb0 0x7f07e81a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07e6a00700 1 -- 172.21.15.116:0/3016249480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07e8106fb0 msgr2=0x7f07e81a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07e6a00700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07e8106fb0 0x7f07e81a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07e6a00700 1 -- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e8104ac0 msgr2=0x7f07e819f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07e6a00700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e8104ac0 0x7f07e819f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07e6a00700 1 -- 172.21.15.116:0/3016249480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e81a3e90 con 0x7f07e806a2b0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07e6a00700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 0x7f07e819ee80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07d800eb30 tx=0x7f07d800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d800cbd0 con 0x7f07e806a2b0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e81a4180 con 0x7f07e806a2b0 2024-03-04T07:38:49.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f07d8004500 con 0x7f07e806a2b0 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8010430 con 0x7f07e806a2b0 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.171+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e81aba00 con 0x7f07e806a2b0 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.172+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f07e8104500 con 0x7f07e806a2b0 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.174+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f07d8004020 con 0x7f07e806a2b0 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.175+0000 7f07df400700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f07cc0a1ea0 0x7f07cc0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.176+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f07d80ce180 con 0x7f07e806a2b0 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.176+0000 7f07df400700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] conn(0x7f07cc0ab680 0x7f07cc0adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:49.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.176+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 --> [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07d801ee50 con 0x7f07cc0ab680 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.176+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07d80d5050 con 0x7f07e806a2b0 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.176+0000 7f07e6000700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f07cc0a1ea0 0x7f07cc0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.176+0000 7f07e7400700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] conn(0x7f07cc0ab680 0x7f07cc0adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.177+0000 7f07e7400700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] conn(0x7f07cc0ab680 0x7f07cc0adaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.177+0000 7f07e6000700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f07cc0a1ea0 0x7f07cc0a4360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07e806ab60 tx=0x7f07d400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.178+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== osd.6 v2:172.21.15.116:6818/836391582 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f07d801ee50 con 0x7f07cc0ab680 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.196+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 --> [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f07e81015a0 con 0x7f07cc0ab680 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.196+0000 7f07df400700 1 -- 172.21.15.116:0/3016249480 <== osd.6 v2:172.21.15.116:6818/836391582 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f07e81015a0 con 0x7f07cc0ab680 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] conn(0x7f07cc0ab680 msgr2=0x7f07cc0adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] conn(0x7f07cc0ab680 0x7f07cc0adaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f07cc0a1ea0 msgr2=0x7f07cc0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f07cc0a1ea0 0x7f07cc0a4360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07e806ab60 tx=0x7f07d400b260 comp rx=0 tx=0).stop 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 msgr2=0x7f07e819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 --2- 172.21.15.116:0/3016249480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07e806a2b0 0x7f07e819ee80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07d800eb30 tx=0x7f07d800ee40 comp rx=0 tx=0).stop 2024-03-04T07:38:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07e7400700 1 -- 172.21.15.116:0/3016249480 reap_dead start 2024-03-04T07:38:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 shutdown_connections 2024-03-04T07:38:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.197+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 >> 172.21.15.116:0/3016249480 conn(0x7f07e8077ef0 msgr2=0x7f07e8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.198+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 shutdown_connections 2024-03-04T07:38:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:48.198+0000 7f07eda00700 1 -- 172.21.15.116:0/3016249480 wait complete. 2024-03-04T07:38:50.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:49 smithi133 ceph-mon[36505]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:50.248 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:49 smithi116 ceph-mon[32150]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:50.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:49 smithi152 ceph-mon[37154]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:50.316 INFO:teuthology.orchestra.run.smithi116.stdout:287762808866 2024-03-04T07:38:50.316 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-04T07:38:50.772 INFO:teuthology.orchestra.run.smithi116.stdout:107374182469 2024-03-04T07:38:50.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-04T07:38:51.184 INFO:teuthology.orchestra.run.smithi116.stdout:253403070505 2024-03-04T07:38:51.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-04T07:38:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:51.732 INFO:teuthology.orchestra.run.smithi116.stdout:77309411404 2024-03-04T07:38:51.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T07:38:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 -- 172.21.15.116:0/1898348236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 msgr2=0x7f1120108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 --2- 172.21.15.116:0/1898348236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 0x7f1120108b50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f110c009990 tx=0x7f110c009ca0 comp rx=0 tx=0).stop 2024-03-04T07:38:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 -- 172.21.15.116:0/1898348236 shutdown_connections 2024-03-04T07:38:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 --2- 172.21.15.116:0/1898348236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f112010e6d0 0x7f112010eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 --2- 172.21.15.116:0/1898348236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1120109090 0x7f11201155a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 --2- 172.21.15.116:0/1898348236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 0x7f1120108b50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:51.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 -- 172.21.15.116:0/1898348236 >> 172.21.15.116:0/1898348236 conn(0x7f1120075890 msgr2=0x7f1120077cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 -- 172.21.15.116:0/1898348236 shutdown_connections 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.906+0000 7f1127a00700 1 -- 172.21.15.116:0/1898348236 wait complete. 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.907+0000 7f1127a00700 1 Processor -- start 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.907+0000 7f1127a00700 1 -- start start 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1127a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 0x7f112019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1127a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1120109090 0x7f112019f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1127a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f112010e6d0 0x7f11201a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1127a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1120118a70 con 0x7f112010e6d0 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1127a00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11201188f0 con 0x7f11201086d0 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1127a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1120118bf0 con 0x7f1120109090 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 0x7f112019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 0x7f112019ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56560/0 (socket says 172.21.15.116:56560) 2024-03-04T07:38:51.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 -- 172.21.15.116:0/2602569204 learned_addr learned my addr 172.21.15.116:0/2602569204 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125a00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f112010e6d0 0x7f11201a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f111fe00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1120109090 0x7f112019f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 -- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1120109090 msgr2=0x7f112019f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1120109090 0x7f112019f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 -- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f112010e6d0 msgr2=0x7f11201a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f112010e6d0 0x7f11201a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.908+0000 7f1125000700 1 -- 172.21.15.116:0/2602569204 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11201a3e70 con 0x7f11201086d0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.909+0000 7f1125000700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 0x7f112019ee60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f111400e9a0 tx=0x7f111400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.909+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111400cbd0 con 0x7f11201086d0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.909+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11201a4160 con 0x7f11201086d0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.909+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f111400cd70 con 0x7f11201086d0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.909+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11140187c0 con 0x7f11201086d0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.909+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11201ab9e0 con 0x7f11201086d0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.910+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1120062850 con 0x7f11201086d0 2024-03-04T07:38:51.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.912+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f1114010430 con 0x7f11201086d0 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.914+0000 7f111d600700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11080a1ea0 0x7f11080a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.914+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f11140ce290 con 0x7f11201086d0 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.914+0000 7f111fe00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11080a1ea0 0x7f11080a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.914+0000 7f111d600700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] conn(0x7f11080ab680 0x7f11080adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.914+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 --> [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1114013ec0 con 0x7f11080ab680 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.914+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11140ce640 con 0x7f11201086d0 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.915+0000 7f1125a00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] conn(0x7f11080ab680 0x7f11080adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.915+0000 7f111fe00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11080a1ea0 0x7f11080a4360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f110c00b2e0 tx=0x7f110c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.915+0000 7f1125a00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] conn(0x7f11080ab680 0x7f11080adaa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.916+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== osd.11 v2:172.21.15.152:6824/1192411911 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1114013ec0 con 0x7f11080ab680 2024-03-04T07:38:51.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.934+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 --> [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1120062c50 con 0x7f11080ab680 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.935+0000 7f111d600700 1 -- 172.21.15.116:0/2602569204 <== osd.11 v2:172.21.15.152:6824/1192411911 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1120062c50 con 0x7f11080ab680 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] conn(0x7f11080ab680 msgr2=0x7f11080adaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] conn(0x7f11080ab680 0x7f11080adaa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11080a1ea0 msgr2=0x7f11080a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11080a1ea0 0x7f11080a4360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f110c00b2e0 tx=0x7f110c01a040 comp rx=0 tx=0).stop 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 msgr2=0x7f112019ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 --2- 172.21.15.116:0/2602569204 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11201086d0 0x7f112019ee60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f111400e9a0 tx=0x7f111400ed60 comp rx=0 tx=0).stop 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1125a00700 1 -- 172.21.15.116:0/2602569204 reap_dead start 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 shutdown_connections 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 >> 172.21.15.116:0/2602569204 conn(0x7f1120075890 msgr2=0x7f1120075db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:51.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 shutdown_connections 2024-03-04T07:38:51.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:49.936+0000 7f1127a00700 1 -- 172.21.15.116:0/2602569204 wait complete. 2024-03-04T07:38:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:52.713 INFO:teuthology.orchestra.run.smithi116.stdout:343597383705 2024-03-04T07:38:52.713 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-04T07:38:52.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:52 smithi133 ceph-mon[36505]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:52.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:52 smithi116 ceph-mon[32150]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:53.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:52 smithi152 ceph-mon[37154]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:53.242 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:53.243 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:54.763 INFO:teuthology.orchestra.run.smithi116.stdout:197568495668 2024-03-04T07:38:54.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-04T07:38:54.782 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:54.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:54 smithi116 ceph-mon[32150]: pgmap v343: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:54.984 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:54 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:38:55.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:54 smithi133 ceph-mon[36505]: pgmap v343: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:55.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:54 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:38:55.254 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:54 smithi152 ceph-mon[37154]: pgmap v343: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:55.254 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:54 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:38:55.493 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:55.499 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:38:56.968 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:56 smithi116 ceph-mon[32150]: pgmap v344: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:56.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:56 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:56.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:56 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:56.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:56 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:56.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:56 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:56 smithi133 ceph-mon[36505]: pgmap v344: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:57.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:56 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:56 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:56 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:56 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:56 smithi152 ceph-mon[37154]: pgmap v344: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:38:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:56 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:56 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:56 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:56 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:38:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-04T07:38:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-04T07:38:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-04T07:38:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-04T07:38:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [5, 7]}]: dispatch 2024-03-04T07:38:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-04T07:38:57.808 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-04T07:38:57.809 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-04T07:38:57.809 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:57 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-04T07:38:58.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [5, 7]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-04T07:38:58.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:57 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-04T07:38:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-04T07:38:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-04T07:38:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-04T07:38:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-04T07:38:58.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [5, 7]}]: dispatch 2024-03-04T07:38:58.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-04T07:38:58.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-04T07:38:58.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-04T07:38:58.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:57 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 -- 172.21.15.116:0/2601386404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 msgr2=0x7fd6d8102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 --2- 172.21.15.116:0/2601386404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d8102750 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009920 tx=0x7fd6c8009c30 comp rx=0 tx=0).stop 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 -- 172.21.15.116:0/2601386404 shutdown_connections 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 --2- 172.21.15.116:0/2601386404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d8102750 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 --2- 172.21.15.116:0/2601386404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6d8107440 0x7fd6d810f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 --2- 172.21.15.116:0/2601386404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6d8102d20 0x7fd6d8106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 -- 172.21.15.116:0/2601386404 >> 172.21.15.116:0/2601386404 conn(0x7fd6d8079220 msgr2=0x7fd6d8079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 -- 172.21.15.116:0/2601386404 shutdown_connections 2024-03-04T07:38:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.991+0000 7fd6de600700 1 -- 172.21.15.116:0/2601386404 wait complete. 2024-03-04T07:38:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.992+0000 7fd6de600700 1 Processor -- start 2024-03-04T07:38:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.992+0000 7fd6de600700 1 -- start start 2024-03-04T07:38:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.992+0000 7fd6de600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d80743c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:58.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.992+0000 7fd6de600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6d8102d20 0x7fd6d8074900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6de600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6d8107440 0x7fd6d8074e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6de600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d81124a0 con 0x7fd6d8102370 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6de600700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8112320 con 0x7fd6d8107440 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6de600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8112620 con 0x7fd6d8102d20 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d80743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d80743c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57990/0 (socket says 172.21.15.116:57990) 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 -- 172.21.15.116:0/1891793531 learned_addr learned my addr 172.21.15.116:0/1891793531 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7e00700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6d8107440 0x7fd6d8074e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d6a00700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6d8102d20 0x7fd6d8074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 -- 172.21.15.116:0/1891793531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6d8102d20 msgr2=0x7fd6d8074900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6d8102d20 0x7fd6d8074900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 -- 172.21.15.116:0/1891793531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6d8107440 msgr2=0x7fd6d8074e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6d8107440 0x7fd6d8074e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7400700 1 -- 172.21.15.116:0/1891793531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d81a6fe0 con 0x7fd6d8102370 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.993+0000 7fd6d7e00700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6d8107440 0x7fd6d8074e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.994+0000 7fd6d7400700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d80743c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009230 tx=0x7fd6c800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.994+0000 7fd6cfe00700 1 -- 172.21.15.116:0/1891793531 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8027070 con 0x7fd6d8102370 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.994+0000 7fd6cfe00700 1 -- 172.21.15.116:0/1891793531 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6c80055a0 con 0x7fd6d8102370 2024-03-04T07:38:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.994+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d81a7270 con 0x7fd6d8102370 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: pgmap v345: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [5, 7]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-04T07:38:58.997 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-04T07:38:58.998 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:58 smithi116 ceph-mon[32150]: osdmap e86: 12 total, 12 up, 12 in 2024-03-04T07:38:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.994+0000 7fd6cfe00700 1 -- 172.21.15.116:0/1891793531 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8004910 con 0x7fd6d8102370 2024-03-04T07:38:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.994+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d81a76c0 con 0x7fd6d8102370 2024-03-04T07:38:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.996+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d8068d80 con 0x7fd6d8102370 2024-03-04T07:38:58.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.998+0000 7fd6cfe00700 1 -- 172.21.15.116:0/1891793531 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fd6c802a070 con 0x7fd6d8102370 2024-03-04T07:38:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.999+0000 7fd6cfe00700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6bc0a1f20 0x7fd6bc0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:38:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:58.999+0000 7fd6cfe00700 1 -- 172.21.15.116:0/1891793531 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd6c80ced20 con 0x7fd6d8102370 2024-03-04T07:38:58.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.000+0000 7fd6d6a00700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6bc0a1f20 0x7fd6bc0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:38:59.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.000+0000 7fd6d6a00700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6bc0a1f20 0x7fd6bc0a43e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009750 tx=0x7fd6c4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:38:59.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.003+0000 7fd6cfe00700 1 -- 172.21.15.116:0/1891793531 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd6c8031020 con 0x7fd6d8102370 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: pgmap v345: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [5, 7]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-04T07:38:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:58 smithi133 ceph-mon[36505]: osdmap e86: 12 total, 12 up, 12 in 2024-03-04T07:38:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.237+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd6d802d9b0 con 0x7fd6d8102370 2024-03-04T07:38:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.238+0000 7fd6cfe00700 1 -- 172.21.15.116:0/1891793531 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd6c8013030 con 0x7fd6d8102370 2024-03-04T07:38:59.238 INFO:teuthology.orchestra.run.smithi116.stdout:219043332144 2024-03-04T07:38:59.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6bc0a1f20 msgr2=0x7fd6bc0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:59.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6bc0a1f20 0x7fd6bc0a43e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009750 tx=0x7fd6c4009310 comp rx=0 tx=0).stop 2024-03-04T07:38:59.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 msgr2=0x7fd6d80743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:38:59.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d80743c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009230 tx=0x7fd6c800fa00 comp rx=0 tx=0).stop 2024-03-04T07:38:59.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 shutdown_connections 2024-03-04T07:38:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d8102370 0x7fd6d80743c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd6bc0a1f20 0x7fd6bc0a43e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd6d8107440 0x7fd6d8074e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 --2- 172.21.15.116:0/1891793531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd6d8102d20 0x7fd6d8074900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:38:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 >> 172.21.15.116:0/1891793531 conn(0x7fd6d8079220 msgr2=0x7fd6d8100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:38:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 shutdown_connections 2024-03-04T07:38:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:38:59.241+0000 7fd6de600700 1 -- 172.21.15.116:0/1891793531 wait complete. 2024-03-04T07:38:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: pgmap v345: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:38:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-04T07:38:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-04T07:38:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-04T07:38:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-04T07:38:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [5, 7]}]': finished 2024-03-04T07:38:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-04T07:38:59.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-04T07:38:59.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-04T07:38:59.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-04T07:38:59.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:58 smithi152 ceph-mon[37154]: osdmap e86: 12 total, 12 up, 12 in 2024-03-04T07:38:59.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:59 smithi116 ceph-mon[32150]: osdmap e87: 12 total, 12 up, 12 in 2024-03-04T07:38:59.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:38:59 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1891793531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T07:39:00.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:59 smithi133 ceph-mon[36505]: osdmap e87: 12 total, 12 up, 12 in 2024-03-04T07:39:00.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:38:59 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1891793531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T07:39:00.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:59 smithi152 ceph-mon[37154]: osdmap e87: 12 total, 12 up, 12 in 2024-03-04T07:39:00.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:38:59 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1891793531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T07:39:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:00 smithi133 ceph-mon[36505]: pgmap v348: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:01.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:00 smithi133 ceph-mon[36505]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-04T07:39:01.168 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:00 smithi116 ceph-mon[32150]: pgmap v348: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:01.168 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:00 smithi116 ceph-mon[32150]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-04T07:39:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:00 smithi152 ceph-mon[37154]: pgmap v348: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:00 smithi152 ceph-mon[37154]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-04T07:39:02.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:01 smithi116 ceph-mon[32150]: pgmap v349: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:02.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:01 smithi133 ceph-mon[36505]: pgmap v349: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:02.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:01 smithi152 ceph-mon[37154]: pgmap v349: 97 pgs: 2 peering, 95 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 -- 172.21.15.116:0/2648866745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074db0 msgr2=0x7fd86010e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 --2- 172.21.15.116:0/2648866745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074db0 0x7fd86010e300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009990 tx=0x7fd84c009ca0 comp rx=0 tx=0).stop 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 -- 172.21.15.116:0/2648866745 shutdown_connections 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 --2- 172.21.15.116:0/2648866745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860074490 0x7fd860074870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 --2- 172.21.15.116:0/2648866745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd86010e840 0x7fd860110c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 --2- 172.21.15.116:0/2648866745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074db0 0x7fd86010e300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 -- 172.21.15.116:0/2648866745 >> 172.21.15.116:0/2648866745 conn(0x7fd8600fda90 msgr2=0x7fd8600ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.886+0000 7fd85f200700 1 -- 172.21.15.116:0/2648866745 shutdown_connections 2024-03-04T07:39:02.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.887+0000 7fd85f200700 1 -- 172.21.15.116:0/2648866745 wait complete. 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 Processor -- start 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 -- start start 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074490 0x7fd8601a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860074db0 0x7fd8601a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd86010e840 0x7fd86019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860114550 con 0x7fd860074db0 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601143d0 con 0x7fd860074490 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.888+0000 7fd85f200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601146d0 con 0x7fd86010e840 2024-03-04T07:39:02.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074490 0x7fd8601a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074490 0x7fd8601a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44396/0 (socket says 172.21.15.116:44396) 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 -- 172.21.15.116:0/2138411338 learned_addr learned my addr 172.21.15.116:0/2138411338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85d400700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860074db0 0x7fd8601a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85e800700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd86010e840 0x7fd86019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 -- 172.21.15.116:0/2138411338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd86010e840 msgr2=0x7fd86019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd86010e840 0x7fd86019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 -- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860074db0 msgr2=0x7fd8601a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860074db0 0x7fd8601a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 -- 172.21.15.116:0/2138411338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86019afd0 con 0x7fd860074490 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85e800700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd86010e840 0x7fd86019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:39:02.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.889+0000 7fd85de00700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074490 0x7fd8601a0670 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd85000eb30 tx=0x7fd85000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:02.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.890+0000 7fd856a00700 1 -- 172.21.15.116:0/2138411338 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85000cc90 con 0x7fd860074490 2024-03-04T07:39:02.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.890+0000 7fd856a00700 1 -- 172.21.15.116:0/2138411338 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd850004500 con 0x7fd860074490 2024-03-04T07:39:02.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.890+0000 7fd856a00700 1 -- 172.21.15.116:0/2138411338 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850010430 con 0x7fd860074490 2024-03-04T07:39:02.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.890+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86019b2c0 con 0x7fd860074490 2024-03-04T07:39:02.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.890+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8601a74d0 con 0x7fd860074490 2024-03-04T07:39:02.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.891+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860075b40 con 0x7fd860074490 2024-03-04T07:39:02.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.893+0000 7fd856a00700 1 -- 172.21.15.116:0/2138411338 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fd850003f70 con 0x7fd860074490 2024-03-04T07:39:02.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.894+0000 7fd856a00700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8440a1f20 0x7fd8440a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:02.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.894+0000 7fd856a00700 1 -- 172.21.15.116:0/2138411338 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd8500ce230 con 0x7fd860074490 2024-03-04T07:39:02.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.894+0000 7fd85d400700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8440a1f20 0x7fd8440a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:02.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.895+0000 7fd85d400700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8440a1f20 0x7fd8440a43e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd860101690 tx=0x7fd84c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:02.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:02.896+0000 7fd856a00700 1 -- 172.21.15.116:0/2138411338 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd85009b4c0 con 0x7fd860074490 2024-03-04T07:39:03.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.127+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd86002d9b0 con 0x7fd860074490 2024-03-04T07:39:03.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.128+0000 7fd856a00700 1 -- 172.21.15.116:0/2138411338 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd85009adc0 con 0x7fd860074490 2024-03-04T07:39:03.128 INFO:teuthology.orchestra.run.smithi116.stdout:141733920836 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.130+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8440a1f20 msgr2=0x7fd8440a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.130+0000 7fd85f200700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8440a1f20 0x7fd8440a43e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd860101690 tx=0x7fd84c005fd0 comp rx=0 tx=0).stop 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.130+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074490 msgr2=0x7fd8601a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.130+0000 7fd85f200700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074490 0x7fd8601a0670 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd85000eb30 tx=0x7fd85000ee40 comp rx=0 tx=0).stop 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.130+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 shutdown_connections 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.130+0000 7fd85f200700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8440a1f20 0x7fd8440a43e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.131+0000 7fd85f200700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd86010e840 0x7fd86019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.131+0000 7fd85f200700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd860074490 0x7fd8601a0670 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.131+0000 7fd85f200700 1 --2- 172.21.15.116:0/2138411338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd860074db0 0x7fd8601a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.131+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 >> 172.21.15.116:0/2138411338 conn(0x7fd8600fda90 msgr2=0x7fd8600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.131+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 shutdown_connections 2024-03-04T07:39:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:03.131+0000 7fd85f200700 1 -- 172.21.15.116:0/2138411338 wait complete. 2024-03-04T07:39:03.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:03 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2138411338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T07:39:03.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:03 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2138411338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T07:39:03.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:03 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2138411338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T07:39:03.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332139 got 219043332144 for osd.7 2024-03-04T07:39:03.632 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:03.927 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:03.928 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:04.446 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:04 smithi116 ceph-mon[32150]: pgmap v350: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:04.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:04 smithi152 ceph-mon[37154]: pgmap v350: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:04.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:04 smithi133 ceph-mon[36505]: pgmap v350: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:05.290 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:05.495 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:05 smithi116 ceph-mon[32150]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-04T07:39:05.495 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:05 smithi116 ceph-mon[32150]: Cluster is now healthy 2024-03-04T07:39:05.495 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:05 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3779592319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T07:39:05.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:05 smithi152 ceph-mon[37154]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-04T07:39:05.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:05 smithi152 ceph-mon[37154]: Cluster is now healthy 2024-03-04T07:39:05.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:05 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3779592319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T07:39:05.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:05 smithi133 ceph-mon[36505]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-04T07:39:05.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:05 smithi133 ceph-mon[36505]: Cluster is now healthy 2024-03-04T07:39:05.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:05 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3779592319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T07:39:06.464 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:06 smithi116 ceph-mon[32150]: pgmap v351: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 0 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:06.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:06 smithi152 ceph-mon[37154]: pgmap v351: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 0 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:06.586 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:06 smithi133 ceph-mon[36505]: pgmap v351: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 0 op/s; 14 B/s, 0 objects/s recovering 2024-03-04T07:39:08.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.081+0000 7f04afe00700 1 -- 172.21.15.116:0/767842448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0100f60 msgr2=0x7f04b0101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.081+0000 7f04afe00700 1 --2- 172.21.15.116:0/767842448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0100f60 0x7f04b0101340 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f04a000b0a0 tx=0x7f04a000b3b0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.082+0000 7f04afe00700 1 -- 172.21.15.116:0/767842448 shutdown_connections 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.082+0000 7f04afe00700 1 --2- 172.21.15.116:0/767842448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b0106370 0x7f04b0106820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.082+0000 7f04afe00700 1 --2- 172.21.15.116:0/767842448 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0100f60 0x7f04b0101340 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.082+0000 7f04afe00700 1 --2- 172.21.15.116:0/767842448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04b0101880 0x7f04b0105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.082+0000 7f04afe00700 1 -- 172.21.15.116:0/767842448 >> 172.21.15.116:0/767842448 conn(0x7f04b00fc980 msgr2=0x7f04b00feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.082+0000 7f04afe00700 1 -- 172.21.15.116:0/767842448 shutdown_connections 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.082+0000 7f04afe00700 1 -- 172.21.15.116:0/767842448 wait complete. 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 Processor -- start 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 -- start start 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b0100f60 0x7f04b019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:08.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04b0101880 0x7f04b019f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0106370 0x7f04b01a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0118950 con 0x7f04b0100f60 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01187d0 con 0x7f04b0106370 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.083+0000 7f04afe00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0118ad0 con 0x7f04b0101880 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04af400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0106370 0x7f04b01a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04af400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0106370 0x7f04b01a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44422/0 (socket says 172.21.15.116:44422) 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04aea00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b0100f60 0x7f04b019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04af400700 1 -- 172.21.15.116:0/3779592319 learned_addr learned my addr 172.21.15.116:0/3779592319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04a6000700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04b0101880 0x7f04b019f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04aea00700 1 -- 172.21.15.116:0/3779592319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04b0101880 msgr2=0x7f04b019f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04aea00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04b0101880 0x7f04b019f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04aea00700 1 -- 172.21.15.116:0/3779592319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0106370 msgr2=0x7f04b01a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:08.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04aea00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0106370 0x7f04b01a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04aea00700 1 -- 172.21.15.116:0/3779592319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b01a3dd0 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04a6000700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04b0101880 0x7f04b019f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04af400700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0106370 0x7f04b01a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.084+0000 7f04aea00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b0100f60 0x7f04b019edc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f04a000b780 tx=0x7f04a0004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.085+0000 7f04a7e00700 1 -- 172.21.15.116:0/3779592319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0021070 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.085+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b01a4060 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.085+0000 7f04a7e00700 1 -- 172.21.15.116:0/3779592319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04a0007b80 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.085+0000 7f04a7e00700 1 -- 172.21.15.116:0/3779592319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a00052a0 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.085+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b01ab940 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.086+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b0068d80 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.088+0000 7f04a7e00700 1 -- 172.21.15.116:0/3779592319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f04a0024070 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.089+0000 7f04a7e00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04940a1ea0 0x7f04940a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.089+0000 7f04a7e00700 1 -- 172.21.15.116:0/3779592319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f04a000f020 con 0x7f04b0100f60 2024-03-04T07:39:08.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.090+0000 7f04a6000700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04940a1ea0 0x7f04940a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.090+0000 7f04a6000700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04940a1ea0 0x7f04940a4360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04b0100000 tx=0x7f0498009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.094+0000 7f04a7e00700 1 -- 172.21.15.116:0/3779592319 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f04a002b020 con 0x7f04b0100f60 2024-03-04T07:39:08.633 INFO:teuthology.orchestra.run.smithi116.stdout:34359738457 2024-03-04T07:39:08.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.319+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f04b01abb10 con 0x7f04b0100f60 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.319+0000 7f04a7e00700 1 -- 172.21.15.116:0/3779592319 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f04a009c750 con 0x7f04b0100f60 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04940a1ea0 msgr2=0x7f04940a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04940a1ea0 0x7f04940a4360 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04b0100000 tx=0x7f0498009260 comp rx=0 tx=0).stop 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b0100f60 msgr2=0x7f04b019edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b0100f60 0x7f04b019edc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f04a000b780 tx=0x7f04a0004d00 comp rx=0 tx=0).stop 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 shutdown_connections 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04b0100f60 0x7f04b019edc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f04940a1ea0 0x7f04940a4360 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f04b0106370 0x7f04b01a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 --2- 172.21.15.116:0/3779592319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04b0101880 0x7f04b019f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 >> 172.21.15.116:0/3779592319 conn(0x7f04b00fc980 msgr2=0x7f04b0103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 shutdown_connections 2024-03-04T07:39:08.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:04.322+0000 7f04afe00700 1 -- 172.21.15.116:0/3779592319 wait complete. 2024-03-04T07:39:08.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:08 smithi116 ceph-mon[32150]: pgmap v352: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 844 B/s rd, 1 op/s; 12 B/s, 0 objects/s recovering 2024-03-04T07:39:08.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:08 smithi133 ceph-mon[36505]: pgmap v352: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 844 B/s rd, 1 op/s; 12 B/s, 0 objects/s recovering 2024-03-04T07:39:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:08 smithi152 ceph-mon[37154]: pgmap v352: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 844 B/s rd, 1 op/s; 12 B/s, 0 objects/s recovering 2024-03-04T07:39:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 -- 172.21.15.116:0/1873533232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a41072f0 msgr2=0x7f83a410eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 --2- 172.21.15.116:0/1873533232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a41072f0 0x7f83a410eec0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f839c00b0a0 tx=0x7f839c00b3b0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 -- 172.21.15.116:0/1873533232 shutdown_connections 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 --2- 172.21.15.116:0/1873533232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a41072f0 0x7f83a410eec0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 --2- 172.21.15.116:0/1873533232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83a4102210 0x7f83a41025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 --2- 172.21.15.116:0/1873533232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a4102bc0 0x7f83a4106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 -- 172.21.15.116:0/1873533232 >> 172.21.15.116:0/1873533232 conn(0x7f83a40fdac0 msgr2=0x7f83a40ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 -- 172.21.15.116:0/1873533232 shutdown_connections 2024-03-04T07:39:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.108+0000 7f83a3200700 1 -- 172.21.15.116:0/1873533232 wait complete. 2024-03-04T07:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.110+0000 7f83a3200700 1 Processor -- start 2024-03-04T07:39:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.110+0000 7f83a3200700 1 -- start start 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a3200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a4102210 0x7f83a40743c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a3200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83a4102bc0 0x7f83a4074900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a3200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a41072f0 0x7f83a4074e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a3200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a4112330 con 0x7f83a41072f0 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a3200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41121b0 con 0x7f83a4102210 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a3200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41124b0 con 0x7f83a4102bc0 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a4102210 0x7f83a40743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a41072f0 0x7f83a4074e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a4102210 0x7f83a40743c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37204/0 (socket says 172.21.15.116:37204) 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 -- 172.21.15.116:0/4026166777 learned_addr learned my addr 172.21.15.116:0/4026166777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a41072f0 0x7f83a4074e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40296/0 (socket says 172.21.15.116:40296) 2024-03-04T07:39:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1400700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83a4102bc0 0x7f83a4074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 -- 172.21.15.116:0/4026166777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83a4102bc0 msgr2=0x7f83a4074900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83a4102bc0 0x7f83a4074900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 -- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a41072f0 msgr2=0x7f83a4074e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a41072f0 0x7f83a4074e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.111+0000 7f83a1e00700 1 -- 172.21.15.116:0/4026166777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a41a6d50 con 0x7f83a4102210 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.112+0000 7f83a1e00700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a4102210 0x7f83a40743c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f839400d950 tx=0x7f839400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.112+0000 7f839aa00700 1 -- 172.21.15.116:0/4026166777 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394016070 con 0x7f83a4102210 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.112+0000 7f839aa00700 1 -- 172.21.15.116:0/4026166777 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f839400fd70 con 0x7f83a4102210 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.112+0000 7f839aa00700 1 -- 172.21.15.116:0/4026166777 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839400e610 con 0x7f83a4102210 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.112+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a41a6fe0 con 0x7f83a4102210 2024-03-04T07:39:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.112+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a41a74a0 con 0x7f83a4102210 2024-03-04T07:39:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.113+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a4068d80 con 0x7f83a4102210 2024-03-04T07:39:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.115+0000 7f839aa00700 1 -- 172.21.15.116:0/4026166777 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f8394019070 con 0x7f83a4102210 2024-03-04T07:39:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.116+0000 7f839aa00700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83880a1f20 0x7f83880a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.116+0000 7f839aa00700 1 -- 172.21.15.116:0/4026166777 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f83940cd970 con 0x7f83a4102210 2024-03-04T07:39:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.117+0000 7f83a1400700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83880a1f20 0x7f83880a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.117+0000 7f83a1400700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83880a1f20 0x7f83880a43e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f83a41011f0 tx=0x7f8390009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.120+0000 7f839aa00700 1 -- 172.21.15.116:0/4026166777 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f839409ac00 con 0x7f83a4102210 2024-03-04T07:39:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.353+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f83a402d9b0 con 0x7f83a4102210 2024-03-04T07:39:09.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.353+0000 7f839aa00700 1 -- 172.21.15.116:0/4026166777 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f839409a500 con 0x7f83a4102210 2024-03-04T07:39:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:287762808871 2024-03-04T07:39:09.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83880a1f20 msgr2=0x7f83880a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83880a1f20 0x7f83880a43e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f83a41011f0 tx=0x7f8390009260 comp rx=0 tx=0).stop 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a4102210 msgr2=0x7f83a40743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a4102210 0x7f83a40743c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f839400d950 tx=0x7f839400dc60 comp rx=0 tx=0).stop 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 shutdown_connections 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83a41072f0 0x7f83a4074e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f83880a1f20 0x7f83880a43e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f83a4102210 0x7f83a40743c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 --2- 172.21.15.116:0/4026166777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83a4102bc0 0x7f83a4074900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.356+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 >> 172.21.15.116:0/4026166777 conn(0x7f83a40fdac0 msgr2=0x7f83a40fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.357+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 shutdown_connections 2024-03-04T07:39:09.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:09.357+0000 7f83a3200700 1 -- 172.21.15.116:0/4026166777 wait complete. 2024-03-04T07:39:09.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:09 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4026166777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-04T07:39:09.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:09 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4026166777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-04T07:39:10.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:09 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4026166777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-04T07:39:10.409 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920836 for osd.4 2024-03-04T07:39:10.409 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:10.602 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:10 smithi116 ceph-mon[32150]: pgmap v353: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 765 B/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-03-04T07:39:10.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.635+0000 7faf66c00700 1 -- 172.21.15.116:0/1163745199 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf60074c10 msgr2=0x7faf6010e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.635+0000 7faf66c00700 1 --2- 172.21.15.116:0/1163745199 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf60074c10 0x7faf6010e580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009990 tx=0x7faf4c009ca0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.636+0000 7faf66c00700 1 -- 172.21.15.116:0/1163745199 shutdown_connections 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.636+0000 7faf66c00700 1 --2- 172.21.15.116:0/1163745199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6010eac0 0x7faf60110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.636+0000 7faf66c00700 1 --2- 172.21.15.116:0/1163745199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf600742f0 0x7faf600746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.636+0000 7faf66c00700 1 --2- 172.21.15.116:0/1163745199 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf60074c10 0x7faf6010e580 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.636+0000 7faf66c00700 1 -- 172.21.15.116:0/1163745199 >> 172.21.15.116:0/1163745199 conn(0x7faf600791f0 msgr2=0x7faf60079600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.636+0000 7faf66c00700 1 -- 172.21.15.116:0/1163745199 shutdown_connections 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.636+0000 7faf66c00700 1 -- 172.21.15.116:0/1163745199 wait complete. 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 Processor -- start 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 -- start start 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf600742f0 0x7faf601a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:10.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60074c10 0x7faf601a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf6010eac0 0x7faf6019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf601145e0 con 0x7faf60074c10 2024-03-04T07:39:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf60114460 con 0x7faf600742f0 2024-03-04T07:39:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.637+0000 7faf66c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf60114760 con 0x7faf6010eac0 2024-03-04T07:39:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf600742f0 0x7faf601a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60074c10 0x7faf601a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:10.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf600742f0 0x7faf601a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37224/0 (socket says 172.21.15.116:37224) 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5fe00700 1 -- 172.21.15.116:0/473410356 learned_addr learned my addr 172.21.15.116:0/473410356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf64c00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf6010eac0 0x7faf6019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5f400700 1 -- 172.21.15.116:0/473410356 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf6010eac0 msgr2=0x7faf6019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5f400700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf6010eac0 0x7faf6019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5f400700 1 -- 172.21.15.116:0/473410356 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf600742f0 msgr2=0x7faf601a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5f400700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf600742f0 0x7faf601a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5f400700 1 -- 172.21.15.116:0/473410356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6019b060 con 0x7faf60074c10 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5fe00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf600742f0 0x7faf601a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.638+0000 7faf5f400700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60074c10 0x7faf601a0c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faf4c000c00 tx=0x7faf4c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.639+0000 7faf5cc00700 1 -- 172.21.15.116:0/473410356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c026070 con 0x7faf60074c10 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.639+0000 7faf5cc00700 1 -- 172.21.15.116:0/473410356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf4c003d70 con 0x7faf60074c10 2024-03-04T07:39:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.639+0000 7faf5cc00700 1 -- 172.21.15.116:0/473410356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c00fdc0 con 0x7faf60074c10 2024-03-04T07:39:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.639+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6019b2f0 con 0x7faf60074c10 2024-03-04T07:39:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.639+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf601a75c0 con 0x7faf60074c10 2024-03-04T07:39:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.640+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf60068d80 con 0x7faf60074c10 2024-03-04T07:39:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.642+0000 7faf5cc00700 1 -- 172.21.15.116:0/473410356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7faf4c029070 con 0x7faf60074c10 2024-03-04T07:39:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.643+0000 7faf5cc00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf440a1ea0 0x7faf440a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.643+0000 7faf5cc00700 1 -- 172.21.15.116:0/473410356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7faf4c0cecb0 con 0x7faf60074c10 2024-03-04T07:39:10.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.644+0000 7faf5fe00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf440a1ea0 0x7faf440a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:10.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.644+0000 7faf5fe00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf440a1ea0 0x7faf440a4360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf60101e70 tx=0x7faf50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:10.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.648+0000 7faf5cc00700 1 -- 172.21.15.116:0/473410356 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faf4c05d280 con 0x7faf60074c10 2024-03-04T07:39:10.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:10 smithi133 ceph-mon[36505]: pgmap v353: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 765 B/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-03-04T07:39:10.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.870+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7faf6002d9b0 con 0x7faf60074c10 2024-03-04T07:39:10.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.871+0000 7faf5cc00700 1 -- 172.21.15.116:0/473410356 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7faf4c013070 con 0x7faf60074c10 2024-03-04T07:39:10.873 INFO:teuthology.orchestra.run.smithi116.stdout:107374182474 2024-03-04T07:39:10.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf440a1ea0 msgr2=0x7faf440a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:10.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf440a1ea0 0x7faf440a4360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf60101e70 tx=0x7faf50009260 comp rx=0 tx=0).stop 2024-03-04T07:39:10.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60074c10 msgr2=0x7faf601a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:10.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60074c10 0x7faf601a0c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faf4c000c00 tx=0x7faf4c00bab0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 shutdown_connections 2024-03-04T07:39:10.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf440a1ea0 0x7faf440a4360 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf6010eac0 0x7faf6019a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faf600742f0 0x7faf601a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 --2- 172.21.15.116:0/473410356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf60074c10 0x7faf601a0c30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:10.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 >> 172.21.15.116:0/473410356 conn(0x7faf600791f0 msgr2=0x7faf600ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:10.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.874+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 shutdown_connections 2024-03-04T07:39:10.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:10.875+0000 7faf66c00700 1 -- 172.21.15.116:0/473410356 wait complete. 2024-03-04T07:39:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:10 smithi152 ceph-mon[37154]: pgmap v353: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 765 B/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-03-04T07:39:11.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:11 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/473410356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T07:39:11.922 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:11 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/473410356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T07:39:11.935 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/473410356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T07:39:12.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:12 smithi133 ceph-mon[36505]: pgmap v354: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-03-04T07:39:12.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:12 smithi152 ceph-mon[37154]: pgmap v354: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-03-04T07:39:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:12 smithi116 ceph-mon[32150]: pgmap v354: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-03-04T07:39:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 -- 172.21.15.116:0/962918695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f937806a710 msgr2=0x7f937810ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:14.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 --2- 172.21.15.116:0/962918695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f937806a710 0x7f937810ef50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9370009fc0 tx=0x7f937000b330 comp rx=0 tx=0).stop 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 -- 172.21.15.116:0/962918695 shutdown_connections 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 --2- 172.21.15.116:0/962918695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f937806a710 0x7f937810ef50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 --2- 172.21.15.116:0/962918695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9378108110 0x7f93781084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 --2- 172.21.15.116:0/962918695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9378069d70 0x7f937806a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 -- 172.21.15.116:0/962918695 >> 172.21.15.116:0/962918695 conn(0x7f9378077ef0 msgr2=0x7f9378078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 -- 172.21.15.116:0/962918695 shutdown_connections 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.505+0000 7f937e800700 1 -- 172.21.15.116:0/962918695 wait complete. 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 Processor -- start 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 -- start start 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378069d70 0x7f9378101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f937806a710 0x7f93781015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9378108110 0x7f9378101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:14.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93781123b0 con 0x7f9378069d70 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9378112230 con 0x7f9378108110 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f937e800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9378112530 con 0x7f937806a710 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f9377400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378069d70 0x7f9378101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f9377400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378069d70 0x7f9378101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40350/0 (socket says 172.21.15.116:40350) 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f9377400700 1 -- 172.21.15.116:0/2206519289 learned_addr learned my addr 172.21.15.116:0/2206519289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f9376a00700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f937806a710 0x7f93781015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.507+0000 7f9377e00700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9378108110 0x7f9378101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f9377400700 1 -- 172.21.15.116:0/2206519289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f937806a710 msgr2=0x7f93781015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f9377400700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f937806a710 0x7f93781015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f9377400700 1 -- 172.21.15.116:0/2206519289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9378108110 msgr2=0x7f9378101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f9377400700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9378108110 0x7f9378101b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f9377400700 1 -- 172.21.15.116:0/2206519289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93781a6d10 con 0x7f9378069d70 2024-03-04T07:39:14.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f9376a00700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f937806a710 0x7f93781015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f9377400700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378069d70 0x7f9378101090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f936800e9a0 tx=0x7f936800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f936fe00700 1 -- 172.21.15.116:0/2206519289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936800ca10 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f936fe00700 1 -- 172.21.15.116:0/2206519289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f936800cbb0 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93781a6fa0 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f936fe00700 1 -- 172.21.15.116:0/2206519289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368018680 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.508+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93781a7500 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.510+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9378068d80 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.512+0000 7f936fe00700 1 -- 172.21.15.116:0/2206519289 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f9368010430 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.514+0000 7f936fe00700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f935c0a1f20 0x7f935c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.514+0000 7f936fe00700 1 -- 172.21.15.116:0/2206519289 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f93680ce080 con 0x7f9378069d70 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.514+0000 7f9376a00700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f935c0a1f20 0x7f935c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.515+0000 7f9376a00700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f935c0a1f20 0x7f935c0a43e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9364009750 tx=0x7f9364009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:14.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.520+0000 7f936fe00700 1 -- 172.21.15.116:0/2206519289 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f936809b310 con 0x7f9378069d70 2024-03-04T07:39:14.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.735+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f937802dad0 con 0x7f9378069d70 2024-03-04T07:39:14.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.735+0000 7f936fe00700 1 -- 172.21.15.116:0/2206519289 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f936809ac10 con 0x7f9378069d70 2024-03-04T07:39:14.736 INFO:teuthology.orchestra.run.smithi116.stdout:163208757312 2024-03-04T07:39:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f935c0a1f20 msgr2=0x7f935c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f935c0a1f20 0x7f935c0a43e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9364009750 tx=0x7f9364009310 comp rx=0 tx=0).stop 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378069d70 msgr2=0x7f9378101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378069d70 0x7f9378101090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f936800e9a0 tx=0x7f936800ecb0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 shutdown_connections 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378069d70 0x7f9378101090 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f935c0a1f20 0x7f935c0a43e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9378108110 0x7f9378101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 --2- 172.21.15.116:0/2206519289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f937806a710 0x7f93781015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.738+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 >> 172.21.15.116:0/2206519289 conn(0x7f9378077ef0 msgr2=0x7f93780ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.739+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 shutdown_connections 2024-03-04T07:39:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:14.739+0000 7f937e800700 1 -- 172.21.15.116:0/2206519289 wait complete. 2024-03-04T07:39:14.774 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:14 smithi116 ceph-mon[32150]: pgmap v355: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T07:39:14.774 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1195821891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-04T07:39:14.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182469 got 107374182474 for osd.3 2024-03-04T07:39:14.778 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:14 smithi152 ceph-mon[37154]: pgmap v355: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T07:39:15.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:14 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1195821891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-04T07:39:15.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:14 smithi133 ceph-mon[36505]: pgmap v355: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T07:39:15.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:14 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1195821891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-04T07:39:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 -- 172.21.15.116:0/3308055140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901086d0 msgr2=0x7fd490108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 --2- 172.21.15.116:0/3308055140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901086d0 0x7fd490108b50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0098f0 tx=0x7fd47c009c00 comp rx=0 tx=0).stop 2024-03-04T07:39:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 -- 172.21.15.116:0/3308055140 shutdown_connections 2024-03-04T07:39:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 --2- 172.21.15.116:0/3308055140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd490109090 0x7fd4901155a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 --2- 172.21.15.116:0/3308055140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd49010e6d0 0x7fd49010eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 --2- 172.21.15.116:0/3308055140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4901086d0 0x7fd490108b50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 -- 172.21.15.116:0/3308055140 >> 172.21.15.116:0/3308055140 conn(0x7fd490075890 msgr2=0x7fd490077cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 -- 172.21.15.116:0/3308055140 shutdown_connections 2024-03-04T07:39:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.348+0000 7fd497000700 1 -- 172.21.15.116:0/3308055140 wait complete. 2024-03-04T07:39:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.349+0000 7fd497000700 1 Processor -- start 2024-03-04T07:39:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.349+0000 7fd497000700 1 -- start start 2024-03-04T07:39:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.349+0000 7fd497000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4901086d0 0x7fd49019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.349+0000 7fd497000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd490109090 0x7fd49019f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.349+0000 7fd497000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd49010e6d0 0x7fd4901a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.349+0000 7fd497000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd490118a00 con 0x7fd490109090 2024-03-04T07:39:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.349+0000 7fd497000700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd490118880 con 0x7fd4901086d0 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd497000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd490118b80 con 0x7fd49010e6d0 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4901086d0 0x7fd49019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4901086d0 0x7fd49019ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37282/0 (socket says 172.21.15.116:37282) 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48f400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd490109090 0x7fd49019f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 -- 172.21.15.116:0/892399786 learned_addr learned my addr 172.21.15.116:0/892399786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd495000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd49010e6d0 0x7fd4901a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 -- 172.21.15.116:0/892399786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd49010e6d0 msgr2=0x7fd4901a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd49010e6d0 0x7fd4901a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 -- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd490109090 msgr2=0x7fd49019f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd490109090 0x7fd49019f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 -- 172.21.15.116:0/892399786 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4901a3e80 con 0x7fd4901086d0 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd495000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd49010e6d0 0x7fd4901a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:39:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.350+0000 7fd48fe00700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4901086d0 0x7fd49019ee70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd48000eb30 tx=0x7fd48000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.351+0000 7fd48cc00700 1 -- 172.21.15.116:0/892399786 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48000cc90 con 0x7fd4901086d0 2024-03-04T07:39:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.351+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4901a4170 con 0x7fd4901086d0 2024-03-04T07:39:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.351+0000 7fd48cc00700 1 -- 172.21.15.116:0/892399786 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd480004500 con 0x7fd4901086d0 2024-03-04T07:39:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.351+0000 7fd48cc00700 1 -- 172.21.15.116:0/892399786 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480010430 con 0x7fd4901086d0 2024-03-04T07:39:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.351+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4901ab9f0 con 0x7fd4901086d0 2024-03-04T07:39:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.352+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd490068d80 con 0x7fd4901086d0 2024-03-04T07:39:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.354+0000 7fd48cc00700 1 -- 172.21.15.116:0/892399786 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fd480003f70 con 0x7fd4901086d0 2024-03-04T07:39:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.355+0000 7fd48cc00700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4740a1ea0 0x7fd4740a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.355+0000 7fd48cc00700 1 -- 172.21.15.116:0/892399786 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd4800ce230 con 0x7fd4901086d0 2024-03-04T07:39:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.355+0000 7fd48f400700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4740a1ea0 0x7fd4740a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.356+0000 7fd48f400700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4740a1ea0 0x7fd4740a4360 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0098c0 tx=0x7fd47c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:15.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.359+0000 7fd48cc00700 1 -- 172.21.15.116:0/892399786 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd48009b4c0 con 0x7fd4901086d0 2024-03-04T07:39:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.722+0000 7fb50f600700 1 -- 172.21.15.116:0/18626376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb508109090 msgr2=0x7fb5081155a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.722+0000 7fb50f600700 1 --2- 172.21.15.116:0/18626376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb508109090 0x7fb5081155a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb50000b0a0 tx=0x7fb50000b3b0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.723+0000 7fb50f600700 1 -- 172.21.15.116:0/18626376 shutdown_connections 2024-03-04T07:39:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.723+0000 7fb50f600700 1 --2- 172.21.15.116:0/18626376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb50810e6d0 0x7fb50810eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.723+0000 7fb50f600700 1 --2- 172.21.15.116:0/18626376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb508109090 0x7fb5081155a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.723+0000 7fb50f600700 1 --2- 172.21.15.116:0/18626376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 0x7fb508108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.723+0000 7fb50f600700 1 -- 172.21.15.116:0/18626376 >> 172.21.15.116:0/18626376 conn(0x7fb508075890 msgr2=0x7fb508077cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.723+0000 7fb50f600700 1 -- 172.21.15.116:0/18626376 shutdown_connections 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.723+0000 7fb50f600700 1 -- 172.21.15.116:0/18626376 wait complete. 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 Processor -- start 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 -- start start 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 0x7fb50819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb508109090 0x7fb50819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb50810e6d0 0x7fb5081a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb508118ab0 con 0x7fb508109090 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb508118930 con 0x7fb5081086d0 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.724+0000 7fb50f600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb508118c30 con 0x7fb50810e6d0 2024-03-04T07:39:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 0x7fb50819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 0x7fb50819ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37240/0 (socket says 172.21.15.116:37240) 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 -- 172.21.15.116:0/1195821891 learned_addr learned my addr 172.21.15.116:0/1195821891 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb507e00700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb508109090 0x7fb50819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50d600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb50810e6d0 0x7fb5081a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 -- 172.21.15.116:0/1195821891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb50810e6d0 msgr2=0x7fb5081a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb50810e6d0 0x7fb5081a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 -- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb508109090 msgr2=0x7fb50819f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb508109090 0x7fb50819f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 -- 172.21.15.116:0/1195821891 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5081a3e00 con 0x7fb5081086d0 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50d600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb50810e6d0 0x7fb5081a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.725+0000 7fb50cc00700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 0x7fb50819ee40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00e9a0 tx=0x7fb4fc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.726+0000 7fb505600700 1 -- 172.21.15.116:0/1195821891 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc00cbd0 con 0x7fb5081086d0 2024-03-04T07:39:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.726+0000 7fb505600700 1 -- 172.21.15.116:0/1195821891 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4fc00cd70 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.726+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5081a40f0 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.726+0000 7fb505600700 1 -- 172.21.15.116:0/1195821891 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc018750 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.726+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5081ab970 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.727+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb508068d80 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.729+0000 7fb505600700 1 -- 172.21.15.116:0/1195821891 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fb4fc010430 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.730+0000 7fb505600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4f00a1f20 0x7fb4f00a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.730+0000 7fb505600700 1 -- 172.21.15.116:0/1195821891 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb4fc0ce170 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.730+0000 7fb507e00700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4f00a1f20 0x7fb4f00a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.731+0000 7fb507e00700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4f00a1f20 0x7fb4f00a43e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb50810db90 tx=0x7fb4f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.734+0000 7fb505600700 1 -- 172.21.15.116:0/1195821891 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb4fc09b400 con 0x7fb5081086d0 2024-03-04T07:39:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.957+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fb50802d9b0 con 0x7fb5081086d0 2024-03-04T07:39:15.419 INFO:teuthology.orchestra.run.smithi116.stdout:309237645347 2024-03-04T07:39:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.957+0000 7fb505600700 1 -- 172.21.15.116:0/1195821891 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb4fc09ad00 con 0x7fb5081086d0 2024-03-04T07:39:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4f00a1f20 msgr2=0x7fb4f00a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4f00a1f20 0x7fb4f00a43e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb50810db90 tx=0x7fb4f4009260 comp rx=0 tx=0).stop 2024-03-04T07:39:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 msgr2=0x7fb50819ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 0x7fb50819ee40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc00e9a0 tx=0x7fb4fc00ecb0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 shutdown_connections 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb4f00a1f20 0x7fb4f00a43e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb50810e6d0 0x7fb5081a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5081086d0 0x7fb50819ee40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 --2- 172.21.15.116:0/1195821891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb508109090 0x7fb50819f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 >> 172.21.15.116:0/1195821891 conn(0x7fb508075890 msgr2=0x7fb508075db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.961+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 shutdown_connections 2024-03-04T07:39:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:13.962+0000 7fb50f600700 1 -- 172.21.15.116:0/1195821891 wait complete. 2024-03-04T07:39:15.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.582+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd4901a01d0 con 0x7fd4901086d0 2024-03-04T07:39:15.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.583+0000 7fd48cc00700 1 -- 172.21.15.116:0/892399786 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd48009adc0 con 0x7fd4901086d0 2024-03-04T07:39:15.584 INFO:teuthology.orchestra.run.smithi116.stdout:77309411409 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.585+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4740a1ea0 msgr2=0x7fd4740a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4740a1ea0 0x7fd4740a4360 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd47c0098c0 tx=0x7fd47c019040 comp rx=0 tx=0).stop 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4901086d0 msgr2=0x7fd49019ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4901086d0 0x7fd49019ee70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd48000eb30 tx=0x7fd48000ee40 comp rx=0 tx=0).stop 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 shutdown_connections 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4740a1ea0 0x7fd4740a4360 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd49010e6d0 0x7fd4901a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4901086d0 0x7fd49019ee70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 --2- 172.21.15.116:0/892399786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd490109090 0x7fd49019f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:15.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 >> 172.21.15.116:0/892399786 conn(0x7fd490075890 msgr2=0x7fd490075ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:15.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 shutdown_connections 2024-03-04T07:39:15.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:15.586+0000 7fd497000700 1 -- 172.21.15.116:0/892399786 wait complete. 2024-03-04T07:39:15.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2206519289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T07:39:15.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/892399786' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T07:39:16.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2206519289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T07:39:16.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/892399786' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T07:39:16.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2206519289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T07:39:16.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/892399786' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T07:39:16.360 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738457 for osd.0 2024-03-04T07:39:16.360 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:16.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 -- 172.21.15.116:0/1421187059 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2034074db0 msgr2=0x7f203410e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:16.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 --2- 172.21.15.116:0/1421187059 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2034074db0 0x7f203410e400 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f201c009990 tx=0x7f201c009ca0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 -- 172.21.15.116:0/1421187059 shutdown_connections 2024-03-04T07:39:16.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 --2- 172.21.15.116:0/1421187059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f203410e940 0x7f2034110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 --2- 172.21.15.116:0/1421187059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2034074490 0x7f2034074870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 --2- 172.21.15.116:0/1421187059 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2034074db0 0x7f203410e400 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 -- 172.21.15.116:0/1421187059 >> 172.21.15.116:0/1421187059 conn(0x7f20340fdb70 msgr2=0x7f20340fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:16.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 -- 172.21.15.116:0/1421187059 shutdown_connections 2024-03-04T07:39:16.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.707+0000 7f2039e00700 1 -- 172.21.15.116:0/1421187059 wait complete. 2024-03-04T07:39:16.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.708+0000 7f2039e00700 1 Processor -- start 2024-03-04T07:39:16.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.708+0000 7f2039e00700 1 -- start start 2024-03-04T07:39:16.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2039e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2034074490 0x7f20341a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:16.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2039e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2034074db0 0x7f20341a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:16.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2039e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f203410e940 0x7f203419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2039e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20341145f0 con 0x7f2034074490 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2039e00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034114470 con 0x7f203410e940 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2039e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034114770 con 0x7f2034074db0 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2032a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2034074490 0x7f20341a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2033400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f203410e940 0x7f203419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2032a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2034074490 0x7f20341a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51586/0 (socket says 172.21.15.116:51586) 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2032a00700 1 -- 172.21.15.116:0/2847282946 learned_addr learned my addr 172.21.15.116:0/2847282946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f202a000700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2034074db0 0x7f20341a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2033400700 1 -- 172.21.15.116:0/2847282946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2034074db0 msgr2=0x7f20341a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2033400700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2034074db0 0x7f20341a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2033400700 1 -- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2034074490 msgr2=0x7f20341a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2033400700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2034074490 0x7f20341a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.709+0000 7f2033400700 1 -- 172.21.15.116:0/2847282946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203419b070 con 0x7f203410e940 2024-03-04T07:39:16.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.710+0000 7f2033400700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f203410e940 0x7f203419a910 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f202c00ef90 tx=0x7f202c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:16.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.710+0000 7f2038a00700 1 -- 172.21.15.116:0/2847282946 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0092e0 con 0x7f203410e940 2024-03-04T07:39:16.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.710+0000 7f2038a00700 1 -- 172.21.15.116:0/2847282946 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f202c004bb0 con 0x7f203410e940 2024-03-04T07:39:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.710+0000 7f2038a00700 1 -- 172.21.15.116:0/2847282946 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c004280 con 0x7f203410e940 2024-03-04T07:39:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.710+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203419b300 con 0x7f203410e940 2024-03-04T07:39:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.710+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20341a75d0 con 0x7f203410e940 2024-03-04T07:39:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.711+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2034068d80 con 0x7f203410e940 2024-03-04T07:39:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.713+0000 7f2038a00700 1 -- 172.21.15.116:0/2847282946 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f202c01b030 con 0x7f203410e940 2024-03-04T07:39:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.714+0000 7f2038a00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f20180a1f20 0x7f20180a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.714+0000 7f2038a00700 1 -- 172.21.15.116:0/2847282946 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f202c0cda80 con 0x7f203410e940 2024-03-04T07:39:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.714+0000 7f2032a00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f20180a1f20 0x7f20180a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.715+0000 7f2032a00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f20180a1f20 0x7f20180a43e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2034111100 tx=0x7f2024009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.718+0000 7f2038a00700 1 -- 172.21.15.116:0/2847282946 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f202c0cfe20 con 0x7f203410e940 2024-03-04T07:39:16.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:16 smithi116 ceph-mon[32150]: pgmap v356: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:39:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.946+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f203402d9b0 con 0x7f203410e940 2024-03-04T07:39:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.947+0000 7f2038a00700 1 -- 172.21.15.116:0/2847282946 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f202c022020 con 0x7f203410e940 2024-03-04T07:39:16.948 INFO:teuthology.orchestra.run.smithi116.stdout:55834574934 2024-03-04T07:39:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f20180a1f20 msgr2=0x7f20180a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f20180a1f20 0x7f20180a43e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2034111100 tx=0x7f2024009260 comp rx=0 tx=0).stop 2024-03-04T07:39:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f203410e940 msgr2=0x7f203419a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f203410e940 0x7f203419a910 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f202c00ef90 tx=0x7f202c00c3a0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 shutdown_connections 2024-03-04T07:39:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2034074490 0x7f20341a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f20180a1f20 0x7f20180a43e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f203410e940 0x7f203419a910 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 --2- 172.21.15.116:0/2847282946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2034074db0 0x7f20341a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 >> 172.21.15.116:0/2847282946 conn(0x7f20340fdb70 msgr2=0x7f20340fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 shutdown_connections 2024-03-04T07:39:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:16.950+0000 7f2039e00700 1 -- 172.21.15.116:0/2847282946 wait complete. 2024-03-04T07:39:17.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:16 smithi152 ceph-mon[37154]: pgmap v356: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:39:17.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:16 smithi133 ceph-mon[36505]: pgmap v356: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:39:17.926 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:17 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2847282946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T07:39:18.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:17 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2847282946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T07:39:18.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:17 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2847282946' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T07:39:18.394 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808871 for osd.9 2024-03-04T07:39:18.395 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:18.840 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:18 smithi116 ceph-mon[32150]: pgmap v357: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:18.987 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:18 smithi133 ceph-mon[36505]: pgmap v357: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:18 smithi152 ceph-mon[37154]: pgmap v357: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:19.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.599+0000 7f63df200700 1 -- 172.21.15.116:0/2675475568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 msgr2=0x7f63d806a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:19.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.599+0000 7f63df200700 1 --2- 172.21.15.116:0/2675475568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 0x7f63d806a9e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f63c8009990 tx=0x7f63c8009ca0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.599+0000 7f63df200700 1 -- 172.21.15.116:0/2675475568 shutdown_connections 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.599+0000 7f63df200700 1 --2- 172.21.15.116:0/2675475568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 0x7f63d806a9e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.599+0000 7f63df200700 1 --2- 172.21.15.116:0/2675475568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 0x7f63d8110f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.599+0000 7f63df200700 1 --2- 172.21.15.116:0/2675475568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63d806af20 0x7f63d810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.599+0000 7f63df200700 1 -- 172.21.15.116:0/2675475568 >> 172.21.15.116:0/2675475568 conn(0x7f63d8077010 msgr2=0x7f63d8077420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.600+0000 7f63df200700 1 -- 172.21.15.116:0/2675475568 shutdown_connections 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.600+0000 7f63df200700 1 -- 172.21.15.116:0/2675475568 wait complete. 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 Processor -- start 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 -- start start 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 0x7f63d81a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63d806af20 0x7f63d81a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 0x7f63d819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d81145c0 con 0x7f63d806a600 2024-03-04T07:39:19.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d8114440 con 0x7f63d810eba0 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63df200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d8114740 con 0x7f63d806af20 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63d7e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 0x7f63d81a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63dd200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 0x7f63d819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63dd200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 0x7f63d819a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44800/0 (socket says 172.21.15.116:44800) 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63dd200700 1 -- 172.21.15.116:0/2779578880 learned_addr learned my addr 172.21.15.116:0/2779578880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.601+0000 7f63d7400700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63d806af20 0x7f63d81a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63dd200700 1 -- 172.21.15.116:0/2779578880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63d806af20 msgr2=0x7f63d81a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63dd200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63d806af20 0x7f63d81a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63dd200700 1 -- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 msgr2=0x7f63d81a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63dd200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 0x7f63d81a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63dd200700 1 -- 172.21.15.116:0/2779578880 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d819b060 con 0x7f63d810eba0 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63d7e00700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 0x7f63d81a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:19.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63dd200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 0x7f63d819a900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c370 tx=0x7f63d000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63d4c00700 1 -- 172.21.15.116:0/2779578880 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00092e0 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63d4c00700 1 -- 172.21.15.116:0/2779578880 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63d0003a00 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.602+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d819b2f0 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.603+0000 7f63d4c00700 1 -- 172.21.15.116:0/2779578880 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00047c0 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.603+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d81a7610 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.604+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d8068d80 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.606+0000 7f63d4c00700 1 -- 172.21.15.116:0/2779578880 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f63d001a020 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.608+0000 7f63d4c00700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63bc0a1ea0 0x7f63bc0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.608+0000 7f63d4c00700 1 -- 172.21.15.116:0/2779578880 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f63d00cd800 con 0x7f63d810eba0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.608+0000 7f63d7e00700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63bc0a1ea0 0x7f63bc0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.608+0000 7f63d7e00700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63bc0a1ea0 0x7f63bc0a4360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f63d819ba40 tx=0x7f63c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:19.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.612+0000 7f63d4c00700 1 -- 172.21.15.116:0/2779578880 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f63d0010040 con 0x7f63d810eba0 2024-03-04T07:39:19.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.838+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f63d802d9b0 con 0x7f63d810eba0 2024-03-04T07:39:19.834 INFO:teuthology.orchestra.run.smithi116.stdout:253403070511 2024-03-04T07:39:19.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.839+0000 7f63d4c00700 1 -- 172.21.15.116:0/2779578880 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f63d009aa90 con 0x7f63d810eba0 2024-03-04T07:39:19.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63bc0a1ea0 msgr2=0x7f63bc0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:19.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63bc0a1ea0 0x7f63bc0a4360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f63d819ba40 tx=0x7f63c8005fd0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 msgr2=0x7f63d819a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 0x7f63d819a900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c370 tx=0x7f63d000c680 comp rx=0 tx=0).stop 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 shutdown_connections 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63d806a600 0x7f63d81a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63bc0a1ea0 0x7f63bc0a4360 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63d810eba0 0x7f63d819a900 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 --2- 172.21.15.116:0/2779578880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63d806af20 0x7f63d81a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 >> 172.21.15.116:0/2779578880 conn(0x7f63d8077010 msgr2=0x7f63d8110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 shutdown_connections 2024-03-04T07:39:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:18.843+0000 7f63df200700 1 -- 172.21.15.116:0/2779578880 wait complete. 2024-03-04T07:39:20.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:19 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2779578880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-04T07:39:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:19 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2779578880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-04T07:39:20.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:19 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2779578880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-04T07:39:20.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:20 smithi116 ceph-mon[32150]: pgmap v358: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:20 smithi152 ceph-mon[37154]: pgmap v358: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:21.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:20 smithi133 ceph-mon[36505]: pgmap v358: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:21.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645340 got 309237645347 for osd.10 2024-03-04T07:39:21.413 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:21.760 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574934 for osd.1 2024-03-04T07:39:21.760 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:21.992 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757312 for osd.5 2024-03-04T07:39:21.993 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:22.605 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411404 got 77309411409 for osd.2 2024-03-04T07:39:22.606 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:22.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:22 smithi116 ceph-mon[32150]: pgmap v359: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:39:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:22 smithi152 ceph-mon[37154]: pgmap v359: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:39:23.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:22 smithi133 ceph-mon[36505]: pgmap v359: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:39:23.821 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070505 got 253403070511 for osd.8 2024-03-04T07:39:23.822 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.567+0000 7fa0d4800700 1 -- 172.21.15.116:0/1469312483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc109090 msgr2=0x7fa0cc1155a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.567+0000 7fa0d4800700 1 --2- 172.21.15.116:0/1469312483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc109090 0x7fa0cc1155a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009f90 tx=0x7fa0c800b330 comp rx=0 tx=0).stop 2024-03-04T07:39:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.568+0000 7fa0d4800700 1 -- 172.21.15.116:0/1469312483 shutdown_connections 2024-03-04T07:39:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.568+0000 7fa0d4800700 1 --2- 172.21.15.116:0/1469312483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc109090 0x7fa0cc1155a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.568+0000 7fa0d4800700 1 --2- 172.21.15.116:0/1469312483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 0x7fa0cc10eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.568+0000 7fa0d4800700 1 --2- 172.21.15.116:0/1469312483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc1086d0 0x7fa0cc108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.568+0000 7fa0d4800700 1 -- 172.21.15.116:0/1469312483 >> 172.21.15.116:0/1469312483 conn(0x7fa0cc075890 msgr2=0x7fa0cc077cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.568+0000 7fa0d4800700 1 -- 172.21.15.116:0/1469312483 shutdown_connections 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.568+0000 7fa0d4800700 1 -- 172.21.15.116:0/1469312483 wait complete. 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 Processor -- start 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 -- start start 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1086d0 0x7fa0cc19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc109090 0x7fa0cc19f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 0x7fa0cc1a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc118a00 con 0x7fa0cc1086d0 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc118880 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d4800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc118b80 con 0x7fa0cc109090 2024-03-04T07:39:24.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1086d0 0x7fa0cc19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d2800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 0x7fa0cc1a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d1e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1086d0 0x7fa0cc19ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51614/0 (socket says 172.21.15.116:51614) 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d2800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 0x7fa0cc1a3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44828/0 (socket says 172.21.15.116:44828) 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.569+0000 7fa0d1e00700 1 -- 172.21.15.116:0/3363702550 learned_addr learned my addr 172.21.15.116:0/3363702550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d1400700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc109090 0x7fa0cc19f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d2800700 1 -- 172.21.15.116:0/3363702550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc109090 msgr2=0x7fa0cc19f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d2800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc109090 0x7fa0cc19f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d2800700 1 -- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1086d0 msgr2=0x7fa0cc19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d2800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1086d0 0x7fa0cc19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d2800700 1 -- 172.21.15.116:0/3363702550 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc1a3da0 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d2800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 0x7fa0cc1a3660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8015000 tx=0x7fa0c8004f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:24.044 INFO:teuthology.orchestra.run.smithi116.stdout:197568495675 2024-03-04T07:39:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0c6a00700 1 -- 172.21.15.116:0/3363702550 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8021070 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0c6a00700 1 -- 172.21.15.116:0/3363702550 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0c8006020 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.570+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc1a4030 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.571+0000 7fa0c6a00700 1 -- 172.21.15.116:0/3363702550 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c80052a0 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.571+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc1ab910 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.572+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc109e10 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.574+0000 7fa0c6a00700 1 -- 172.21.15.116:0/3363702550 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7fa0c8024070 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.575+0000 7fa0c6a00700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0b40a1ea0 0x7fa0b40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.575+0000 7fa0c6a00700 1 -- 172.21.15.116:0/3363702550 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa0c80cef20 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.575+0000 7fa0d1e00700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0b40a1ea0 0x7fa0b40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.576+0000 7fa0d1e00700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0b40a1ea0 0x7fa0b40a4360 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0005950 tx=0x7fa0c0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.577+0000 7fa0c6a00700 1 -- 172.21.15.116:0/3363702550 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa0c809c280 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.810+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa0cc1abae0 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.810+0000 7fa0c6a00700 1 -- 172.21.15.116:0/3363702550 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa0c809bb80 con 0x7fa0cc10e6d0 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0b40a1ea0 msgr2=0x7fa0b40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0b40a1ea0 0x7fa0b40a4360 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0005950 tx=0x7fa0c0009310 comp rx=0 tx=0).stop 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 msgr2=0x7fa0cc1a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 0x7fa0cc1a3660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8015000 tx=0x7fa0c8004f00 comp rx=0 tx=0).stop 2024-03-04T07:39:24.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 shutdown_connections 2024-03-04T07:39:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1086d0 0x7fa0cc19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa0b40a1ea0 0x7fa0b40a4360 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0cc10e6d0 0x7fa0cc1a3660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 --2- 172.21.15.116:0/3363702550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0cc109090 0x7fa0cc19f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 >> 172.21.15.116:0/3363702550 conn(0x7fa0cc075890 msgr2=0x7fa0cc075d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 shutdown_connections 2024-03-04T07:39:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:23.815+0000 7fa0d4800700 1 -- 172.21.15.116:0/3363702550 wait complete. 2024-03-04T07:39:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 -- 172.21.15.116:0/3071546953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc10eac0 msgr2=0x7f02fc110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 --2- 172.21.15.116:0/3071546953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc10eac0 0x7f02fc110eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f02f400b0a0 tx=0x7f02f400b3b0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 -- 172.21.15.116:0/3071546953 shutdown_connections 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 --2- 172.21.15.116:0/3071546953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc10eac0 0x7f02fc110eb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 --2- 172.21.15.116:0/3071546953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc0742f0 0x7f02fc0746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 --2- 172.21.15.116:0/3071546953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc074c10 0x7f02fc10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 -- 172.21.15.116:0/3071546953 >> 172.21.15.116:0/3071546953 conn(0x7f02fc0791f0 msgr2=0x7f02fc079600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 -- 172.21.15.116:0/3071546953 shutdown_connections 2024-03-04T07:39:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.698+0000 7f0304200700 1 -- 172.21.15.116:0/3071546953 wait complete. 2024-03-04T07:39:24.700 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:24 smithi116 ceph-mon[32150]: pgmap v360: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:24.701 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:24 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3363702550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T07:39:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 Processor -- start 2024-03-04T07:39:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 -- start start 2024-03-04T07:39:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc0742f0 0x7f02fc1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc074c10 0x7f02fc1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc10eac0 0x7f02fc19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc1145d0 con 0x7f02fc074c10 2024-03-04T07:39:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc114450 con 0x7f02fc10eac0 2024-03-04T07:39:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.700+0000 7f0304200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc114750 con 0x7f02fc0742f0 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc10eac0 0x7f02fc19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc10eac0 0x7f02fc19a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44832/0 (socket says 172.21.15.116:44832) 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 -- 172.21.15.116:0/3848122371 learned_addr learned my addr 172.21.15.116:0/3848122371 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0300e00700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc074c10 0x7f02fc1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0301800700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc0742f0 0x7f02fc1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 -- 172.21.15.116:0/3848122371 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc0742f0 msgr2=0x7f02fc1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc0742f0 0x7f02fc1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 -- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc074c10 msgr2=0x7f02fc1a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc074c10 0x7f02fc1a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0302200700 1 -- 172.21.15.116:0/3848122371 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc19b050 con 0x7f02fc10eac0 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.701+0000 7f0301800700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc0742f0 0x7f02fc1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.702+0000 7f0302200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc10eac0 0x7f02fc19a870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f02f400b070 tx=0x7f02f4004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.702+0000 7f02fa000700 1 -- 172.21.15.116:0/3848122371 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4021070 con 0x7f02fc10eac0 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.702+0000 7f02fa000700 1 -- 172.21.15.116:0/3848122371 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f4003c00 con 0x7f02fc10eac0 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.702+0000 7f02fa000700 1 -- 172.21.15.116:0/3848122371 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4004740 con 0x7f02fc10eac0 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.702+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc19b2e0 con 0x7f02fc10eac0 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.702+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc1a7500 con 0x7f02fc10eac0 2024-03-04T07:39:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.703+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc075990 con 0x7f02fc10eac0 2024-03-04T07:39:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.705+0000 7f02fa000700 1 -- 172.21.15.116:0/3848122371 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 136808+0+0 (secure 0 0 0) 0x7f02f4007980 con 0x7f02fc10eac0 2024-03-04T07:39:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.706+0000 7f02fa000700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e40a1f20 0x7f02e40a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.706+0000 7f02fa000700 1 -- 172.21.15.116:0/3848122371 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f02f40ced30 con 0x7f02fc10eac0 2024-03-04T07:39:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.706+0000 7f0301800700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e40a1f20 0x7f02e40a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.707+0000 7f0301800700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e40a1f20 0x7f02e40a43e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f02f0011490 tx=0x7f02f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.710+0000 7f02fa000700 1 -- 172.21.15.116:0/3848122371 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f02f409c050 con 0x7f02fc10eac0 2024-03-04T07:39:24.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.935+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f02fc1a76d0 con 0x7f02fc10eac0 2024-03-04T07:39:24.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.936+0000 7f02fa000700 1 -- 172.21.15.116:0/3848122371 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f02f409b950 con 0x7f02fc10eac0 2024-03-04T07:39:24.937 INFO:teuthology.orchestra.run.smithi116.stdout:343597383712 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.938+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e40a1f20 msgr2=0x7f02e40a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.938+0000 7f0304200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e40a1f20 0x7f02e40a43e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f02f0011490 tx=0x7f02f0009260 comp rx=0 tx=0).stop 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.938+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc10eac0 msgr2=0x7f02fc19a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.938+0000 7f0304200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc10eac0 0x7f02fc19a870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f02f400b070 tx=0x7f02f4004c60 comp rx=0 tx=0).stop 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 shutdown_connections 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e40a1f20 0x7f02e40a43e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f02fc10eac0 0x7f02fc19a870 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02fc0742f0 0x7f02fc1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 --2- 172.21.15.116:0/3848122371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc074c10 0x7f02fc1a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 >> 172.21.15.116:0/3848122371 conn(0x7f02fc0791f0 msgr2=0x7f02fc0ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 shutdown_connections 2024-03-04T07:39:24.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:24.939+0000 7f0304200700 1 -- 172.21.15.116:0/3848122371 wait complete. 2024-03-04T07:39:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:24 smithi152 ceph-mon[37154]: pgmap v360: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:25.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:24 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3363702550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T07:39:25.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:24 smithi133 ceph-mon[36505]: pgmap v360: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:24 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3363702550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T07:39:25.247 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495675 for osd.6 2024-03-04T07:39:25.247 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:26.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:25 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3848122371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-04T07:39:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:25 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3848122371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-04T07:39:26.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:25 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3848122371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-04T07:39:26.123 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383705 got 343597383712 for osd.11 2024-03-04T07:39:26.124 DEBUG:teuthology.parallel:result is None 2024-03-04T07:39:26.124 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T07:39:26.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T07:39:26.567 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:26.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:26 smithi116 ceph-mon[32150]: pgmap v361: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:26.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:26 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:26.954 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:26 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:26 smithi152 ceph-mon[37154]: pgmap v361: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:26 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:26 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:27.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:26 smithi133 ceph-mon[36505]: pgmap v361: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:26 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:26 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 -- 172.21.15.116:0/583958702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c074c10 msgr2=0x7f131c10e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 --2- 172.21.15.116:0/583958702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c074c10 0x7f131c10e580 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1308009990 tx=0x7f1308009ca0 comp rx=0 tx=0).stop 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 -- 172.21.15.116:0/583958702 shutdown_connections 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 --2- 172.21.15.116:0/583958702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131c10eac0 0x7f131c110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 --2- 172.21.15.116:0/583958702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f131c0742f0 0x7f131c0746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 --2- 172.21.15.116:0/583958702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c074c10 0x7f131c10e580 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 -- 172.21.15.116:0/583958702 >> 172.21.15.116:0/583958702 conn(0x7f131c0791f0 msgr2=0x7f131c079600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 -- 172.21.15.116:0/583958702 shutdown_connections 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.908+0000 7f1321800700 1 -- 172.21.15.116:0/583958702 wait complete. 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 Processor -- start 2024-03-04T07:39:28.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 -- start start 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c0742f0 0x7f131c1a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131c074c10 0x7f131c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f131c10eac0 0x7f131c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c114610 con 0x7f131c074c10 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c114490 con 0x7f131c0742f0 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.910+0000 7f1321800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c114790 con 0x7f131c10eac0 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c0742f0 0x7f131c1a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131a000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131c074c10 0x7f131c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c0742f0 0x7f131c1a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:33742/0 (socket says 172.21.15.116:33742) 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 -- 172.21.15.116:0/2080041618 learned_addr learned my addr 172.21.15.116:0/2080041618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131b400700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f131c10eac0 0x7f131c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 -- 172.21.15.116:0/2080041618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f131c10eac0 msgr2=0x7f131c19a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:28.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f131c10eac0 0x7f131c19a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 -- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131c074c10 msgr2=0x7f131c1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131c074c10 0x7f131c1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 -- 172.21.15.116:0/2080041618 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c19b030 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131a000700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131c074c10 0x7f131c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131b400700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f131c10eac0 0x7f131c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f131aa00700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c0742f0 0x7f131c1a0710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f130c00d980 tx=0x7f130c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.911+0000 7f1313400700 1 -- 172.21.15.116:0/2080041618 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c016070 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.912+0000 7f1313400700 1 -- 172.21.15.116:0/2080041618 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f130c004d10 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.912+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c19b320 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.912+0000 7f1313400700 1 -- 172.21.15.116:0/2080041618 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c0055f0 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.912+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c1a75e0 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.913+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c068d80 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.914+0000 7f1313400700 1 -- 172.21.15.116:0/2080041618 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f130c004750 con 0x7f131c0742f0 2024-03-04T07:39:28.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.916+0000 7f1313400700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13000a1e90 0x7f13000a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:28.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.916+0000 7f1313400700 1 -- 172.21.15.116:0/2080041618 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f130c0cdbc0 con 0x7f131c0742f0 2024-03-04T07:39:28.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.916+0000 7f131a000700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13000a1e90 0x7f13000a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:28.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.917+0000 7f131a000700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13000a1e90 0x7f13000a4350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f131c1012c0 tx=0x7f130800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:28.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:28.920+0000 7f1313400700 1 -- 172.21.15.116:0/2080041618 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f130c01f020 con 0x7f131c0742f0 2024-03-04T07:39:28.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:28 smithi116 ceph-mon[32150]: pgmap v362: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:28.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:39:28.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:39:28.940 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:28 smithi152 ceph-mon[37154]: pgmap v362: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:39:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:39:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:29.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:28 smithi133 ceph-mon[36505]: pgmap v362: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:29.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:39:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:39:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:39:29.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.136+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f131c100a90 con 0x7f13000a1e90 2024-03-04T07:39:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.145+0000 7f1313400700 1 -- 172.21.15.116:0/2080041618 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258960 (secure 0 0 0) 0x7f131c100a90 con 0x7f13000a1e90 2024-03-04T07:39:29.147 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.152+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13000a1e90 msgr2=0x7f13000a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.152+0000 7f1321800700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13000a1e90 0x7f13000a4350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f131c1012c0 tx=0x7f130800b260 comp rx=0 tx=0).stop 2024-03-04T07:39:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.152+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c0742f0 msgr2=0x7f131c1a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:29.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.152+0000 7f1321800700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c0742f0 0x7f131c1a0710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f130c00d980 tx=0x7f130c00dc90 comp rx=0 tx=0).stop 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 shutdown_connections 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13000a1e90 0x7f13000a4350 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f131c10eac0 0x7f131c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f131c0742f0 0x7f131c1a0710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 --2- 172.21.15.116:0/2080041618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131c074c10 0x7f131c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 >> 172.21.15.116:0/2080041618 conn(0x7f131c0791f0 msgr2=0x7f131c100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 shutdown_connections 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:29.153+0000 7f1321800700 1 -- 172.21.15.116:0/2080041618 wait complete. 2024-03-04T07:39:29.155 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T07:39:29.841 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:29 smithi116 ceph-mon[32150]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-04T07:39:30.006 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-03-04T07:39:27.510209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652446,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":318,"num_read_kb":266,"num_write":258,"num_write_kb":3023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360644,"kb_used_data":11412,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581180,"statfs":{"total":1151940427776,"available":1151571128320,"internally_reserved":0,"allocated":11685888,"data_stored":7426050,"data_compressed":182424,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005358"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808545+0000","last_change":"2024-03-04T07:37:55.137540+0000","last_active":"2024-03-04T07:38:58.808545+0000","last_peered":"2024-03-04T07:38:58.808545+0000","last_clean":"2024-03-04T07:38:58.808545+0000","last_became_active":"2024-03-04T07:37:55.137183+0000","last_became_peered":"2024-03-04T07:37:55.137183+0000","last_unstale":"2024-03-04T07:38:58.808545+0000","last_undegraded":"2024-03-04T07:38:58.808545+0000","last_fullsized":"2024-03-04T07:38:58.808545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:15.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503448+0000","last_change":"2024-03-04T07:37:55.136049+0000","last_active":"2024-03-04T07:39:27.503448+0000","last_peered":"2024-03-04T07:39:27.503448+0000","last_clean":"2024-03-04T07:39:27.503448+0000","last_became_active":"2024-03-04T07:37:55.132153+0000","last_became_peered":"2024-03-04T07:37:55.132153+0000","last_unstale":"2024-03-04T07:39:27.503448+0000","last_undegraded":"2024-03-04T07:39:27.503448+0000","last_fullsized":"2024-03-04T07:39:27.503448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:52.426892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.507866+0000","last_change":"2024-03-04T07:37:55.133571+0000","last_active":"2024-03-04T07:39:27.507866+0000","last_peered":"2024-03-04T07:39:27.507866+0000","last_clean":"2024-03-04T07:39:27.507866+0000","last_became_active":"2024-03-04T07:37:55.133214+0000","last_became_peered":"2024-03-04T07:37:55.133214+0000","last_unstale":"2024-03-04T07:39:27.507866+0000","last_undegraded":"2024-03-04T07:39:27.507866+0000","last_fullsized":"2024-03-04T07:39:27.507866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:17.924826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021258+0000","last_change":"2024-03-04T07:37:55.118146+0000","last_active":"2024-03-04T07:38:59.021258+0000","last_peered":"2024-03-04T07:38:59.021258+0000","last_clean":"2024-03-04T07:38:59.021258+0000","last_became_active":"2024-03-04T07:37:55.117687+0000","last_became_peered":"2024-03-04T07:37:55.117687+0000","last_unstale":"2024-03-04T07:38:59.021258+0000","last_undegraded":"2024-03-04T07:38:59.021258+0000","last_fullsized":"2024-03-04T07:38:59.021258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:34:35.343400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808113+0000","last_change":"2024-03-04T07:37:55.135943+0000","last_active":"2024-03-04T07:38:58.808113+0000","last_peered":"2024-03-04T07:38:58.808113+0000","last_clean":"2024-03-04T07:38:58.808113+0000","last_became_active":"2024-03-04T07:37:55.135549+0000","last_became_peered":"2024-03-04T07:37:55.135549+0000","last_unstale":"2024-03-04T07:38:58.808113+0000","last_undegraded":"2024-03-04T07:38:58.808113+0000","last_fullsized":"2024-03-04T07:38:58.808113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:35:53.953771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807386+0000","last_change":"2024-03-04T07:37:55.116949+0000","last_active":"2024-03-04T07:38:58.807386+0000","last_peered":"2024-03-04T07:38:58.807386+0000","last_clean":"2024-03-04T07:38:58.807386+0000","last_became_active":"2024-03-04T07:37:55.116620+0000","last_became_peered":"2024-03-04T07:37:55.116620+0000","last_unstale":"2024-03-04T07:38:58.807386+0000","last_undegraded":"2024-03-04T07:38:58.807386+0000","last_fullsized":"2024-03-04T07:38:58.807386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:39.655217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807114+0000","last_change":"2024-03-04T07:37:55.100975+0000","last_active":"2024-03-04T07:38:58.807114+0000","last_peered":"2024-03-04T07:38:58.807114+0000","last_clean":"2024-03-04T07:38:58.807114+0000","last_became_active":"2024-03-04T07:37:55.100636+0000","last_became_peered":"2024-03-04T07:37:55.100636+0000","last_unstale":"2024-03-04T07:38:58.807114+0000","last_undegraded":"2024-03-04T07:38:58.807114+0000","last_fullsized":"2024-03-04T07:38:58.807114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:29.593763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805855+0000","last_change":"2024-03-04T07:37:55.131895+0000","last_active":"2024-03-04T07:38:58.805855+0000","last_peered":"2024-03-04T07:38:58.805855+0000","last_clean":"2024-03-04T07:38:58.805855+0000","last_became_active":"2024-03-04T07:37:55.129682+0000","last_became_peered":"2024-03-04T07:37:55.129682+0000","last_unstale":"2024-03-04T07:38:58.805855+0000","last_undegraded":"2024-03-04T07:38:58.805855+0000","last_fullsized":"2024-03-04T07:38:58.805855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:54.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.802989+0000","last_change":"2024-03-04T07:37:55.114475+0000","last_active":"2024-03-04T07:38:58.802989+0000","last_peered":"2024-03-04T07:38:58.802989+0000","last_clean":"2024-03-04T07:38:58.802989+0000","last_became_active":"2024-03-04T07:37:55.114068+0000","last_became_peered":"2024-03-04T07:37:55.114068+0000","last_unstale":"2024-03-04T07:38:58.802989+0000","last_undegraded":"2024-03-04T07:38:58.802989+0000","last_fullsized":"2024-03-04T07:38:58.802989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:21.466143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807988+0000","last_change":"2024-03-04T07:37:55.115660+0000","last_active":"2024-03-04T07:38:58.807988+0000","last_peered":"2024-03-04T07:38:58.807988+0000","last_clean":"2024-03-04T07:38:58.807988+0000","last_became_active":"2024-03-04T07:37:55.115324+0000","last_became_peered":"2024-03-04T07:37:55.115324+0000","last_unstale":"2024-03-04T07:38:58.807988+0000","last_undegraded":"2024-03-04T07:38:58.807988+0000","last_fullsized":"2024-03-04T07:38:58.807988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:42.119728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807946+0000","last_change":"2024-03-04T07:37:55.119834+0000","last_active":"2024-03-04T07:38:58.807946+0000","last_peered":"2024-03-04T07:38:58.807946+0000","last_clean":"2024-03-04T07:38:58.807946+0000","last_became_active":"2024-03-04T07:37:55.119467+0000","last_became_peered":"2024-03-04T07:37:55.119467+0000","last_unstale":"2024-03-04T07:38:58.807946+0000","last_undegraded":"2024-03-04T07:38:58.807946+0000","last_fullsized":"2024-03-04T07:38:58.807946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:49:19.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021148+0000","last_change":"2024-03-04T07:37:55.118154+0000","last_active":"2024-03-04T07:38:59.021148+0000","last_peered":"2024-03-04T07:38:59.021148+0000","last_clean":"2024-03-04T07:38:59.021148+0000","last_became_active":"2024-03-04T07:37:55.117809+0000","last_became_peered":"2024-03-04T07:37:55.117809+0000","last_unstale":"2024-03-04T07:38:59.021148+0000","last_undegraded":"2024-03-04T07:38:59.021148+0000","last_fullsized":"2024-03-04T07:38:59.021148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:50.195946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804648+0000","last_change":"2024-03-04T07:37:55.123325+0000","last_active":"2024-03-04T07:38:58.804648+0000","last_peered":"2024-03-04T07:38:58.804648+0000","last_clean":"2024-03-04T07:38:58.804648+0000","last_became_active":"2024-03-04T07:37:55.122848+0000","last_became_peered":"2024-03-04T07:37:55.122848+0000","last_unstale":"2024-03-04T07:38:58.804648+0000","last_undegraded":"2024-03-04T07:38:58.804648+0000","last_fullsized":"2024-03-04T07:38:58.804648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:18.842011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502983+0000","last_change":"2024-03-04T07:37:55.132447+0000","last_active":"2024-03-04T07:39:27.502983+0000","last_peered":"2024-03-04T07:39:27.502983+0000","last_clean":"2024-03-04T07:39:27.502983+0000","last_became_active":"2024-03-04T07:37:55.131512+0000","last_became_peered":"2024-03-04T07:37:55.131512+0000","last_unstale":"2024-03-04T07:39:27.502983+0000","last_undegraded":"2024-03-04T07:39:27.502983+0000","last_fullsized":"2024-03-04T07:39:27.502983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:23.960029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504228+0000","last_change":"2024-03-04T07:37:55.131715+0000","last_active":"2024-03-04T07:39:27.504228+0000","last_peered":"2024-03-04T07:39:27.504228+0000","last_clean":"2024-03-04T07:39:27.504228+0000","last_became_active":"2024-03-04T07:37:55.131326+0000","last_became_peered":"2024-03-04T07:37:55.131326+0000","last_unstale":"2024-03-04T07:39:27.504228+0000","last_undegraded":"2024-03-04T07:39:27.504228+0000","last_fullsized":"2024-03-04T07:39:27.504228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:56.231578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.802439+0000","last_change":"2024-03-04T07:37:55.137395+0000","last_active":"2024-03-04T07:38:58.802439+0000","last_peered":"2024-03-04T07:38:58.802439+0000","last_clean":"2024-03-04T07:38:58.802439+0000","last_became_active":"2024-03-04T07:37:55.137054+0000","last_became_peered":"2024-03-04T07:37:55.137054+0000","last_unstale":"2024-03-04T07:38:58.802439+0000","last_undegraded":"2024-03-04T07:38:58.802439+0000","last_fullsized":"2024-03-04T07:38:58.802439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:19.222481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808821+0000","last_change":"2024-03-04T07:37:55.137414+0000","last_active":"2024-03-04T07:38:58.808821+0000","last_peered":"2024-03-04T07:38:58.808821+0000","last_clean":"2024-03-04T07:38:58.808821+0000","last_became_active":"2024-03-04T07:37:55.137157+0000","last_became_peered":"2024-03-04T07:37:55.137157+0000","last_unstale":"2024-03-04T07:38:58.808821+0000","last_undegraded":"2024-03-04T07:38:58.808821+0000","last_fullsized":"2024-03-04T07:38:58.808821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:36.593491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804964+0000","last_change":"2024-03-04T07:37:55.121960+0000","last_active":"2024-03-04T07:38:58.804964+0000","last_peered":"2024-03-04T07:38:58.804964+0000","last_clean":"2024-03-04T07:38:58.804964+0000","last_became_active":"2024-03-04T07:37:55.115543+0000","last_became_peered":"2024-03-04T07:37:55.115543+0000","last_unstale":"2024-03-04T07:38:58.804964+0000","last_undegraded":"2024-03-04T07:38:58.804964+0000","last_fullsized":"2024-03-04T07:38:58.804964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:36.320475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812632+0000","last_change":"2024-03-04T07:37:55.128908+0000","last_active":"2024-03-04T07:38:58.812632+0000","last_peered":"2024-03-04T07:38:58.812632+0000","last_clean":"2024-03-04T07:38:58.812632+0000","last_became_active":"2024-03-04T07:37:55.128500+0000","last_became_peered":"2024-03-04T07:37:55.128500+0000","last_unstale":"2024-03-04T07:38:58.812632+0000","last_undegraded":"2024-03-04T07:38:58.812632+0000","last_fullsized":"2024-03-04T07:38:58.812632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:10.598424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804966+0000","last_change":"2024-03-04T07:37:54.121610+0000","last_active":"2024-03-04T07:38:58.804966+0000","last_peered":"2024-03-04T07:38:58.804966+0000","last_clean":"2024-03-04T07:38:58.804966+0000","last_became_active":"2024-03-04T07:37:54.121150+0000","last_became_peered":"2024-03-04T07:37:54.121150+0000","last_unstale":"2024-03-04T07:38:58.804966+0000","last_undegraded":"2024-03-04T07:38:58.804966+0000","last_fullsized":"2024-03-04T07:38:58.804966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:05:03.965626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807498+0000","last_change":"2024-03-04T07:37:55.138904+0000","last_active":"2024-03-04T07:38:58.807498+0000","last_peered":"2024-03-04T07:38:58.807498+0000","last_clean":"2024-03-04T07:38:58.807498+0000","last_became_active":"2024-03-04T07:37:55.138376+0000","last_became_peered":"2024-03-04T07:37:55.138376+0000","last_unstale":"2024-03-04T07:38:58.807498+0000","last_undegraded":"2024-03-04T07:38:58.807498+0000","last_fullsized":"2024-03-04T07:38:58.807498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:28.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807796+0000","last_change":"2024-03-04T07:37:54.139218+0000","last_active":"2024-03-04T07:38:58.807796+0000","last_peered":"2024-03-04T07:38:58.807796+0000","last_clean":"2024-03-04T07:38:58.807796+0000","last_became_active":"2024-03-04T07:37:54.138893+0000","last_became_peered":"2024-03-04T07:37:54.138893+0000","last_unstale":"2024-03-04T07:38:58.807796+0000","last_undegraded":"2024-03-04T07:38:58.807796+0000","last_fullsized":"2024-03-04T07:38:58.807796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:01.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.508413+0000","last_change":"2024-03-04T07:37:55.135953+0000","last_active":"2024-03-04T07:39:27.508413+0000","last_peered":"2024-03-04T07:39:27.508413+0000","last_clean":"2024-03-04T07:39:27.508413+0000","last_became_active":"2024-03-04T07:37:55.132880+0000","last_became_peered":"2024-03-04T07:37:55.132880+0000","last_unstale":"2024-03-04T07:39:27.508413+0000","last_undegraded":"2024-03-04T07:39:27.508413+0000","last_fullsized":"2024-03-04T07:39:27.508413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:03.848428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808582+0000","last_change":"2024-03-04T07:37:54.122493+0000","last_active":"2024-03-04T07:38:58.808582+0000","last_peered":"2024-03-04T07:38:58.808582+0000","last_clean":"2024-03-04T07:38:58.808582+0000","last_became_active":"2024-03-04T07:37:54.122058+0000","last_became_peered":"2024-03-04T07:37:54.122058+0000","last_unstale":"2024-03-04T07:38:58.808582+0000","last_undegraded":"2024-03-04T07:38:58.808582+0000","last_fullsized":"2024-03-04T07:38:58.808582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:02.516175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.802661+0000","last_change":"2024-03-04T07:37:55.136155+0000","last_active":"2024-03-04T07:38:58.802661+0000","last_peered":"2024-03-04T07:38:58.802661+0000","last_clean":"2024-03-04T07:38:58.802661+0000","last_became_active":"2024-03-04T07:37:55.135796+0000","last_became_peered":"2024-03-04T07:37:55.135796+0000","last_unstale":"2024-03-04T07:38:58.802661+0000","last_undegraded":"2024-03-04T07:38:58.802661+0000","last_fullsized":"2024-03-04T07:38:58.802661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:37.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808437+0000","last_change":"2024-03-04T07:37:54.118780+0000","last_active":"2024-03-04T07:38:58.808437+0000","last_peered":"2024-03-04T07:38:58.808437+0000","last_clean":"2024-03-04T07:38:58.808437+0000","last_became_active":"2024-03-04T07:37:54.118448+0000","last_became_peered":"2024-03-04T07:37:54.118448+0000","last_unstale":"2024-03-04T07:38:58.808437+0000","last_undegraded":"2024-03-04T07:38:58.808437+0000","last_fullsized":"2024-03-04T07:38:58.808437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:25.547046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.022350+0000","last_change":"2024-03-04T07:38:59.021729+0000","last_active":"2024-03-04T07:38:59.022350+0000","last_peered":"2024-03-04T07:38:59.022350+0000","last_clean":"2024-03-04T07:38:59.022350+0000","last_became_active":"2024-03-04T07:38:59.021127+0000","last_became_peered":"2024-03-04T07:38:59.021127+0000","last_unstale":"2024-03-04T07:38:59.022350+0000","last_undegraded":"2024-03-04T07:38:59.022350+0000","last_fullsized":"2024-03-04T07:38:59.022350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:08:07.825530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807825+0000","last_change":"2024-03-04T07:37:54.122241+0000","last_active":"2024-03-04T07:38:58.807825+0000","last_peered":"2024-03-04T07:38:58.807825+0000","last_clean":"2024-03-04T07:38:58.807825+0000","last_became_active":"2024-03-04T07:37:54.120990+0000","last_became_peered":"2024-03-04T07:37:54.120990+0000","last_unstale":"2024-03-04T07:38:58.807825+0000","last_undegraded":"2024-03-04T07:38:58.807825+0000","last_fullsized":"2024-03-04T07:38:58.807825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:03.233754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807929+0000","last_change":"2024-03-04T07:37:55.121692+0000","last_active":"2024-03-04T07:38:58.807929+0000","last_peered":"2024-03-04T07:38:58.807929+0000","last_clean":"2024-03-04T07:38:58.807929+0000","last_became_active":"2024-03-04T07:37:55.121345+0000","last_became_peered":"2024-03-04T07:37:55.121345+0000","last_unstale":"2024-03-04T07:38:58.807929+0000","last_undegraded":"2024-03-04T07:38:58.807929+0000","last_fullsized":"2024-03-04T07:38:58.807929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:52:42.045265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807382+0000","last_change":"2024-03-04T07:37:54.133489+0000","last_active":"2024-03-04T07:38:58.807382+0000","last_peered":"2024-03-04T07:38:58.807382+0000","last_clean":"2024-03-04T07:38:58.807382+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807382+0000","last_undegraded":"2024-03-04T07:38:58.807382+0000","last_fullsized":"2024-03-04T07:38:58.807382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:49:03.600907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807960+0000","last_change":"2024-03-04T07:37:55.128278+0000","last_active":"2024-03-04T07:38:58.807960+0000","last_peered":"2024-03-04T07:38:58.807960+0000","last_clean":"2024-03-04T07:38:58.807960+0000","last_became_active":"2024-03-04T07:37:55.127913+0000","last_became_peered":"2024-03-04T07:37:55.127913+0000","last_unstale":"2024-03-04T07:38:58.807960+0000","last_undegraded":"2024-03-04T07:38:58.807960+0000","last_fullsized":"2024-03-04T07:38:58.807960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:54:18.623855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803806+0000","last_change":"2024-03-04T07:37:54.132195+0000","last_active":"2024-03-04T07:38:58.803806+0000","last_peered":"2024-03-04T07:38:58.803806+0000","last_clean":"2024-03-04T07:38:58.803806+0000","last_became_active":"2024-03-04T07:37:54.131830+0000","last_became_peered":"2024-03-04T07:37:54.131830+0000","last_unstale":"2024-03-04T07:38:58.803806+0000","last_undegraded":"2024-03-04T07:38:58.803806+0000","last_fullsized":"2024-03-04T07:38:58.803806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.744342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.806054+0000","last_change":"2024-03-04T07:37:55.138281+0000","last_active":"2024-03-04T07:38:58.806054+0000","last_peered":"2024-03-04T07:38:58.806054+0000","last_clean":"2024-03-04T07:38:58.806054+0000","last_became_active":"2024-03-04T07:37:55.137919+0000","last_became_peered":"2024-03-04T07:37:55.137919+0000","last_unstale":"2024-03-04T07:38:58.806054+0000","last_undegraded":"2024-03-04T07:38:58.806054+0000","last_fullsized":"2024-03-04T07:38:58.806054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:53:06.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807552+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:38:58.807552+0000","last_peered":"2024-03-04T07:38:58.807552+0000","last_clean":"2024-03-04T07:38:58.807552+0000","last_became_active":"2024-03-04T07:37:54.121503+0000","last_became_peered":"2024-03-04T07:37:54.121503+0000","last_unstale":"2024-03-04T07:38:58.807552+0000","last_undegraded":"2024-03-04T07:38:58.807552+0000","last_fullsized":"2024-03-04T07:38:58.807552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:03:07.531246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804317+0000","last_change":"2024-03-04T07:37:55.126924+0000","last_active":"2024-03-04T07:38:58.804317+0000","last_peered":"2024-03-04T07:38:58.804317+0000","last_clean":"2024-03-04T07:38:58.804317+0000","last_became_active":"2024-03-04T07:37:55.126266+0000","last_became_peered":"2024-03-04T07:37:55.126266+0000","last_unstale":"2024-03-04T07:38:58.804317+0000","last_undegraded":"2024-03-04T07:38:58.804317+0000","last_fullsized":"2024-03-04T07:38:58.804317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:01.603430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807869+0000","last_change":"2024-03-04T07:37:54.119864+0000","last_active":"2024-03-04T07:38:58.807869+0000","last_peered":"2024-03-04T07:38:58.807869+0000","last_clean":"2024-03-04T07:38:58.807869+0000","last_became_active":"2024-03-04T07:37:54.119515+0000","last_became_peered":"2024-03-04T07:37:54.119515+0000","last_unstale":"2024-03-04T07:38:58.807869+0000","last_undegraded":"2024-03-04T07:38:58.807869+0000","last_fullsized":"2024-03-04T07:38:58.807869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:10.422458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.802853+0000","last_change":"2024-03-04T07:37:55.120135+0000","last_active":"2024-03-04T07:38:58.802853+0000","last_peered":"2024-03-04T07:38:58.802853+0000","last_clean":"2024-03-04T07:38:58.802853+0000","last_became_active":"2024-03-04T07:37:55.119670+0000","last_became_peered":"2024-03-04T07:37:55.119670+0000","last_unstale":"2024-03-04T07:38:58.802853+0000","last_undegraded":"2024-03-04T07:38:58.802853+0000","last_fullsized":"2024-03-04T07:38:58.802853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:09.688447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807178+0000","last_change":"2024-03-04T07:37:54.120222+0000","last_active":"2024-03-04T07:38:58.807178+0000","last_peered":"2024-03-04T07:38:58.807178+0000","last_clean":"2024-03-04T07:38:58.807178+0000","last_became_active":"2024-03-04T07:37:54.119802+0000","last_became_peered":"2024-03-04T07:37:54.119802+0000","last_unstale":"2024-03-04T07:38:58.807178+0000","last_undegraded":"2024-03-04T07:38:58.807178+0000","last_fullsized":"2024-03-04T07:38:58.807178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:12:40.811870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.508249+0000","last_change":"2024-03-04T07:37:55.136371+0000","last_active":"2024-03-04T07:39:27.508249+0000","last_peered":"2024-03-04T07:39:27.508249+0000","last_clean":"2024-03-04T07:39:27.508249+0000","last_became_active":"2024-03-04T07:37:55.132587+0000","last_became_peered":"2024-03-04T07:37:55.132587+0000","last_unstale":"2024-03-04T07:39:27.508249+0000","last_undegraded":"2024-03-04T07:39:27.508249+0000","last_fullsized":"2024-03-04T07:39:27.508249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:24.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.809008+0000","last_change":"2024-03-04T07:37:54.123041+0000","last_active":"2024-03-04T07:38:58.809008+0000","last_peered":"2024-03-04T07:38:58.809008+0000","last_clean":"2024-03-04T07:38:58.809008+0000","last_became_active":"2024-03-04T07:37:54.122723+0000","last_became_peered":"2024-03-04T07:37:54.122723+0000","last_unstale":"2024-03-04T07:38:58.809008+0000","last_undegraded":"2024-03-04T07:38:58.809008+0000","last_fullsized":"2024-03-04T07:38:58.809008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:06:30.845376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.802828+0000","last_change":"2024-03-04T07:37:55.115387+0000","last_active":"2024-03-04T07:38:58.802828+0000","last_peered":"2024-03-04T07:38:58.802828+0000","last_clean":"2024-03-04T07:38:58.802828+0000","last_became_active":"2024-03-04T07:37:55.114947+0000","last_became_peered":"2024-03-04T07:37:55.114947+0000","last_unstale":"2024-03-04T07:38:58.802828+0000","last_undegraded":"2024-03-04T07:38:58.802828+0000","last_fullsized":"2024-03-04T07:38:58.802828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:09:11.840989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'26","reported_seq":233,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:26.951193+0000","last_change":"2024-03-04T07:37:54.117767+0000","last_active":"2024-03-04T07:39:26.951193+0000","last_peered":"2024-03-04T07:39:26.951193+0000","last_clean":"2024-03-04T07:39:26.951193+0000","last_became_active":"2024-03-04T07:37:54.117371+0000","last_became_peered":"2024-03-04T07:37:54.117371+0000","last_unstale":"2024-03-04T07:39:26.951193+0000","last_undegraded":"2024-03-04T07:39:26.951193+0000","last_fullsized":"2024-03-04T07:39:26.951193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:08:32.164636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":188,"num_read_kb":153,"num_write":26,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808437+0000","last_change":"2024-03-04T07:37:55.112707+0000","last_active":"2024-03-04T07:38:58.808437+0000","last_peered":"2024-03-04T07:38:58.808437+0000","last_clean":"2024-03-04T07:38:58.808437+0000","last_became_active":"2024-03-04T07:37:55.112327+0000","last_became_peered":"2024-03-04T07:37:55.112327+0000","last_unstale":"2024-03-04T07:38:58.808437+0000","last_undegraded":"2024-03-04T07:38:58.808437+0000","last_fullsized":"2024-03-04T07:38:58.808437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:03.901687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811717+0000","last_change":"2024-03-04T07:37:54.125286+0000","last_active":"2024-03-04T07:38:58.811717+0000","last_peered":"2024-03-04T07:38:58.811717+0000","last_clean":"2024-03-04T07:38:58.811717+0000","last_became_active":"2024-03-04T07:37:54.124966+0000","last_became_peered":"2024-03-04T07:37:54.124966+0000","last_unstale":"2024-03-04T07:38:58.811717+0000","last_undegraded":"2024-03-04T07:38:58.811717+0000","last_fullsized":"2024-03-04T07:38:58.811717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:19.880681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803955+0000","last_change":"2024-03-04T07:37:55.137952+0000","last_active":"2024-03-04T07:38:58.803955+0000","last_peered":"2024-03-04T07:38:58.803955+0000","last_clean":"2024-03-04T07:38:58.803955+0000","last_became_active":"2024-03-04T07:37:55.137583+0000","last_became_peered":"2024-03-04T07:37:55.137583+0000","last_unstale":"2024-03-04T07:38:58.803955+0000","last_undegraded":"2024-03-04T07:38:58.803955+0000","last_fullsized":"2024-03-04T07:38:58.803955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:30.145453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807526+0000","last_change":"2024-03-04T07:37:54.134733+0000","last_active":"2024-03-04T07:38:58.807526+0000","last_peered":"2024-03-04T07:38:58.807526+0000","last_clean":"2024-03-04T07:38:58.807526+0000","last_became_active":"2024-03-04T07:37:54.134400+0000","last_became_peered":"2024-03-04T07:37:54.134400+0000","last_unstale":"2024-03-04T07:38:58.807526+0000","last_undegraded":"2024-03-04T07:38:58.807526+0000","last_fullsized":"2024-03-04T07:38:58.807526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:24.192479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807780+0000","last_change":"2024-03-04T07:37:55.124991+0000","last_active":"2024-03-04T07:38:58.807780+0000","last_peered":"2024-03-04T07:38:58.807780+0000","last_clean":"2024-03-04T07:38:58.807780+0000","last_became_active":"2024-03-04T07:37:55.124621+0000","last_became_peered":"2024-03-04T07:37:55.124621+0000","last_unstale":"2024-03-04T07:38:58.807780+0000","last_undegraded":"2024-03-04T07:38:58.807780+0000","last_fullsized":"2024-03-04T07:38:58.807780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:05.252145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.020968+0000","last_change":"2024-03-04T07:37:54.118743+0000","last_active":"2024-03-04T07:38:59.020968+0000","last_peered":"2024-03-04T07:38:59.020968+0000","last_clean":"2024-03-04T07:38:59.020968+0000","last_became_active":"2024-03-04T07:37:54.118386+0000","last_became_peered":"2024-03-04T07:37:54.118386+0000","last_unstale":"2024-03-04T07:38:59.020968+0000","last_undegraded":"2024-03-04T07:38:59.020968+0000","last_fullsized":"2024-03-04T07:38:59.020968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:27.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807028+0000","last_change":"2024-03-04T07:37:55.115431+0000","last_active":"2024-03-04T07:38:58.807028+0000","last_peered":"2024-03-04T07:38:58.807028+0000","last_clean":"2024-03-04T07:38:58.807028+0000","last_became_active":"2024-03-04T07:37:55.115064+0000","last_became_peered":"2024-03-04T07:37:55.115064+0000","last_unstale":"2024-03-04T07:38:58.807028+0000","last_undegraded":"2024-03-04T07:38:58.807028+0000","last_fullsized":"2024-03-04T07:38:58.807028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:46:57.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.028943+0000","last_change":"2024-03-04T07:38:59.028943+0000","last_active":"2024-03-04T07:38:59.028943+0000","last_peered":"2024-03-04T07:38:59.028943+0000","last_clean":"2024-03-04T07:38:59.028943+0000","last_became_active":"2024-03-04T07:38:59.022232+0000","last_became_peered":"2024-03-04T07:38:59.022232+0000","last_unstale":"2024-03-04T07:38:59.028943+0000","last_undegraded":"2024-03-04T07:38:59.028943+0000","last_fullsized":"2024-03-04T07:38:59.028943+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:49.281601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807560+0000","last_change":"2024-03-04T07:37:55.114882+0000","last_active":"2024-03-04T07:38:58.807560+0000","last_peered":"2024-03-04T07:38:58.807560+0000","last_clean":"2024-03-04T07:38:58.807560+0000","last_became_active":"2024-03-04T07:37:55.113139+0000","last_became_peered":"2024-03-04T07:37:55.113139+0000","last_unstale":"2024-03-04T07:38:58.807560+0000","last_undegraded":"2024-03-04T07:38:58.807560+0000","last_fullsized":"2024-03-04T07:38:58.807560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:28.133710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808725+0000","last_change":"2024-03-04T07:37:54.119350+0000","last_active":"2024-03-04T07:38:58.808725+0000","last_peered":"2024-03-04T07:38:58.808725+0000","last_clean":"2024-03-04T07:38:58.808725+0000","last_became_active":"2024-03-04T07:37:54.119017+0000","last_became_peered":"2024-03-04T07:37:54.119017+0000","last_unstale":"2024-03-04T07:38:58.808725+0000","last_undegraded":"2024-03-04T07:38:58.808725+0000","last_fullsized":"2024-03-04T07:38:58.808725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:00.503583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812598+0000","last_change":"2024-03-04T07:38:58.812598+0000","last_active":"2024-03-04T07:38:58.812598+0000","last_peered":"2024-03-04T07:38:58.812598+0000","last_clean":"2024-03-04T07:38:58.812598+0000","last_became_active":"2024-03-04T07:38:58.812187+0000","last_became_peered":"2024-03-04T07:38:58.812187+0000","last_unstale":"2024-03-04T07:38:58.812598+0000","last_undegraded":"2024-03-04T07:38:58.812598+0000","last_fullsized":"2024-03-04T07:38:58.812598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:43.753007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804378+0000","last_change":"2024-03-04T07:37:54.125548+0000","last_active":"2024-03-04T07:38:58.804378+0000","last_peered":"2024-03-04T07:38:58.804378+0000","last_clean":"2024-03-04T07:38:58.804378+0000","last_became_active":"2024-03-04T07:37:54.125127+0000","last_became_peered":"2024-03-04T07:37:54.125127+0000","last_unstale":"2024-03-04T07:38:58.804378+0000","last_undegraded":"2024-03-04T07:38:58.804378+0000","last_fullsized":"2024-03-04T07:38:58.804378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:26.256247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808153+0000","last_change":"2024-03-04T07:37:55.128542+0000","last_active":"2024-03-04T07:38:58.808153+0000","last_peered":"2024-03-04T07:38:58.808153+0000","last_clean":"2024-03-04T07:38:58.808153+0000","last_became_active":"2024-03-04T07:37:55.128239+0000","last_became_peered":"2024-03-04T07:37:55.128239+0000","last_unstale":"2024-03-04T07:38:58.808153+0000","last_undegraded":"2024-03-04T07:38:58.808153+0000","last_fullsized":"2024-03-04T07:38:58.808153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:17.078095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807344+0000","last_change":"2024-03-04T07:37:54.133672+0000","last_active":"2024-03-04T07:38:58.807344+0000","last_peered":"2024-03-04T07:38:58.807344+0000","last_clean":"2024-03-04T07:38:58.807344+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807344+0000","last_undegraded":"2024-03-04T07:38:58.807344+0000","last_fullsized":"2024-03-04T07:38:58.807344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:52.345158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.815766+0000","last_change":"2024-03-04T07:38:58.815766+0000","last_active":"2024-03-04T07:38:58.815766+0000","last_peered":"2024-03-04T07:38:58.815766+0000","last_clean":"2024-03-04T07:38:58.815766+0000","last_became_active":"2024-03-04T07:38:58.815387+0000","last_became_peered":"2024-03-04T07:38:58.815387+0000","last_unstale":"2024-03-04T07:38:58.815766+0000","last_undegraded":"2024-03-04T07:38:58.815766+0000","last_fullsized":"2024-03-04T07:38:58.815766+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:21.582879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.814581+0000","last_change":"2024-03-04T07:38:58.814581+0000","last_active":"2024-03-04T07:38:58.814581+0000","last_peered":"2024-03-04T07:38:58.814581+0000","last_clean":"2024-03-04T07:38:58.814581+0000","last_became_active":"2024-03-04T07:38:58.814186+0000","last_became_peered":"2024-03-04T07:38:58.814186+0000","last_unstale":"2024-03-04T07:38:58.814581+0000","last_undegraded":"2024-03-04T07:38:58.814581+0000","last_fullsized":"2024-03-04T07:38:58.814581+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:03:33.824292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803718+0000","last_change":"2024-03-04T07:37:54.139335+0000","last_active":"2024-03-04T07:38:58.803718+0000","last_peered":"2024-03-04T07:38:58.803718+0000","last_clean":"2024-03-04T07:38:58.803718+0000","last_became_active":"2024-03-04T07:37:54.138969+0000","last_became_peered":"2024-03-04T07:37:54.138969+0000","last_unstale":"2024-03-04T07:38:58.803718+0000","last_undegraded":"2024-03-04T07:38:58.803718+0000","last_fullsized":"2024-03-04T07:38:58.803718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:41:15.204738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.810554+0000","last_change":"2024-03-04T07:38:58.810554+0000","last_active":"2024-03-04T07:38:58.810554+0000","last_peered":"2024-03-04T07:38:58.810554+0000","last_clean":"2024-03-04T07:38:58.810554+0000","last_became_active":"2024-03-04T07:38:58.809978+0000","last_became_peered":"2024-03-04T07:38:58.809978+0000","last_unstale":"2024-03-04T07:38:58.810554+0000","last_undegraded":"2024-03-04T07:38:58.810554+0000","last_fullsized":"2024-03-04T07:38:58.810554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:33.417630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.020981+0000","last_change":"2024-03-04T07:37:54.119906+0000","last_active":"2024-03-04T07:38:59.020981+0000","last_peered":"2024-03-04T07:38:59.020981+0000","last_clean":"2024-03-04T07:38:59.020981+0000","last_became_active":"2024-03-04T07:37:54.119592+0000","last_became_peered":"2024-03-04T07:37:54.119592+0000","last_unstale":"2024-03-04T07:38:59.020981+0000","last_undegraded":"2024-03-04T07:38:59.020981+0000","last_fullsized":"2024-03-04T07:38:59.020981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:29.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":343,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807128+0000","last_change":"2024-03-04T07:36:35.278528+0000","last_active":"2024-03-04T07:38:58.807128+0000","last_peered":"2024-03-04T07:38:58.807128+0000","last_clean":"2024-03-04T07:38:58.807128+0000","last_became_active":"2024-03-04T07:36:35.277953+0000","last_became_peered":"2024-03-04T07:36:35.277953+0000","last_unstale":"2024-03-04T07:38:58.807128+0000","last_undegraded":"2024-03-04T07:38:58.807128+0000","last_fullsized":"2024-03-04T07:38:58.807128+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_clean_scrub_stamp":"2024-03-04T07:32:42.283525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:58:47.223527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803587+0000","last_change":"2024-03-04T07:37:55.124204+0000","last_active":"2024-03-04T07:38:58.803587+0000","last_peered":"2024-03-04T07:38:58.803587+0000","last_clean":"2024-03-04T07:38:58.803587+0000","last_became_active":"2024-03-04T07:37:55.123863+0000","last_became_peered":"2024-03-04T07:37:55.123863+0000","last_unstale":"2024-03-04T07:38:58.803587+0000","last_undegraded":"2024-03-04T07:38:58.803587+0000","last_fullsized":"2024-03-04T07:38:58.803587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:33.830367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807200+0000","last_change":"2024-03-04T07:37:54.121538+0000","last_active":"2024-03-04T07:38:58.807200+0000","last_peered":"2024-03-04T07:38:58.807200+0000","last_clean":"2024-03-04T07:38:58.807200+0000","last_became_active":"2024-03-04T07:37:54.120939+0000","last_became_peered":"2024-03-04T07:37:54.120939+0000","last_unstale":"2024-03-04T07:38:58.807200+0000","last_undegraded":"2024-03-04T07:38:58.807200+0000","last_fullsized":"2024-03-04T07:38:58.807200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:29.597247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808482+0000","last_change":"2024-03-04T07:37:54.128180+0000","last_active":"2024-03-04T07:38:58.808482+0000","last_peered":"2024-03-04T07:38:58.808482+0000","last_clean":"2024-03-04T07:38:58.808482+0000","last_became_active":"2024-03-04T07:37:54.127800+0000","last_became_peered":"2024-03-04T07:37:54.127800+0000","last_unstale":"2024-03-04T07:38:58.808482+0000","last_undegraded":"2024-03-04T07:38:58.808482+0000","last_fullsized":"2024-03-04T07:38:58.808482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:54.046873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804616+0000","last_change":"2024-03-04T07:37:55.133482+0000","last_active":"2024-03-04T07:38:58.804616+0000","last_peered":"2024-03-04T07:38:58.804616+0000","last_clean":"2024-03-04T07:38:58.804616+0000","last_became_active":"2024-03-04T07:37:55.133137+0000","last_became_peered":"2024-03-04T07:37:55.133137+0000","last_unstale":"2024-03-04T07:38:58.804616+0000","last_undegraded":"2024-03-04T07:38:58.804616+0000","last_fullsized":"2024-03-04T07:38:58.804616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:53.509564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805099+0000","last_change":"2024-03-04T07:37:54.134294+0000","last_active":"2024-03-04T07:38:58.805099+0000","last_peered":"2024-03-04T07:38:58.805099+0000","last_clean":"2024-03-04T07:38:58.805099+0000","last_became_active":"2024-03-04T07:37:54.133677+0000","last_became_peered":"2024-03-04T07:37:54.133677+0000","last_unstale":"2024-03-04T07:38:58.805099+0000","last_undegraded":"2024-03-04T07:38:58.805099+0000","last_fullsized":"2024-03-04T07:38:58.805099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:57:54.082419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807245+0000","last_change":"2024-03-04T07:37:55.138305+0000","last_active":"2024-03-04T07:38:58.807245+0000","last_peered":"2024-03-04T07:38:58.807245+0000","last_clean":"2024-03-04T07:38:58.807245+0000","last_became_active":"2024-03-04T07:37:55.129937+0000","last_became_peered":"2024-03-04T07:37:55.129937+0000","last_unstale":"2024-03-04T07:38:58.807245+0000","last_undegraded":"2024-03-04T07:38:58.807245+0000","last_fullsized":"2024-03-04T07:38:58.807245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:45.136539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021251+0000","last_change":"2024-03-04T07:37:54.133817+0000","last_active":"2024-03-04T07:38:59.021251+0000","last_peered":"2024-03-04T07:38:59.021251+0000","last_clean":"2024-03-04T07:38:59.021251+0000","last_became_active":"2024-03-04T07:37:54.133471+0000","last_became_peered":"2024-03-04T07:37:54.133471+0000","last_unstale":"2024-03-04T07:38:59.021251+0000","last_undegraded":"2024-03-04T07:38:59.021251+0000","last_fullsized":"2024-03-04T07:38:59.021251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:00:51.483859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.814628+0000","last_change":"2024-03-04T07:37:55.144639+0000","last_active":"2024-03-04T07:38:58.814628+0000","last_peered":"2024-03-04T07:38:58.814628+0000","last_clean":"2024-03-04T07:38:58.814628+0000","last_became_active":"2024-03-04T07:37:55.144194+0000","last_became_peered":"2024-03-04T07:37:55.144194+0000","last_unstale":"2024-03-04T07:38:58.814628+0000","last_undegraded":"2024-03-04T07:38:58.814628+0000","last_fullsized":"2024-03-04T07:38:58.814628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:42.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.815642+0000","last_change":"2024-03-04T07:38:58.815642+0000","last_active":"2024-03-04T07:38:58.815642+0000","last_peered":"2024-03-04T07:38:58.815642+0000","last_clean":"2024-03-04T07:38:58.815642+0000","last_became_active":"2024-03-04T07:38:58.815361+0000","last_became_peered":"2024-03-04T07:38:58.815361+0000","last_unstale":"2024-03-04T07:38:58.815642+0000","last_undegraded":"2024-03-04T07:38:58.815642+0000","last_fullsized":"2024-03-04T07:38:58.815642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:48:15.751929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807722+0000","last_change":"2024-03-04T07:37:55.115145+0000","last_active":"2024-03-04T07:38:58.807722+0000","last_peered":"2024-03-04T07:38:58.807722+0000","last_clean":"2024-03-04T07:38:58.807722+0000","last_became_active":"2024-03-04T07:37:55.114791+0000","last_became_peered":"2024-03-04T07:37:55.114791+0000","last_unstale":"2024-03-04T07:38:58.807722+0000","last_undegraded":"2024-03-04T07:38:58.807722+0000","last_fullsized":"2024-03-04T07:38:58.807722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:19.685977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.806378+0000","last_change":"2024-03-04T07:37:54.141630+0000","last_active":"2024-03-04T07:38:58.806378+0000","last_peered":"2024-03-04T07:38:58.806378+0000","last_clean":"2024-03-04T07:38:58.806378+0000","last_became_active":"2024-03-04T07:37:54.139820+0000","last_became_peered":"2024-03-04T07:37:54.139820+0000","last_unstale":"2024-03-04T07:38:58.806378+0000","last_undegraded":"2024-03-04T07:38:58.806378+0000","last_fullsized":"2024-03-04T07:38:58.806378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:22.419437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808319+0000","last_change":"2024-03-04T07:37:55.133270+0000","last_active":"2024-03-04T07:38:58.808319+0000","last_peered":"2024-03-04T07:38:58.808319+0000","last_clean":"2024-03-04T07:38:58.808319+0000","last_became_active":"2024-03-04T07:37:55.132909+0000","last_became_peered":"2024-03-04T07:37:55.132909+0000","last_unstale":"2024-03-04T07:38:58.808319+0000","last_undegraded":"2024-03-04T07:38:58.808319+0000","last_fullsized":"2024-03-04T07:38:58.808319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:21.026141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812525+0000","last_change":"2024-03-04T07:37:54.135988+0000","last_active":"2024-03-04T07:38:58.812525+0000","last_peered":"2024-03-04T07:38:58.812525+0000","last_clean":"2024-03-04T07:38:58.812525+0000","last_became_active":"2024-03-04T07:37:54.135627+0000","last_became_peered":"2024-03-04T07:37:54.135627+0000","last_unstale":"2024-03-04T07:38:58.812525+0000","last_undegraded":"2024-03-04T07:38:58.812525+0000","last_fullsized":"2024-03-04T07:38:58.812525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:06.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805784+0000","last_change":"2024-03-04T07:37:55.144551+0000","last_active":"2024-03-04T07:38:58.805784+0000","last_peered":"2024-03-04T07:38:58.805784+0000","last_clean":"2024-03-04T07:38:58.805784+0000","last_became_active":"2024-03-04T07:37:55.143882+0000","last_became_peered":"2024-03-04T07:37:55.143882+0000","last_unstale":"2024-03-04T07:38:58.805784+0000","last_undegraded":"2024-03-04T07:38:58.805784+0000","last_fullsized":"2024-03-04T07:38:58.805784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:07.254941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.817917+0000","last_change":"2024-03-04T07:38:58.817440+0000","last_active":"2024-03-04T07:38:58.817917+0000","last_peered":"2024-03-04T07:38:58.817917+0000","last_clean":"2024-03-04T07:38:58.817917+0000","last_became_active":"2024-03-04T07:38:58.813620+0000","last_became_peered":"2024-03-04T07:38:58.813620+0000","last_unstale":"2024-03-04T07:38:58.817917+0000","last_undegraded":"2024-03-04T07:38:58.817917+0000","last_fullsized":"2024-03-04T07:38:58.817917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:10:41.554608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807406+0000","last_change":"2024-03-04T07:37:55.133870+0000","last_active":"2024-03-04T07:38:58.807406+0000","last_peered":"2024-03-04T07:38:58.807406+0000","last_clean":"2024-03-04T07:38:58.807406+0000","last_became_active":"2024-03-04T07:37:55.133542+0000","last_became_peered":"2024-03-04T07:37:55.133542+0000","last_unstale":"2024-03-04T07:38:58.807406+0000","last_undegraded":"2024-03-04T07:38:58.807406+0000","last_fullsized":"2024-03-04T07:38:58.807406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:17:44.373426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811695+0000","last_change":"2024-03-04T07:37:54.126326+0000","last_active":"2024-03-04T07:38:58.811695+0000","last_peered":"2024-03-04T07:38:58.811695+0000","last_clean":"2024-03-04T07:38:58.811695+0000","last_became_active":"2024-03-04T07:37:54.125989+0000","last_became_peered":"2024-03-04T07:37:54.125989+0000","last_unstale":"2024-03-04T07:38:58.811695+0000","last_undegraded":"2024-03-04T07:38:58.811695+0000","last_fullsized":"2024-03-04T07:38:58.811695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:41.108528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808103+0000","last_change":"2024-03-04T07:37:55.120052+0000","last_active":"2024-03-04T07:38:58.808103+0000","last_peered":"2024-03-04T07:38:58.808103+0000","last_clean":"2024-03-04T07:38:58.808103+0000","last_became_active":"2024-03-04T07:37:55.119708+0000","last_became_peered":"2024-03-04T07:37:55.119708+0000","last_unstale":"2024-03-04T07:38:58.808103+0000","last_undegraded":"2024-03-04T07:38:58.808103+0000","last_fullsized":"2024-03-04T07:38:58.808103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:25:17.679416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808459+0000","last_change":"2024-03-04T07:37:54.134018+0000","last_active":"2024-03-04T07:38:58.808459+0000","last_peered":"2024-03-04T07:38:58.808459+0000","last_clean":"2024-03-04T07:38:58.808459+0000","last_became_active":"2024-03-04T07:37:54.133673+0000","last_became_peered":"2024-03-04T07:37:54.133673+0000","last_unstale":"2024-03-04T07:38:58.808459+0000","last_undegraded":"2024-03-04T07:38:58.808459+0000","last_fullsized":"2024-03-04T07:38:58.808459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:49.438338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.817071+0000","last_change":"2024-03-04T07:38:58.817071+0000","last_active":"2024-03-04T07:38:58.817071+0000","last_peered":"2024-03-04T07:38:58.817071+0000","last_clean":"2024-03-04T07:38:58.817071+0000","last_became_active":"2024-03-04T07:38:58.812139+0000","last_became_peered":"2024-03-04T07:38:58.812139+0000","last_unstale":"2024-03-04T07:38:58.817071+0000","last_undegraded":"2024-03-04T07:38:58.817071+0000","last_fullsized":"2024-03-04T07:38:58.817071+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:37.557255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805022+0000","last_change":"2024-03-04T07:37:54.140806+0000","last_active":"2024-03-04T07:38:58.805022+0000","last_peered":"2024-03-04T07:38:58.805022+0000","last_clean":"2024-03-04T07:38:58.805022+0000","last_became_active":"2024-03-04T07:37:54.140419+0000","last_became_peered":"2024-03-04T07:37:54.140419+0000","last_unstale":"2024-03-04T07:38:58.805022+0000","last_undegraded":"2024-03-04T07:38:58.805022+0000","last_fullsized":"2024-03-04T07:38:58.805022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:13:59.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808277+0000","last_change":"2024-03-04T07:37:55.113658+0000","last_active":"2024-03-04T07:38:58.808277+0000","last_peered":"2024-03-04T07:38:58.808277+0000","last_clean":"2024-03-04T07:38:58.808277+0000","last_became_active":"2024-03-04T07:37:55.113300+0000","last_became_peered":"2024-03-04T07:37:55.113300+0000","last_unstale":"2024-03-04T07:38:58.808277+0000","last_undegraded":"2024-03-04T07:38:58.808277+0000","last_fullsized":"2024-03-04T07:38:58.808277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:43:56.907387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808475+0000","last_change":"2024-03-04T07:37:55.117769+0000","last_active":"2024-03-04T07:38:58.808475+0000","last_peered":"2024-03-04T07:38:58.808475+0000","last_clean":"2024-03-04T07:38:58.808475+0000","last_became_active":"2024-03-04T07:37:55.117408+0000","last_became_peered":"2024-03-04T07:37:55.117408+0000","last_unstale":"2024-03-04T07:38:58.808475+0000","last_undegraded":"2024-03-04T07:38:58.808475+0000","last_fullsized":"2024-03-04T07:38:58.808475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:22.143108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811215+0000","last_change":"2024-03-04T07:37:55.115581+0000","last_active":"2024-03-04T07:38:58.811215+0000","last_peered":"2024-03-04T07:38:58.811215+0000","last_clean":"2024-03-04T07:38:58.811215+0000","last_became_active":"2024-03-04T07:37:55.113387+0000","last_became_peered":"2024-03-04T07:37:55.113387+0000","last_unstale":"2024-03-04T07:38:58.811215+0000","last_undegraded":"2024-03-04T07:38:58.811215+0000","last_fullsized":"2024-03-04T07:38:58.811215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:59.273650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811830+0000","last_change":"2024-03-04T07:37:55.135175+0000","last_active":"2024-03-04T07:38:58.811830+0000","last_peered":"2024-03-04T07:38:58.811830+0000","last_clean":"2024-03-04T07:38:58.811830+0000","last_became_active":"2024-03-04T07:37:55.134787+0000","last_became_peered":"2024-03-04T07:37:55.134787+0000","last_unstale":"2024-03-04T07:38:58.811830+0000","last_undegraded":"2024-03-04T07:38:58.811830+0000","last_fullsized":"2024-03-04T07:38:58.811830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:04.227886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501056+0000","last_change":"2024-03-04T07:37:55.138146+0000","last_active":"2024-03-04T07:39:27.501056+0000","last_peered":"2024-03-04T07:39:27.501056+0000","last_clean":"2024-03-04T07:39:27.501056+0000","last_became_active":"2024-03-04T07:37:55.137806+0000","last_became_peered":"2024-03-04T07:37:55.137806+0000","last_unstale":"2024-03-04T07:39:27.501056+0000","last_undegraded":"2024-03-04T07:39:27.501056+0000","last_fullsized":"2024-03-04T07:39:27.501056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:10.596224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804787+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:38:58.804787+0000","last_peered":"2024-03-04T07:38:58.804787+0000","last_clean":"2024-03-04T07:38:58.804787+0000","last_became_active":"2024-03-04T07:37:55.140670+0000","last_became_peered":"2024-03-04T07:37:55.140670+0000","last_unstale":"2024-03-04T07:38:58.804787+0000","last_undegraded":"2024-03-04T07:38:58.804787+0000","last_fullsized":"2024-03-04T07:38:58.804787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:29:54.439417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808611+0000","last_change":"2024-03-04T07:37:55.115015+0000","last_active":"2024-03-04T07:38:58.808611+0000","last_peered":"2024-03-04T07:38:58.808611+0000","last_clean":"2024-03-04T07:38:58.808611+0000","last_became_active":"2024-03-04T07:37:55.114668+0000","last_became_peered":"2024-03-04T07:37:55.114668+0000","last_unstale":"2024-03-04T07:38:58.808611+0000","last_undegraded":"2024-03-04T07:38:58.808611+0000","last_fullsized":"2024-03-04T07:38:58.808611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:29:03.309234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812327+0000","last_change":"2024-03-04T07:37:55.119514+0000","last_active":"2024-03-04T07:38:58.812327+0000","last_peered":"2024-03-04T07:38:58.812327+0000","last_clean":"2024-03-04T07:38:58.812327+0000","last_became_active":"2024-03-04T07:37:55.119166+0000","last_became_peered":"2024-03-04T07:37:55.119166+0000","last_unstale":"2024-03-04T07:38:58.812327+0000","last_undegraded":"2024-03-04T07:38:58.812327+0000","last_fullsized":"2024-03-04T07:38:58.812327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:33.468311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807059+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:38:58.807059+0000","last_peered":"2024-03-04T07:38:58.807059+0000","last_clean":"2024-03-04T07:38:58.807059+0000","last_became_active":"2024-03-04T07:37:55.140723+0000","last_became_peered":"2024-03-04T07:37:55.140723+0000","last_unstale":"2024-03-04T07:38:58.807059+0000","last_undegraded":"2024-03-04T07:38:58.807059+0000","last_fullsized":"2024-03-04T07:38:58.807059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:29.751190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808349+0000","last_change":"2024-03-04T07:37:55.127627+0000","last_active":"2024-03-04T07:38:58.808349+0000","last_peered":"2024-03-04T07:38:58.808349+0000","last_clean":"2024-03-04T07:38:58.808349+0000","last_became_active":"2024-03-04T07:37:55.127217+0000","last_became_peered":"2024-03-04T07:37:55.127217+0000","last_unstale":"2024-03-04T07:38:58.808349+0000","last_undegraded":"2024-03-04T07:38:58.808349+0000","last_fullsized":"2024-03-04T07:38:58.808349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:47.826970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.806517+0000","last_change":"2024-03-04T07:37:55.129445+0000","last_active":"2024-03-04T07:38:58.806517+0000","last_peered":"2024-03-04T07:38:58.806517+0000","last_clean":"2024-03-04T07:38:58.806517+0000","last_became_active":"2024-03-04T07:37:55.129013+0000","last_became_peered":"2024-03-04T07:37:55.129013+0000","last_unstale":"2024-03-04T07:38:58.806517+0000","last_undegraded":"2024-03-04T07:38:58.806517+0000","last_fullsized":"2024-03-04T07:38:58.806517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:15.578436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.802551+0000","last_change":"2024-03-04T07:37:55.126397+0000","last_active":"2024-03-04T07:38:58.802551+0000","last_peered":"2024-03-04T07:38:58.802551+0000","last_clean":"2024-03-04T07:38:58.802551+0000","last_became_active":"2024-03-04T07:37:55.126092+0000","last_became_peered":"2024-03-04T07:37:55.126092+0000","last_unstale":"2024-03-04T07:38:58.802551+0000","last_undegraded":"2024-03-04T07:38:58.802551+0000","last_fullsized":"2024-03-04T07:38:58.802551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:47.968250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804603+0000","last_change":"2024-03-04T07:37:55.140265+0000","last_active":"2024-03-04T07:38:58.804603+0000","last_peered":"2024-03-04T07:38:58.804603+0000","last_clean":"2024-03-04T07:38:58.804603+0000","last_became_active":"2024-03-04T07:37:55.139768+0000","last_became_peered":"2024-03-04T07:37:55.139768+0000","last_unstale":"2024-03-04T07:38:58.804603+0000","last_undegraded":"2024-03-04T07:38:58.804603+0000","last_fullsized":"2024-03-04T07:38:58.804603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:52.924188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808817+0000","last_change":"2024-03-04T07:37:55.118929+0000","last_active":"2024-03-04T07:38:58.808817+0000","last_peered":"2024-03-04T07:38:58.808817+0000","last_clean":"2024-03-04T07:38:58.808817+0000","last_became_active":"2024-03-04T07:37:55.118549+0000","last_became_peered":"2024-03-04T07:37:55.118549+0000","last_unstale":"2024-03-04T07:38:58.808817+0000","last_undegraded":"2024-03-04T07:38:58.808817+0000","last_fullsized":"2024-03-04T07:38:58.808817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:15.599080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49,"ondisk_log_size":49,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244954,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61844,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":192,"num_read_kb":157,"num_write":75,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":185532,"data_compressed":5550,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":937984,"data_stored":455726,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":1107451,"data_compressed":43363,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":456425,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":456330,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":1046793,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":455843,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":516772,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":455816,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":516806,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":455947,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":455748,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":1046393,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61046,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61080,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61240,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T07:39:30.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T07:39:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:29 smithi152 ceph-mon[37154]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-04T07:39:30.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:29 smithi133 ceph-mon[36505]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-04T07:39:30.498 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:30.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:30 smithi116 ceph-mon[32150]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:39:30.996 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:30 smithi116 ceph-mon[32150]: pgmap v363: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:30 smithi152 ceph-mon[37154]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:39:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:30 smithi152 ceph-mon[37154]: pgmap v363: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:30 smithi133 ceph-mon[36505]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:39:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:30 smithi133 ceph-mon[36505]: pgmap v363: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:32.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:32 smithi116 ceph-mon[32150]: pgmap v364: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-04T07:39:33.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 -- 172.21.15.116:0/1224539610 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec0742f0 msgr2=0x7f98ec0746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:33.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 --2- 172.21.15.116:0/1224539610 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec0742f0 0x7f98ec0746d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009990 tx=0x7f98dc009ca0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 -- 172.21.15.116:0/1224539610 shutdown_connections 2024-03-04T07:39:33.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 --2- 172.21.15.116:0/1224539610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec10eac0 0x7f98ec110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 --2- 172.21.15.116:0/1224539610 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec0742f0 0x7f98ec0746d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 --2- 172.21.15.116:0/1224539610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 0x7f98ec10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 -- 172.21.15.116:0/1224539610 >> 172.21.15.116:0/1224539610 conn(0x7f98ec0791f0 msgr2=0x7f98ec079600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:33.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.946+0000 7f98f1000700 1 -- 172.21.15.116:0/1224539610 shutdown_connections 2024-03-04T07:39:33.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.948+0000 7f98f1000700 1 -- 172.21.15.116:0/1224539610 wait complete. 2024-03-04T07:39:33.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 Processor -- start 2024-03-04T07:39:33.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 -- start start 2024-03-04T07:39:33.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec0742f0 0x7f98ec1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:33.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 0x7f98ec1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:33.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec10eac0 0x7f98ec19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:33.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec114570 con 0x7f98ec074c10 2024-03-04T07:39:33.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec1143f0 con 0x7f98ec10eac0 2024-03-04T07:39:33.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98f1000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec1146f0 con 0x7f98ec0742f0 2024-03-04T07:39:33.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98e1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 0x7f98ec1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:33.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98e9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec0742f0 0x7f98ec1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:33.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.949+0000 7f98ea800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec10eac0 0x7f98ec19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:33.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec0742f0 0x7f98ec1a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:40636/0 (socket says 172.21.15.116:40636) 2024-03-04T07:39:33.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 -- 172.21.15.116:0/2621112248 learned_addr learned my addr 172.21.15.116:0/2621112248 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:33.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e1400700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 0x7f98ec1a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38866/0 (socket says 172.21.15.116:38866) 2024-03-04T07:39:33.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 -- 172.21.15.116:0/2621112248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec10eac0 msgr2=0x7f98ec19a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:33.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec10eac0 0x7f98ec19a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 -- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 msgr2=0x7f98ec1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:33.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 0x7f98ec1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 -- 172.21.15.116:0/2621112248 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec19b030 con 0x7f98ec0742f0 2024-03-04T07:39:33.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e1400700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 0x7f98ec1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:33.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98ea800700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec10eac0 0x7f98ec19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:33.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.950+0000 7f98e9e00700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec0742f0 0x7f98ec1a06d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00bae0 tx=0x7f98dc00bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:33.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.951+0000 7f98e3400700 1 -- 172.21.15.116:0/2621112248 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc026070 con 0x7f98ec0742f0 2024-03-04T07:39:33.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.951+0000 7f98e3400700 1 -- 172.21.15.116:0/2621112248 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98dc003d70 con 0x7f98ec0742f0 2024-03-04T07:39:33.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.951+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec19b2c0 con 0x7f98ec0742f0 2024-03-04T07:39:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.951+0000 7f98e3400700 1 -- 172.21.15.116:0/2621112248 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc021410 con 0x7f98ec0742f0 2024-03-04T07:39:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.951+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec1a7530 con 0x7f98ec0742f0 2024-03-04T07:39:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.952+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec075990 con 0x7f98ec0742f0 2024-03-04T07:39:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.954+0000 7f98e3400700 1 -- 172.21.15.116:0/2621112248 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f98dc029070 con 0x7f98ec0742f0 2024-03-04T07:39:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.955+0000 7f98e3400700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98d00a1e90 0x7f98d00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:33.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.955+0000 7f98e3400700 1 -- 172.21.15.116:0/2621112248 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f98dc0cef40 con 0x7f98ec0742f0 2024-03-04T07:39:33.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.955+0000 7f98e1400700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98d00a1e90 0x7f98d00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:33.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.956+0000 7f98e1400700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98d00a1e90 0x7f98d00a4350 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98ec1107e0 tx=0x7f98d400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:33.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:32.958+0000 7f98e3400700 1 -- 172.21.15.116:0/2621112248 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f98dc013070 con 0x7f98ec0742f0 2024-03-04T07:39:33.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:32 smithi152 ceph-mon[37154]: pgmap v364: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-04T07:39:33.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:32 smithi133 ceph-mon[36505]: pgmap v364: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-04T07:39:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.176+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98ec062500 con 0x7f98d00a1e90 2024-03-04T07:39:33.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.198+0000 7f98e3400700 1 -- 172.21.15.116:0/2621112248 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259116 (secure 0 0 0) 0x7f98ec062500 con 0x7f98d00a1e90 2024-03-04T07:39:33.200 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:33.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.205+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98d00a1e90 msgr2=0x7f98d00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.205+0000 7f98f1000700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98d00a1e90 0x7f98d00a4350 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98ec1107e0 tx=0x7f98d400b040 comp rx=0 tx=0).stop 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.205+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec0742f0 msgr2=0x7f98ec1a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.205+0000 7f98f1000700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec0742f0 0x7f98ec1a06d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00bae0 tx=0x7f98dc00bb10 comp rx=0 tx=0).stop 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 shutdown_connections 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98d00a1e90 0x7f98d00a4350 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98ec10eac0 0x7f98ec19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98ec0742f0 0x7f98ec1a06d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 --2- 172.21.15.116:0/2621112248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98ec074c10 0x7f98ec1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 >> 172.21.15.116:0/2621112248 conn(0x7f98ec0791f0 msgr2=0x7f98ec0ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 shutdown_connections 2024-03-04T07:39:33.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:33.206+0000 7f98f1000700 1 -- 172.21.15.116:0/2621112248 wait complete. 2024-03-04T07:39:33.211 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T07:39:34.216 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-03-04T07:39:31.511590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652446,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":318,"num_read_kb":266,"num_write":260,"num_write_kb":3023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":569,"ondisk_log_size":569,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360592,"kb_used_data":11412,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581232,"statfs":{"total":1151940427776,"available":1151571181568,"internally_reserved":0,"allocated":11685888,"data_stored":7426299,"data_compressed":182424,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004832"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808545+0000","last_change":"2024-03-04T07:37:55.137540+0000","last_active":"2024-03-04T07:38:58.808545+0000","last_peered":"2024-03-04T07:38:58.808545+0000","last_clean":"2024-03-04T07:38:58.808545+0000","last_became_active":"2024-03-04T07:37:55.137183+0000","last_became_peered":"2024-03-04T07:37:55.137183+0000","last_unstale":"2024-03-04T07:38:58.808545+0000","last_undegraded":"2024-03-04T07:38:58.808545+0000","last_fullsized":"2024-03-04T07:38:58.808545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:15.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503448+0000","last_change":"2024-03-04T07:37:55.136049+0000","last_active":"2024-03-04T07:39:27.503448+0000","last_peered":"2024-03-04T07:39:27.503448+0000","last_clean":"2024-03-04T07:39:27.503448+0000","last_became_active":"2024-03-04T07:37:55.132153+0000","last_became_peered":"2024-03-04T07:37:55.132153+0000","last_unstale":"2024-03-04T07:39:27.503448+0000","last_undegraded":"2024-03-04T07:39:27.503448+0000","last_fullsized":"2024-03-04T07:39:27.503448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:52.426892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.507866+0000","last_change":"2024-03-04T07:37:55.133571+0000","last_active":"2024-03-04T07:39:27.507866+0000","last_peered":"2024-03-04T07:39:27.507866+0000","last_clean":"2024-03-04T07:39:27.507866+0000","last_became_active":"2024-03-04T07:37:55.133214+0000","last_became_peered":"2024-03-04T07:37:55.133214+0000","last_unstale":"2024-03-04T07:39:27.507866+0000","last_undegraded":"2024-03-04T07:39:27.507866+0000","last_fullsized":"2024-03-04T07:39:27.507866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:17.924826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.479661+0000","last_change":"2024-03-04T07:37:55.118146+0000","last_active":"2024-03-04T07:39:27.479661+0000","last_peered":"2024-03-04T07:39:27.479661+0000","last_clean":"2024-03-04T07:39:27.479661+0000","last_became_active":"2024-03-04T07:37:55.117687+0000","last_became_peered":"2024-03-04T07:37:55.117687+0000","last_unstale":"2024-03-04T07:39:27.479661+0000","last_undegraded":"2024-03-04T07:39:27.479661+0000","last_fullsized":"2024-03-04T07:39:27.479661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:34:35.343400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.499117+0000","last_change":"2024-03-04T07:37:55.135943+0000","last_active":"2024-03-04T07:39:27.499117+0000","last_peered":"2024-03-04T07:39:27.499117+0000","last_clean":"2024-03-04T07:39:27.499117+0000","last_became_active":"2024-03-04T07:37:55.135549+0000","last_became_peered":"2024-03-04T07:37:55.135549+0000","last_unstale":"2024-03-04T07:39:27.499117+0000","last_undegraded":"2024-03-04T07:39:27.499117+0000","last_fullsized":"2024-03-04T07:39:27.499117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:35:53.953771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807386+0000","last_change":"2024-03-04T07:37:55.116949+0000","last_active":"2024-03-04T07:38:58.807386+0000","last_peered":"2024-03-04T07:38:58.807386+0000","last_clean":"2024-03-04T07:38:58.807386+0000","last_became_active":"2024-03-04T07:37:55.116620+0000","last_became_peered":"2024-03-04T07:37:55.116620+0000","last_unstale":"2024-03-04T07:38:58.807386+0000","last_undegraded":"2024-03-04T07:38:58.807386+0000","last_fullsized":"2024-03-04T07:38:58.807386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:39.655217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489650+0000","last_change":"2024-03-04T07:37:55.100975+0000","last_active":"2024-03-04T07:39:27.489650+0000","last_peered":"2024-03-04T07:39:27.489650+0000","last_clean":"2024-03-04T07:39:27.489650+0000","last_became_active":"2024-03-04T07:37:55.100636+0000","last_became_peered":"2024-03-04T07:37:55.100636+0000","last_unstale":"2024-03-04T07:39:27.489650+0000","last_undegraded":"2024-03-04T07:39:27.489650+0000","last_fullsized":"2024-03-04T07:39:27.489650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:29.593763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523017+0000","last_change":"2024-03-04T07:37:55.131895+0000","last_active":"2024-03-04T07:39:27.523017+0000","last_peered":"2024-03-04T07:39:27.523017+0000","last_clean":"2024-03-04T07:39:27.523017+0000","last_became_active":"2024-03-04T07:37:55.129682+0000","last_became_peered":"2024-03-04T07:37:55.129682+0000","last_unstale":"2024-03-04T07:39:27.523017+0000","last_undegraded":"2024-03-04T07:39:27.523017+0000","last_fullsized":"2024-03-04T07:39:27.523017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:54.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489383+0000","last_change":"2024-03-04T07:37:55.114475+0000","last_active":"2024-03-04T07:39:27.489383+0000","last_peered":"2024-03-04T07:39:27.489383+0000","last_clean":"2024-03-04T07:39:27.489383+0000","last_became_active":"2024-03-04T07:37:55.114068+0000","last_became_peered":"2024-03-04T07:37:55.114068+0000","last_unstale":"2024-03-04T07:39:27.489383+0000","last_undegraded":"2024-03-04T07:39:27.489383+0000","last_fullsized":"2024-03-04T07:39:27.489383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:21.466143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807988+0000","last_change":"2024-03-04T07:37:55.115660+0000","last_active":"2024-03-04T07:38:58.807988+0000","last_peered":"2024-03-04T07:38:58.807988+0000","last_clean":"2024-03-04T07:38:58.807988+0000","last_became_active":"2024-03-04T07:37:55.115324+0000","last_became_peered":"2024-03-04T07:37:55.115324+0000","last_unstale":"2024-03-04T07:38:58.807988+0000","last_undegraded":"2024-03-04T07:38:58.807988+0000","last_fullsized":"2024-03-04T07:38:58.807988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:42.119728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489174+0000","last_change":"2024-03-04T07:37:55.119834+0000","last_active":"2024-03-04T07:39:27.489174+0000","last_peered":"2024-03-04T07:39:27.489174+0000","last_clean":"2024-03-04T07:39:27.489174+0000","last_became_active":"2024-03-04T07:37:55.119467+0000","last_became_peered":"2024-03-04T07:37:55.119467+0000","last_unstale":"2024-03-04T07:39:27.489174+0000","last_undegraded":"2024-03-04T07:39:27.489174+0000","last_fullsized":"2024-03-04T07:39:27.489174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:49:19.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.479387+0000","last_change":"2024-03-04T07:37:55.118154+0000","last_active":"2024-03-04T07:39:27.479387+0000","last_peered":"2024-03-04T07:39:27.479387+0000","last_clean":"2024-03-04T07:39:27.479387+0000","last_became_active":"2024-03-04T07:37:55.117809+0000","last_became_peered":"2024-03-04T07:37:55.117809+0000","last_unstale":"2024-03-04T07:39:27.479387+0000","last_undegraded":"2024-03-04T07:39:27.479387+0000","last_fullsized":"2024-03-04T07:39:27.479387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:50.195946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502728+0000","last_change":"2024-03-04T07:37:55.123325+0000","last_active":"2024-03-04T07:39:27.502728+0000","last_peered":"2024-03-04T07:39:27.502728+0000","last_clean":"2024-03-04T07:39:27.502728+0000","last_became_active":"2024-03-04T07:37:55.122848+0000","last_became_peered":"2024-03-04T07:37:55.122848+0000","last_unstale":"2024-03-04T07:39:27.502728+0000","last_undegraded":"2024-03-04T07:39:27.502728+0000","last_fullsized":"2024-03-04T07:39:27.502728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:18.842011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502983+0000","last_change":"2024-03-04T07:37:55.132447+0000","last_active":"2024-03-04T07:39:27.502983+0000","last_peered":"2024-03-04T07:39:27.502983+0000","last_clean":"2024-03-04T07:39:27.502983+0000","last_became_active":"2024-03-04T07:37:55.131512+0000","last_became_peered":"2024-03-04T07:37:55.131512+0000","last_unstale":"2024-03-04T07:39:27.502983+0000","last_undegraded":"2024-03-04T07:39:27.502983+0000","last_fullsized":"2024-03-04T07:39:27.502983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:23.960029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504228+0000","last_change":"2024-03-04T07:37:55.131715+0000","last_active":"2024-03-04T07:39:27.504228+0000","last_peered":"2024-03-04T07:39:27.504228+0000","last_clean":"2024-03-04T07:39:27.504228+0000","last_became_active":"2024-03-04T07:37:55.131326+0000","last_became_peered":"2024-03-04T07:37:55.131326+0000","last_unstale":"2024-03-04T07:39:27.504228+0000","last_undegraded":"2024-03-04T07:39:27.504228+0000","last_fullsized":"2024-03-04T07:39:27.504228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:56.231578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502226+0000","last_change":"2024-03-04T07:37:55.137395+0000","last_active":"2024-03-04T07:39:27.502226+0000","last_peered":"2024-03-04T07:39:27.502226+0000","last_clean":"2024-03-04T07:39:27.502226+0000","last_became_active":"2024-03-04T07:37:55.137054+0000","last_became_peered":"2024-03-04T07:37:55.137054+0000","last_unstale":"2024-03-04T07:39:27.502226+0000","last_undegraded":"2024-03-04T07:39:27.502226+0000","last_fullsized":"2024-03-04T07:39:27.502226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:19.222481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808821+0000","last_change":"2024-03-04T07:37:55.137414+0000","last_active":"2024-03-04T07:38:58.808821+0000","last_peered":"2024-03-04T07:38:58.808821+0000","last_clean":"2024-03-04T07:38:58.808821+0000","last_became_active":"2024-03-04T07:37:55.137157+0000","last_became_peered":"2024-03-04T07:37:55.137157+0000","last_unstale":"2024-03-04T07:38:58.808821+0000","last_undegraded":"2024-03-04T07:38:58.808821+0000","last_fullsized":"2024-03-04T07:38:58.808821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:36.593491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501722+0000","last_change":"2024-03-04T07:37:55.121960+0000","last_active":"2024-03-04T07:39:27.501722+0000","last_peered":"2024-03-04T07:39:27.501722+0000","last_clean":"2024-03-04T07:39:27.501722+0000","last_became_active":"2024-03-04T07:37:55.115543+0000","last_became_peered":"2024-03-04T07:37:55.115543+0000","last_unstale":"2024-03-04T07:39:27.501722+0000","last_undegraded":"2024-03-04T07:39:27.501722+0000","last_fullsized":"2024-03-04T07:39:27.501722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:36.320475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812632+0000","last_change":"2024-03-04T07:37:55.128908+0000","last_active":"2024-03-04T07:38:58.812632+0000","last_peered":"2024-03-04T07:38:58.812632+0000","last_clean":"2024-03-04T07:38:58.812632+0000","last_became_active":"2024-03-04T07:37:55.128500+0000","last_became_peered":"2024-03-04T07:37:55.128500+0000","last_unstale":"2024-03-04T07:38:58.812632+0000","last_undegraded":"2024-03-04T07:38:58.812632+0000","last_fullsized":"2024-03-04T07:38:58.812632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:10.598424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804966+0000","last_change":"2024-03-04T07:37:54.121610+0000","last_active":"2024-03-04T07:38:58.804966+0000","last_peered":"2024-03-04T07:38:58.804966+0000","last_clean":"2024-03-04T07:38:58.804966+0000","last_became_active":"2024-03-04T07:37:54.121150+0000","last_became_peered":"2024-03-04T07:37:54.121150+0000","last_unstale":"2024-03-04T07:38:58.804966+0000","last_undegraded":"2024-03-04T07:38:58.804966+0000","last_fullsized":"2024-03-04T07:38:58.804966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:05:03.965626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.517166+0000","last_change":"2024-03-04T07:37:55.138904+0000","last_active":"2024-03-04T07:39:27.517166+0000","last_peered":"2024-03-04T07:39:27.517166+0000","last_clean":"2024-03-04T07:39:27.517166+0000","last_became_active":"2024-03-04T07:37:55.138376+0000","last_became_peered":"2024-03-04T07:37:55.138376+0000","last_unstale":"2024-03-04T07:39:27.517166+0000","last_undegraded":"2024-03-04T07:39:27.517166+0000","last_fullsized":"2024-03-04T07:39:27.517166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:28.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807796+0000","last_change":"2024-03-04T07:37:54.139218+0000","last_active":"2024-03-04T07:38:58.807796+0000","last_peered":"2024-03-04T07:38:58.807796+0000","last_clean":"2024-03-04T07:38:58.807796+0000","last_became_active":"2024-03-04T07:37:54.138893+0000","last_became_peered":"2024-03-04T07:37:54.138893+0000","last_unstale":"2024-03-04T07:38:58.807796+0000","last_undegraded":"2024-03-04T07:38:58.807796+0000","last_fullsized":"2024-03-04T07:38:58.807796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:01.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.508413+0000","last_change":"2024-03-04T07:37:55.135953+0000","last_active":"2024-03-04T07:39:27.508413+0000","last_peered":"2024-03-04T07:39:27.508413+0000","last_clean":"2024-03-04T07:39:27.508413+0000","last_became_active":"2024-03-04T07:37:55.132880+0000","last_became_peered":"2024-03-04T07:37:55.132880+0000","last_unstale":"2024-03-04T07:39:27.508413+0000","last_undegraded":"2024-03-04T07:39:27.508413+0000","last_fullsized":"2024-03-04T07:39:27.508413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:03.848428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808582+0000","last_change":"2024-03-04T07:37:54.122493+0000","last_active":"2024-03-04T07:38:58.808582+0000","last_peered":"2024-03-04T07:38:58.808582+0000","last_clean":"2024-03-04T07:38:58.808582+0000","last_became_active":"2024-03-04T07:37:54.122058+0000","last_became_peered":"2024-03-04T07:37:54.122058+0000","last_unstale":"2024-03-04T07:38:58.808582+0000","last_undegraded":"2024-03-04T07:38:58.808582+0000","last_fullsized":"2024-03-04T07:38:58.808582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:02.516175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.491455+0000","last_change":"2024-03-04T07:37:55.136155+0000","last_active":"2024-03-04T07:39:27.491455+0000","last_peered":"2024-03-04T07:39:27.491455+0000","last_clean":"2024-03-04T07:39:27.491455+0000","last_became_active":"2024-03-04T07:37:55.135796+0000","last_became_peered":"2024-03-04T07:37:55.135796+0000","last_unstale":"2024-03-04T07:39:27.491455+0000","last_undegraded":"2024-03-04T07:39:27.491455+0000","last_fullsized":"2024-03-04T07:39:27.491455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:37.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808437+0000","last_change":"2024-03-04T07:37:54.118780+0000","last_active":"2024-03-04T07:38:58.808437+0000","last_peered":"2024-03-04T07:38:58.808437+0000","last_clean":"2024-03-04T07:38:58.808437+0000","last_became_active":"2024-03-04T07:37:54.118448+0000","last_became_peered":"2024-03-04T07:37:54.118448+0000","last_unstale":"2024-03-04T07:38:58.808437+0000","last_undegraded":"2024-03-04T07:38:58.808437+0000","last_fullsized":"2024-03-04T07:38:58.808437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:25.547046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501807+0000","last_change":"2024-03-04T07:38:59.021729+0000","last_active":"2024-03-04T07:39:27.501807+0000","last_peered":"2024-03-04T07:39:27.501807+0000","last_clean":"2024-03-04T07:39:27.501807+0000","last_became_active":"2024-03-04T07:38:59.021127+0000","last_became_peered":"2024-03-04T07:38:59.021127+0000","last_unstale":"2024-03-04T07:39:27.501807+0000","last_undegraded":"2024-03-04T07:39:27.501807+0000","last_fullsized":"2024-03-04T07:39:27.501807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:08:07.825530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807825+0000","last_change":"2024-03-04T07:37:54.122241+0000","last_active":"2024-03-04T07:38:58.807825+0000","last_peered":"2024-03-04T07:38:58.807825+0000","last_clean":"2024-03-04T07:38:58.807825+0000","last_became_active":"2024-03-04T07:37:54.120990+0000","last_became_peered":"2024-03-04T07:37:54.120990+0000","last_unstale":"2024-03-04T07:38:58.807825+0000","last_undegraded":"2024-03-04T07:38:58.807825+0000","last_fullsized":"2024-03-04T07:38:58.807825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:03.233754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501628+0000","last_change":"2024-03-04T07:37:55.121692+0000","last_active":"2024-03-04T07:39:27.501628+0000","last_peered":"2024-03-04T07:39:27.501628+0000","last_clean":"2024-03-04T07:39:27.501628+0000","last_became_active":"2024-03-04T07:37:55.121345+0000","last_became_peered":"2024-03-04T07:37:55.121345+0000","last_unstale":"2024-03-04T07:39:27.501628+0000","last_undegraded":"2024-03-04T07:39:27.501628+0000","last_fullsized":"2024-03-04T07:39:27.501628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:52:42.045265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807382+0000","last_change":"2024-03-04T07:37:54.133489+0000","last_active":"2024-03-04T07:38:58.807382+0000","last_peered":"2024-03-04T07:38:58.807382+0000","last_clean":"2024-03-04T07:38:58.807382+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807382+0000","last_undegraded":"2024-03-04T07:38:58.807382+0000","last_fullsized":"2024-03-04T07:38:58.807382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:49:03.600907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509783+0000","last_change":"2024-03-04T07:37:55.128278+0000","last_active":"2024-03-04T07:39:27.509783+0000","last_peered":"2024-03-04T07:39:27.509783+0000","last_clean":"2024-03-04T07:39:27.509783+0000","last_became_active":"2024-03-04T07:37:55.127913+0000","last_became_peered":"2024-03-04T07:37:55.127913+0000","last_unstale":"2024-03-04T07:39:27.509783+0000","last_undegraded":"2024-03-04T07:39:27.509783+0000","last_fullsized":"2024-03-04T07:39:27.509783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:54:18.623855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803806+0000","last_change":"2024-03-04T07:37:54.132195+0000","last_active":"2024-03-04T07:38:58.803806+0000","last_peered":"2024-03-04T07:38:58.803806+0000","last_clean":"2024-03-04T07:38:58.803806+0000","last_became_active":"2024-03-04T07:37:54.131830+0000","last_became_peered":"2024-03-04T07:37:54.131830+0000","last_unstale":"2024-03-04T07:38:58.803806+0000","last_undegraded":"2024-03-04T07:38:58.803806+0000","last_fullsized":"2024-03-04T07:38:58.803806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.744342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522806+0000","last_change":"2024-03-04T07:37:55.138281+0000","last_active":"2024-03-04T07:39:27.522806+0000","last_peered":"2024-03-04T07:39:27.522806+0000","last_clean":"2024-03-04T07:39:27.522806+0000","last_became_active":"2024-03-04T07:37:55.137919+0000","last_became_peered":"2024-03-04T07:37:55.137919+0000","last_unstale":"2024-03-04T07:39:27.522806+0000","last_undegraded":"2024-03-04T07:39:27.522806+0000","last_fullsized":"2024-03-04T07:39:27.522806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:53:06.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807552+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:38:58.807552+0000","last_peered":"2024-03-04T07:38:58.807552+0000","last_clean":"2024-03-04T07:38:58.807552+0000","last_became_active":"2024-03-04T07:37:54.121503+0000","last_became_peered":"2024-03-04T07:37:54.121503+0000","last_unstale":"2024-03-04T07:38:58.807552+0000","last_undegraded":"2024-03-04T07:38:58.807552+0000","last_fullsized":"2024-03-04T07:38:58.807552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:03:07.531246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519776+0000","last_change":"2024-03-04T07:37:55.126924+0000","last_active":"2024-03-04T07:39:27.519776+0000","last_peered":"2024-03-04T07:39:27.519776+0000","last_clean":"2024-03-04T07:39:27.519776+0000","last_became_active":"2024-03-04T07:37:55.126266+0000","last_became_peered":"2024-03-04T07:37:55.126266+0000","last_unstale":"2024-03-04T07:39:27.519776+0000","last_undegraded":"2024-03-04T07:39:27.519776+0000","last_fullsized":"2024-03-04T07:39:27.519776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:01.603430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807869+0000","last_change":"2024-03-04T07:37:54.119864+0000","last_active":"2024-03-04T07:38:58.807869+0000","last_peered":"2024-03-04T07:38:58.807869+0000","last_clean":"2024-03-04T07:38:58.807869+0000","last_became_active":"2024-03-04T07:37:54.119515+0000","last_became_peered":"2024-03-04T07:37:54.119515+0000","last_unstale":"2024-03-04T07:38:58.807869+0000","last_undegraded":"2024-03-04T07:38:58.807869+0000","last_fullsized":"2024-03-04T07:38:58.807869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:10.422458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.490780+0000","last_change":"2024-03-04T07:37:55.120135+0000","last_active":"2024-03-04T07:39:27.490780+0000","last_peered":"2024-03-04T07:39:27.490780+0000","last_clean":"2024-03-04T07:39:27.490780+0000","last_became_active":"2024-03-04T07:37:55.119670+0000","last_became_peered":"2024-03-04T07:37:55.119670+0000","last_unstale":"2024-03-04T07:39:27.490780+0000","last_undegraded":"2024-03-04T07:39:27.490780+0000","last_fullsized":"2024-03-04T07:39:27.490780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:09.688447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807178+0000","last_change":"2024-03-04T07:37:54.120222+0000","last_active":"2024-03-04T07:38:58.807178+0000","last_peered":"2024-03-04T07:38:58.807178+0000","last_clean":"2024-03-04T07:38:58.807178+0000","last_became_active":"2024-03-04T07:37:54.119802+0000","last_became_peered":"2024-03-04T07:37:54.119802+0000","last_unstale":"2024-03-04T07:38:58.807178+0000","last_undegraded":"2024-03-04T07:38:58.807178+0000","last_fullsized":"2024-03-04T07:38:58.807178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:12:40.811870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.508249+0000","last_change":"2024-03-04T07:37:55.136371+0000","last_active":"2024-03-04T07:39:27.508249+0000","last_peered":"2024-03-04T07:39:27.508249+0000","last_clean":"2024-03-04T07:39:27.508249+0000","last_became_active":"2024-03-04T07:37:55.132587+0000","last_became_peered":"2024-03-04T07:37:55.132587+0000","last_unstale":"2024-03-04T07:39:27.508249+0000","last_undegraded":"2024-03-04T07:39:27.508249+0000","last_fullsized":"2024-03-04T07:39:27.508249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:24.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.809008+0000","last_change":"2024-03-04T07:37:54.123041+0000","last_active":"2024-03-04T07:38:58.809008+0000","last_peered":"2024-03-04T07:38:58.809008+0000","last_clean":"2024-03-04T07:38:58.809008+0000","last_became_active":"2024-03-04T07:37:54.122723+0000","last_became_peered":"2024-03-04T07:37:54.122723+0000","last_unstale":"2024-03-04T07:38:58.809008+0000","last_undegraded":"2024-03-04T07:38:58.809008+0000","last_fullsized":"2024-03-04T07:38:58.809008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:06:30.845376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488310+0000","last_change":"2024-03-04T07:37:55.115387+0000","last_active":"2024-03-04T07:39:27.488310+0000","last_peered":"2024-03-04T07:39:27.488310+0000","last_clean":"2024-03-04T07:39:27.488310+0000","last_became_active":"2024-03-04T07:37:55.114947+0000","last_became_peered":"2024-03-04T07:37:55.114947+0000","last_unstale":"2024-03-04T07:39:27.488310+0000","last_undegraded":"2024-03-04T07:39:27.488310+0000","last_fullsized":"2024-03-04T07:39:27.488310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:09:11.840989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'26","reported_seq":233,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:26.951193+0000","last_change":"2024-03-04T07:37:54.117767+0000","last_active":"2024-03-04T07:39:26.951193+0000","last_peered":"2024-03-04T07:39:26.951193+0000","last_clean":"2024-03-04T07:39:26.951193+0000","last_became_active":"2024-03-04T07:37:54.117371+0000","last_became_peered":"2024-03-04T07:37:54.117371+0000","last_unstale":"2024-03-04T07:39:26.951193+0000","last_undegraded":"2024-03-04T07:39:26.951193+0000","last_fullsized":"2024-03-04T07:39:26.951193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:08:32.164636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":188,"num_read_kb":153,"num_write":26,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488252+0000","last_change":"2024-03-04T07:37:55.112707+0000","last_active":"2024-03-04T07:39:27.488252+0000","last_peered":"2024-03-04T07:39:27.488252+0000","last_clean":"2024-03-04T07:39:27.488252+0000","last_became_active":"2024-03-04T07:37:55.112327+0000","last_became_peered":"2024-03-04T07:37:55.112327+0000","last_unstale":"2024-03-04T07:39:27.488252+0000","last_undegraded":"2024-03-04T07:39:27.488252+0000","last_fullsized":"2024-03-04T07:39:27.488252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:03.901687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811717+0000","last_change":"2024-03-04T07:37:54.125286+0000","last_active":"2024-03-04T07:38:58.811717+0000","last_peered":"2024-03-04T07:38:58.811717+0000","last_clean":"2024-03-04T07:38:58.811717+0000","last_became_active":"2024-03-04T07:37:54.124966+0000","last_became_peered":"2024-03-04T07:37:54.124966+0000","last_unstale":"2024-03-04T07:38:58.811717+0000","last_undegraded":"2024-03-04T07:38:58.811717+0000","last_fullsized":"2024-03-04T07:38:58.811717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:19.880681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523997+0000","last_change":"2024-03-04T07:37:55.137952+0000","last_active":"2024-03-04T07:39:27.523997+0000","last_peered":"2024-03-04T07:39:27.523997+0000","last_clean":"2024-03-04T07:39:27.523997+0000","last_became_active":"2024-03-04T07:37:55.137583+0000","last_became_peered":"2024-03-04T07:37:55.137583+0000","last_unstale":"2024-03-04T07:39:27.523997+0000","last_undegraded":"2024-03-04T07:39:27.523997+0000","last_fullsized":"2024-03-04T07:39:27.523997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:30.145453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807526+0000","last_change":"2024-03-04T07:37:54.134733+0000","last_active":"2024-03-04T07:38:58.807526+0000","last_peered":"2024-03-04T07:38:58.807526+0000","last_clean":"2024-03-04T07:38:58.807526+0000","last_became_active":"2024-03-04T07:37:54.134400+0000","last_became_peered":"2024-03-04T07:37:54.134400+0000","last_unstale":"2024-03-04T07:38:58.807526+0000","last_undegraded":"2024-03-04T07:38:58.807526+0000","last_fullsized":"2024-03-04T07:38:58.807526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:24.192479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503753+0000","last_change":"2024-03-04T07:37:55.124991+0000","last_active":"2024-03-04T07:39:27.503753+0000","last_peered":"2024-03-04T07:39:27.503753+0000","last_clean":"2024-03-04T07:39:27.503753+0000","last_became_active":"2024-03-04T07:37:55.124621+0000","last_became_peered":"2024-03-04T07:37:55.124621+0000","last_unstale":"2024-03-04T07:39:27.503753+0000","last_undegraded":"2024-03-04T07:39:27.503753+0000","last_fullsized":"2024-03-04T07:39:27.503753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:05.252145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.020968+0000","last_change":"2024-03-04T07:37:54.118743+0000","last_active":"2024-03-04T07:38:59.020968+0000","last_peered":"2024-03-04T07:38:59.020968+0000","last_clean":"2024-03-04T07:38:59.020968+0000","last_became_active":"2024-03-04T07:37:54.118386+0000","last_became_peered":"2024-03-04T07:37:54.118386+0000","last_unstale":"2024-03-04T07:38:59.020968+0000","last_undegraded":"2024-03-04T07:38:59.020968+0000","last_fullsized":"2024-03-04T07:38:59.020968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:27.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.473432+0000","last_change":"2024-03-04T07:37:55.115431+0000","last_active":"2024-03-04T07:39:27.473432+0000","last_peered":"2024-03-04T07:39:27.473432+0000","last_clean":"2024-03-04T07:39:27.473432+0000","last_became_active":"2024-03-04T07:37:55.115064+0000","last_became_peered":"2024-03-04T07:37:55.115064+0000","last_unstale":"2024-03-04T07:39:27.473432+0000","last_undegraded":"2024-03-04T07:39:27.473432+0000","last_fullsized":"2024-03-04T07:39:27.473432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:46:57.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.028943+0000","last_change":"2024-03-04T07:38:59.028943+0000","last_active":"2024-03-04T07:38:59.028943+0000","last_peered":"2024-03-04T07:38:59.028943+0000","last_clean":"2024-03-04T07:38:59.028943+0000","last_became_active":"2024-03-04T07:38:59.022232+0000","last_became_peered":"2024-03-04T07:38:59.022232+0000","last_unstale":"2024-03-04T07:38:59.028943+0000","last_undegraded":"2024-03-04T07:38:59.028943+0000","last_fullsized":"2024-03-04T07:38:59.028943+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:49.281601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.485250+0000","last_change":"2024-03-04T07:37:55.114882+0000","last_active":"2024-03-04T07:39:27.485250+0000","last_peered":"2024-03-04T07:39:27.485250+0000","last_clean":"2024-03-04T07:39:27.485250+0000","last_became_active":"2024-03-04T07:37:55.113139+0000","last_became_peered":"2024-03-04T07:37:55.113139+0000","last_unstale":"2024-03-04T07:39:27.485250+0000","last_undegraded":"2024-03-04T07:39:27.485250+0000","last_fullsized":"2024-03-04T07:39:27.485250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:28.133710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808725+0000","last_change":"2024-03-04T07:37:54.119350+0000","last_active":"2024-03-04T07:38:58.808725+0000","last_peered":"2024-03-04T07:38:58.808725+0000","last_clean":"2024-03-04T07:38:58.808725+0000","last_became_active":"2024-03-04T07:37:54.119017+0000","last_became_peered":"2024-03-04T07:37:54.119017+0000","last_unstale":"2024-03-04T07:38:58.808725+0000","last_undegraded":"2024-03-04T07:38:58.808725+0000","last_fullsized":"2024-03-04T07:38:58.808725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:00.503583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812598+0000","last_change":"2024-03-04T07:38:58.812598+0000","last_active":"2024-03-04T07:38:58.812598+0000","last_peered":"2024-03-04T07:38:58.812598+0000","last_clean":"2024-03-04T07:38:58.812598+0000","last_became_active":"2024-03-04T07:38:58.812187+0000","last_became_peered":"2024-03-04T07:38:58.812187+0000","last_unstale":"2024-03-04T07:38:58.812598+0000","last_undegraded":"2024-03-04T07:38:58.812598+0000","last_fullsized":"2024-03-04T07:38:58.812598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:43.753007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804378+0000","last_change":"2024-03-04T07:37:54.125548+0000","last_active":"2024-03-04T07:38:58.804378+0000","last_peered":"2024-03-04T07:38:58.804378+0000","last_clean":"2024-03-04T07:38:58.804378+0000","last_became_active":"2024-03-04T07:37:54.125127+0000","last_became_peered":"2024-03-04T07:37:54.125127+0000","last_unstale":"2024-03-04T07:38:58.804378+0000","last_undegraded":"2024-03-04T07:38:58.804378+0000","last_fullsized":"2024-03-04T07:38:58.804378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:26.256247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808153+0000","last_change":"2024-03-04T07:37:55.128542+0000","last_active":"2024-03-04T07:38:58.808153+0000","last_peered":"2024-03-04T07:38:58.808153+0000","last_clean":"2024-03-04T07:38:58.808153+0000","last_became_active":"2024-03-04T07:37:55.128239+0000","last_became_peered":"2024-03-04T07:37:55.128239+0000","last_unstale":"2024-03-04T07:38:58.808153+0000","last_undegraded":"2024-03-04T07:38:58.808153+0000","last_fullsized":"2024-03-04T07:38:58.808153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:17.078095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807344+0000","last_change":"2024-03-04T07:37:54.133672+0000","last_active":"2024-03-04T07:38:58.807344+0000","last_peered":"2024-03-04T07:38:58.807344+0000","last_clean":"2024-03-04T07:38:58.807344+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807344+0000","last_undegraded":"2024-03-04T07:38:58.807344+0000","last_fullsized":"2024-03-04T07:38:58.807344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:52.345158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.815766+0000","last_change":"2024-03-04T07:38:58.815766+0000","last_active":"2024-03-04T07:38:58.815766+0000","last_peered":"2024-03-04T07:38:58.815766+0000","last_clean":"2024-03-04T07:38:58.815766+0000","last_became_active":"2024-03-04T07:38:58.815387+0000","last_became_peered":"2024-03-04T07:38:58.815387+0000","last_unstale":"2024-03-04T07:38:58.815766+0000","last_undegraded":"2024-03-04T07:38:58.815766+0000","last_fullsized":"2024-03-04T07:38:58.815766+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:21.582879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487963+0000","last_change":"2024-03-04T07:38:58.814581+0000","last_active":"2024-03-04T07:39:27.487963+0000","last_peered":"2024-03-04T07:39:27.487963+0000","last_clean":"2024-03-04T07:39:27.487963+0000","last_became_active":"2024-03-04T07:38:58.814186+0000","last_became_peered":"2024-03-04T07:38:58.814186+0000","last_unstale":"2024-03-04T07:39:27.487963+0000","last_undegraded":"2024-03-04T07:39:27.487963+0000","last_fullsized":"2024-03-04T07:39:27.487963+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:03:33.824292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803718+0000","last_change":"2024-03-04T07:37:54.139335+0000","last_active":"2024-03-04T07:38:58.803718+0000","last_peered":"2024-03-04T07:38:58.803718+0000","last_clean":"2024-03-04T07:38:58.803718+0000","last_became_active":"2024-03-04T07:37:54.138969+0000","last_became_peered":"2024-03-04T07:37:54.138969+0000","last_unstale":"2024-03-04T07:38:58.803718+0000","last_undegraded":"2024-03-04T07:38:58.803718+0000","last_fullsized":"2024-03-04T07:38:58.803718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:41:15.204738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523670+0000","last_change":"2024-03-04T07:38:58.810554+0000","last_active":"2024-03-04T07:39:27.523670+0000","last_peered":"2024-03-04T07:39:27.523670+0000","last_clean":"2024-03-04T07:39:27.523670+0000","last_became_active":"2024-03-04T07:38:58.809978+0000","last_became_peered":"2024-03-04T07:38:58.809978+0000","last_unstale":"2024-03-04T07:39:27.523670+0000","last_undegraded":"2024-03-04T07:39:27.523670+0000","last_fullsized":"2024-03-04T07:39:27.523670+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:33.417630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.020981+0000","last_change":"2024-03-04T07:37:54.119906+0000","last_active":"2024-03-04T07:38:59.020981+0000","last_peered":"2024-03-04T07:38:59.020981+0000","last_clean":"2024-03-04T07:38:59.020981+0000","last_became_active":"2024-03-04T07:37:54.119592+0000","last_became_peered":"2024-03-04T07:37:54.119592+0000","last_unstale":"2024-03-04T07:38:59.020981+0000","last_undegraded":"2024-03-04T07:38:59.020981+0000","last_fullsized":"2024-03-04T07:38:59.020981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:29.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":343,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807128+0000","last_change":"2024-03-04T07:36:35.278528+0000","last_active":"2024-03-04T07:38:58.807128+0000","last_peered":"2024-03-04T07:38:58.807128+0000","last_clean":"2024-03-04T07:38:58.807128+0000","last_became_active":"2024-03-04T07:36:35.277953+0000","last_became_peered":"2024-03-04T07:36:35.277953+0000","last_unstale":"2024-03-04T07:38:58.807128+0000","last_undegraded":"2024-03-04T07:38:58.807128+0000","last_fullsized":"2024-03-04T07:38:58.807128+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_clean_scrub_stamp":"2024-03-04T07:32:42.283525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:58:47.223527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.496228+0000","last_change":"2024-03-04T07:37:55.124204+0000","last_active":"2024-03-04T07:39:27.496228+0000","last_peered":"2024-03-04T07:39:27.496228+0000","last_clean":"2024-03-04T07:39:27.496228+0000","last_became_active":"2024-03-04T07:37:55.123863+0000","last_became_peered":"2024-03-04T07:37:55.123863+0000","last_unstale":"2024-03-04T07:39:27.496228+0000","last_undegraded":"2024-03-04T07:39:27.496228+0000","last_fullsized":"2024-03-04T07:39:27.496228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:33.830367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807200+0000","last_change":"2024-03-04T07:37:54.121538+0000","last_active":"2024-03-04T07:38:58.807200+0000","last_peered":"2024-03-04T07:38:58.807200+0000","last_clean":"2024-03-04T07:38:58.807200+0000","last_became_active":"2024-03-04T07:37:54.120939+0000","last_became_peered":"2024-03-04T07:37:54.120939+0000","last_unstale":"2024-03-04T07:38:58.807200+0000","last_undegraded":"2024-03-04T07:38:58.807200+0000","last_fullsized":"2024-03-04T07:38:58.807200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:29.597247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808482+0000","last_change":"2024-03-04T07:37:54.128180+0000","last_active":"2024-03-04T07:38:58.808482+0000","last_peered":"2024-03-04T07:38:58.808482+0000","last_clean":"2024-03-04T07:38:58.808482+0000","last_became_active":"2024-03-04T07:37:54.127800+0000","last_became_peered":"2024-03-04T07:37:54.127800+0000","last_unstale":"2024-03-04T07:38:58.808482+0000","last_undegraded":"2024-03-04T07:38:58.808482+0000","last_fullsized":"2024-03-04T07:38:58.808482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:54.046873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.497328+0000","last_change":"2024-03-04T07:37:55.133482+0000","last_active":"2024-03-04T07:39:27.497328+0000","last_peered":"2024-03-04T07:39:27.497328+0000","last_clean":"2024-03-04T07:39:27.497328+0000","last_became_active":"2024-03-04T07:37:55.133137+0000","last_became_peered":"2024-03-04T07:37:55.133137+0000","last_unstale":"2024-03-04T07:39:27.497328+0000","last_undegraded":"2024-03-04T07:39:27.497328+0000","last_fullsized":"2024-03-04T07:39:27.497328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:53.509564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805155+0000","last_change":"2024-03-04T07:37:54.134294+0000","last_active":"2024-03-04T07:38:58.805155+0000","last_peered":"2024-03-04T07:38:58.805155+0000","last_clean":"2024-03-04T07:38:58.805155+0000","last_became_active":"2024-03-04T07:37:54.133677+0000","last_became_peered":"2024-03-04T07:37:54.133677+0000","last_unstale":"2024-03-04T07:38:58.805155+0000","last_undegraded":"2024-03-04T07:38:58.805155+0000","last_fullsized":"2024-03-04T07:38:58.805155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:57:54.082419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522061+0000","last_change":"2024-03-04T07:37:55.138305+0000","last_active":"2024-03-04T07:39:27.522061+0000","last_peered":"2024-03-04T07:39:27.522061+0000","last_clean":"2024-03-04T07:39:27.522061+0000","last_became_active":"2024-03-04T07:37:55.129937+0000","last_became_peered":"2024-03-04T07:37:55.129937+0000","last_unstale":"2024-03-04T07:39:27.522061+0000","last_undegraded":"2024-03-04T07:39:27.522061+0000","last_fullsized":"2024-03-04T07:39:27.522061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:45.136539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021251+0000","last_change":"2024-03-04T07:37:54.133817+0000","last_active":"2024-03-04T07:38:59.021251+0000","last_peered":"2024-03-04T07:38:59.021251+0000","last_clean":"2024-03-04T07:38:59.021251+0000","last_became_active":"2024-03-04T07:37:54.133471+0000","last_became_peered":"2024-03-04T07:37:54.133471+0000","last_unstale":"2024-03-04T07:38:59.021251+0000","last_undegraded":"2024-03-04T07:38:59.021251+0000","last_fullsized":"2024-03-04T07:38:59.021251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:00:51.483859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.814628+0000","last_change":"2024-03-04T07:37:55.144639+0000","last_active":"2024-03-04T07:38:58.814628+0000","last_peered":"2024-03-04T07:38:58.814628+0000","last_clean":"2024-03-04T07:38:58.814628+0000","last_became_active":"2024-03-04T07:37:55.144194+0000","last_became_peered":"2024-03-04T07:37:55.144194+0000","last_unstale":"2024-03-04T07:38:58.814628+0000","last_undegraded":"2024-03-04T07:38:58.814628+0000","last_fullsized":"2024-03-04T07:38:58.814628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:42.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.815642+0000","last_change":"2024-03-04T07:38:58.815642+0000","last_active":"2024-03-04T07:38:58.815642+0000","last_peered":"2024-03-04T07:38:58.815642+0000","last_clean":"2024-03-04T07:38:58.815642+0000","last_became_active":"2024-03-04T07:38:58.815361+0000","last_became_peered":"2024-03-04T07:38:58.815361+0000","last_unstale":"2024-03-04T07:38:58.815642+0000","last_undegraded":"2024-03-04T07:38:58.815642+0000","last_fullsized":"2024-03-04T07:38:58.815642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:48:15.751929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.450900+0000","last_change":"2024-03-04T07:37:55.115145+0000","last_active":"2024-03-04T07:39:27.450900+0000","last_peered":"2024-03-04T07:39:27.450900+0000","last_clean":"2024-03-04T07:39:27.450900+0000","last_became_active":"2024-03-04T07:37:55.114791+0000","last_became_peered":"2024-03-04T07:37:55.114791+0000","last_unstale":"2024-03-04T07:39:27.450900+0000","last_undegraded":"2024-03-04T07:39:27.450900+0000","last_fullsized":"2024-03-04T07:39:27.450900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:19.685977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.806378+0000","last_change":"2024-03-04T07:37:54.141630+0000","last_active":"2024-03-04T07:38:58.806378+0000","last_peered":"2024-03-04T07:38:58.806378+0000","last_clean":"2024-03-04T07:38:58.806378+0000","last_became_active":"2024-03-04T07:37:54.139820+0000","last_became_peered":"2024-03-04T07:37:54.139820+0000","last_unstale":"2024-03-04T07:38:58.806378+0000","last_undegraded":"2024-03-04T07:38:58.806378+0000","last_fullsized":"2024-03-04T07:38:58.806378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:22.419437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808319+0000","last_change":"2024-03-04T07:37:55.133270+0000","last_active":"2024-03-04T07:38:58.808319+0000","last_peered":"2024-03-04T07:38:58.808319+0000","last_clean":"2024-03-04T07:38:58.808319+0000","last_became_active":"2024-03-04T07:37:55.132909+0000","last_became_peered":"2024-03-04T07:37:55.132909+0000","last_unstale":"2024-03-04T07:38:58.808319+0000","last_undegraded":"2024-03-04T07:38:58.808319+0000","last_fullsized":"2024-03-04T07:38:58.808319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:21.026141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812525+0000","last_change":"2024-03-04T07:37:54.135988+0000","last_active":"2024-03-04T07:38:58.812525+0000","last_peered":"2024-03-04T07:38:58.812525+0000","last_clean":"2024-03-04T07:38:58.812525+0000","last_became_active":"2024-03-04T07:37:54.135627+0000","last_became_peered":"2024-03-04T07:37:54.135627+0000","last_unstale":"2024-03-04T07:38:58.812525+0000","last_undegraded":"2024-03-04T07:38:58.812525+0000","last_fullsized":"2024-03-04T07:38:58.812525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:06.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.520786+0000","last_change":"2024-03-04T07:37:55.144551+0000","last_active":"2024-03-04T07:39:27.520786+0000","last_peered":"2024-03-04T07:39:27.520786+0000","last_clean":"2024-03-04T07:39:27.520786+0000","last_became_active":"2024-03-04T07:37:55.143882+0000","last_became_peered":"2024-03-04T07:37:55.143882+0000","last_unstale":"2024-03-04T07:39:27.520786+0000","last_undegraded":"2024-03-04T07:39:27.520786+0000","last_fullsized":"2024-03-04T07:39:27.520786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:07.254941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.817917+0000","last_change":"2024-03-04T07:38:58.817440+0000","last_active":"2024-03-04T07:38:58.817917+0000","last_peered":"2024-03-04T07:38:58.817917+0000","last_clean":"2024-03-04T07:38:58.817917+0000","last_became_active":"2024-03-04T07:38:58.813620+0000","last_became_peered":"2024-03-04T07:38:58.813620+0000","last_unstale":"2024-03-04T07:38:58.817917+0000","last_undegraded":"2024-03-04T07:38:58.817917+0000","last_fullsized":"2024-03-04T07:38:58.817917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:10:41.554608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.505080+0000","last_change":"2024-03-04T07:37:55.133870+0000","last_active":"2024-03-04T07:39:27.505080+0000","last_peered":"2024-03-04T07:39:27.505080+0000","last_clean":"2024-03-04T07:39:27.505080+0000","last_became_active":"2024-03-04T07:37:55.133542+0000","last_became_peered":"2024-03-04T07:37:55.133542+0000","last_unstale":"2024-03-04T07:39:27.505080+0000","last_undegraded":"2024-03-04T07:39:27.505080+0000","last_fullsized":"2024-03-04T07:39:27.505080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:17:44.373426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811695+0000","last_change":"2024-03-04T07:37:54.126326+0000","last_active":"2024-03-04T07:38:58.811695+0000","last_peered":"2024-03-04T07:38:58.811695+0000","last_clean":"2024-03-04T07:38:58.811695+0000","last_became_active":"2024-03-04T07:37:54.125989+0000","last_became_peered":"2024-03-04T07:37:54.125989+0000","last_unstale":"2024-03-04T07:38:58.811695+0000","last_undegraded":"2024-03-04T07:38:58.811695+0000","last_fullsized":"2024-03-04T07:38:58.811695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:41.108528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.485426+0000","last_change":"2024-03-04T07:37:55.120052+0000","last_active":"2024-03-04T07:39:27.485426+0000","last_peered":"2024-03-04T07:39:27.485426+0000","last_clean":"2024-03-04T07:39:27.485426+0000","last_became_active":"2024-03-04T07:37:55.119708+0000","last_became_peered":"2024-03-04T07:37:55.119708+0000","last_unstale":"2024-03-04T07:39:27.485426+0000","last_undegraded":"2024-03-04T07:39:27.485426+0000","last_fullsized":"2024-03-04T07:39:27.485426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:25:17.679416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808459+0000","last_change":"2024-03-04T07:37:54.134018+0000","last_active":"2024-03-04T07:38:58.808459+0000","last_peered":"2024-03-04T07:38:58.808459+0000","last_clean":"2024-03-04T07:38:58.808459+0000","last_became_active":"2024-03-04T07:37:54.133673+0000","last_became_peered":"2024-03-04T07:37:54.133673+0000","last_unstale":"2024-03-04T07:38:58.808459+0000","last_undegraded":"2024-03-04T07:38:58.808459+0000","last_fullsized":"2024-03-04T07:38:58.808459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:49.438338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.486651+0000","last_change":"2024-03-04T07:38:58.817071+0000","last_active":"2024-03-04T07:39:27.486651+0000","last_peered":"2024-03-04T07:39:27.486651+0000","last_clean":"2024-03-04T07:39:27.486651+0000","last_became_active":"2024-03-04T07:38:58.812139+0000","last_became_peered":"2024-03-04T07:38:58.812139+0000","last_unstale":"2024-03-04T07:39:27.486651+0000","last_undegraded":"2024-03-04T07:39:27.486651+0000","last_fullsized":"2024-03-04T07:39:27.486651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:37.557255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805022+0000","last_change":"2024-03-04T07:37:54.140806+0000","last_active":"2024-03-04T07:38:58.805022+0000","last_peered":"2024-03-04T07:38:58.805022+0000","last_clean":"2024-03-04T07:38:58.805022+0000","last_became_active":"2024-03-04T07:37:54.140419+0000","last_became_peered":"2024-03-04T07:37:54.140419+0000","last_unstale":"2024-03-04T07:38:58.805022+0000","last_undegraded":"2024-03-04T07:38:58.805022+0000","last_fullsized":"2024-03-04T07:38:58.805022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:13:59.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488864+0000","last_change":"2024-03-04T07:37:55.113658+0000","last_active":"2024-03-04T07:39:27.488864+0000","last_peered":"2024-03-04T07:39:27.488864+0000","last_clean":"2024-03-04T07:39:27.488864+0000","last_became_active":"2024-03-04T07:37:55.113300+0000","last_became_peered":"2024-03-04T07:37:55.113300+0000","last_unstale":"2024-03-04T07:39:27.488864+0000","last_undegraded":"2024-03-04T07:39:27.488864+0000","last_fullsized":"2024-03-04T07:39:27.488864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:43:56.907387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808475+0000","last_change":"2024-03-04T07:37:55.117769+0000","last_active":"2024-03-04T07:38:58.808475+0000","last_peered":"2024-03-04T07:38:58.808475+0000","last_clean":"2024-03-04T07:38:58.808475+0000","last_became_active":"2024-03-04T07:37:55.117408+0000","last_became_peered":"2024-03-04T07:37:55.117408+0000","last_unstale":"2024-03-04T07:38:58.808475+0000","last_undegraded":"2024-03-04T07:38:58.808475+0000","last_fullsized":"2024-03-04T07:38:58.808475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:22.143108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811215+0000","last_change":"2024-03-04T07:37:55.115581+0000","last_active":"2024-03-04T07:38:58.811215+0000","last_peered":"2024-03-04T07:38:58.811215+0000","last_clean":"2024-03-04T07:38:58.811215+0000","last_became_active":"2024-03-04T07:37:55.113387+0000","last_became_peered":"2024-03-04T07:37:55.113387+0000","last_unstale":"2024-03-04T07:38:58.811215+0000","last_undegraded":"2024-03-04T07:38:58.811215+0000","last_fullsized":"2024-03-04T07:38:58.811215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:59.273650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811830+0000","last_change":"2024-03-04T07:37:55.135175+0000","last_active":"2024-03-04T07:38:58.811830+0000","last_peered":"2024-03-04T07:38:58.811830+0000","last_clean":"2024-03-04T07:38:58.811830+0000","last_became_active":"2024-03-04T07:37:55.134787+0000","last_became_peered":"2024-03-04T07:37:55.134787+0000","last_unstale":"2024-03-04T07:38:58.811830+0000","last_undegraded":"2024-03-04T07:38:58.811830+0000","last_fullsized":"2024-03-04T07:38:58.811830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:04.227886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501056+0000","last_change":"2024-03-04T07:37:55.138146+0000","last_active":"2024-03-04T07:39:27.501056+0000","last_peered":"2024-03-04T07:39:27.501056+0000","last_clean":"2024-03-04T07:39:27.501056+0000","last_became_active":"2024-03-04T07:37:55.137806+0000","last_became_peered":"2024-03-04T07:37:55.137806+0000","last_unstale":"2024-03-04T07:39:27.501056+0000","last_undegraded":"2024-03-04T07:39:27.501056+0000","last_fullsized":"2024-03-04T07:39:27.501056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:10.596224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.506394+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.506394+0000","last_peered":"2024-03-04T07:39:27.506394+0000","last_clean":"2024-03-04T07:39:27.506394+0000","last_became_active":"2024-03-04T07:37:55.140670+0000","last_became_peered":"2024-03-04T07:37:55.140670+0000","last_unstale":"2024-03-04T07:39:27.506394+0000","last_undegraded":"2024-03-04T07:39:27.506394+0000","last_fullsized":"2024-03-04T07:39:27.506394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:29:54.439417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808611+0000","last_change":"2024-03-04T07:37:55.115015+0000","last_active":"2024-03-04T07:38:58.808611+0000","last_peered":"2024-03-04T07:38:58.808611+0000","last_clean":"2024-03-04T07:38:58.808611+0000","last_became_active":"2024-03-04T07:37:55.114668+0000","last_became_peered":"2024-03-04T07:37:55.114668+0000","last_unstale":"2024-03-04T07:38:58.808611+0000","last_undegraded":"2024-03-04T07:38:58.808611+0000","last_fullsized":"2024-03-04T07:38:58.808611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:29:03.309234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812327+0000","last_change":"2024-03-04T07:37:55.119514+0000","last_active":"2024-03-04T07:38:58.812327+0000","last_peered":"2024-03-04T07:38:58.812327+0000","last_clean":"2024-03-04T07:38:58.812327+0000","last_became_active":"2024-03-04T07:37:55.119166+0000","last_became_peered":"2024-03-04T07:37:55.119166+0000","last_unstale":"2024-03-04T07:38:58.812327+0000","last_undegraded":"2024-03-04T07:38:58.812327+0000","last_fullsized":"2024-03-04T07:38:58.812327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:33.468311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.517836+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.517836+0000","last_peered":"2024-03-04T07:39:27.517836+0000","last_clean":"2024-03-04T07:39:27.517836+0000","last_became_active":"2024-03-04T07:37:55.140723+0000","last_became_peered":"2024-03-04T07:37:55.140723+0000","last_unstale":"2024-03-04T07:39:27.517836+0000","last_undegraded":"2024-03-04T07:39:27.517836+0000","last_fullsized":"2024-03-04T07:39:27.517836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:29.751190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.510153+0000","last_change":"2024-03-04T07:37:55.127627+0000","last_active":"2024-03-04T07:39:27.510153+0000","last_peered":"2024-03-04T07:39:27.510153+0000","last_clean":"2024-03-04T07:39:27.510153+0000","last_became_active":"2024-03-04T07:37:55.127217+0000","last_became_peered":"2024-03-04T07:37:55.127217+0000","last_unstale":"2024-03-04T07:39:27.510153+0000","last_undegraded":"2024-03-04T07:39:27.510153+0000","last_fullsized":"2024-03-04T07:39:27.510153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:47.826970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519614+0000","last_change":"2024-03-04T07:37:55.129445+0000","last_active":"2024-03-04T07:39:27.519614+0000","last_peered":"2024-03-04T07:39:27.519614+0000","last_clean":"2024-03-04T07:39:27.519614+0000","last_became_active":"2024-03-04T07:37:55.129013+0000","last_became_peered":"2024-03-04T07:37:55.129013+0000","last_unstale":"2024-03-04T07:39:27.519614+0000","last_undegraded":"2024-03-04T07:39:27.519614+0000","last_fullsized":"2024-03-04T07:39:27.519614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:15.578436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523543+0000","last_change":"2024-03-04T07:37:55.126397+0000","last_active":"2024-03-04T07:39:27.523543+0000","last_peered":"2024-03-04T07:39:27.523543+0000","last_clean":"2024-03-04T07:39:27.523543+0000","last_became_active":"2024-03-04T07:37:55.126092+0000","last_became_peered":"2024-03-04T07:37:55.126092+0000","last_unstale":"2024-03-04T07:39:27.523543+0000","last_undegraded":"2024-03-04T07:39:27.523543+0000","last_fullsized":"2024-03-04T07:39:27.523543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:47.968250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511028+0000","last_change":"2024-03-04T07:37:55.140265+0000","last_active":"2024-03-04T07:39:27.511028+0000","last_peered":"2024-03-04T07:39:27.511028+0000","last_clean":"2024-03-04T07:39:27.511028+0000","last_became_active":"2024-03-04T07:37:55.139768+0000","last_became_peered":"2024-03-04T07:37:55.139768+0000","last_unstale":"2024-03-04T07:39:27.511028+0000","last_undegraded":"2024-03-04T07:39:27.511028+0000","last_fullsized":"2024-03-04T07:39:27.511028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:52.924188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808817+0000","last_change":"2024-03-04T07:37:55.118929+0000","last_active":"2024-03-04T07:38:58.808817+0000","last_peered":"2024-03-04T07:38:58.808817+0000","last_clean":"2024-03-04T07:38:58.808817+0000","last_became_active":"2024-03-04T07:37:55.118549+0000","last_became_peered":"2024-03-04T07:37:55.118549+0000","last_unstale":"2024-03-04T07:38:58.808817+0000","last_undegraded":"2024-03-04T07:38:58.808817+0000","last_fullsized":"2024-03-04T07:38:58.808817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:15.599080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244954,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61844,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":192,"num_read_kb":157,"num_write":77,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":185781,"data_compressed":5550,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":937984,"data_stored":455726,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":1107451,"data_compressed":43363,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456425,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456330,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1046793,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455843,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":517021,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":455816,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1007616,"data_stored":516806,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":455947,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455748,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":1046393,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61080,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61240,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T07:39:34.220 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T07:39:34.220 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T07:39:34.220 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-04T07:39:34.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-04T07:39:34.669 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:34.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:34 smithi116 ceph-mon[32150]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:39:34.988 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:34 smithi116 ceph-mon[32150]: pgmap v365: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:34 smithi152 ceph-mon[37154]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:39:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:34 smithi152 ceph-mon[37154]: pgmap v365: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:35.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:34 smithi133 ceph-mon[36505]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:39:35.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:34 smithi133 ceph-mon[36505]: pgmap v365: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:36.725 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:36 smithi116 ceph-mon[32150]: pgmap v366: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:36.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 -- 172.21.15.116:0/1744777269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74109090 msgr2=0x7f0a741155a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:36.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1744777269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74109090 0x7f0a741155a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00b0a0 tx=0x7f0a6c00b3b0 comp rx=0 tx=0).stop 2024-03-04T07:39:36.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 -- 172.21.15.116:0/1744777269 shutdown_connections 2024-03-04T07:39:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1744777269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74109090 0x7f0a741155a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1744777269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a7410e6d0 0x7f0a7410eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1744777269 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a741086d0 0x7f0a74108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 -- 172.21.15.116:0/1744777269 >> 172.21.15.116:0/1744777269 conn(0x7f0a74075890 msgr2=0x7f0a74077cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 -- 172.21.15.116:0/1744777269 shutdown_connections 2024-03-04T07:39:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.998+0000 7f0a79e00700 1 -- 172.21.15.116:0/1744777269 wait complete. 2024-03-04T07:39:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:36.999+0000 7f0a79e00700 1 Processor -- start 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a79e00700 1 -- start start 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a79e00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a741086d0 0x7f0a7419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a79e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a74109090 0x7f0a7419f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a79e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a7410e6d0 0x7f0a741a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a79e00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74118ab0 con 0x7f0a741086d0 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a79e00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74118930 con 0x7f0a74109090 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a79e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74118c30 con 0x7f0a7410e6d0 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a72000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a74109090 0x7f0a7419f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a72a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a741086d0 0x7f0a7419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a72000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a74109090 0x7f0a7419f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34756/0 (socket says 172.21.15.116:34756) 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.000+0000 7f0a72000700 1 -- 172.21.15.116:0/1553250332 learned_addr learned my addr 172.21.15.116:0/1553250332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a73400700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a7410e6d0 0x7f0a741a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a72000700 1 -- 172.21.15.116:0/1553250332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a7410e6d0 msgr2=0x7f0a741a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a72000700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a7410e6d0 0x7f0a741a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a72000700 1 -- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a741086d0 msgr2=0x7f0a7419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a72000700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a741086d0 0x7f0a7419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a72000700 1 -- 172.21.15.116:0/1553250332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a741a3e30 con 0x7f0a74109090 2024-03-04T07:39:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a72000700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a74109090 0x7f0a7419f360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000eb30 tx=0x7f0a6000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a78a00700 1 -- 172.21.15.116:0/1553250332 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000cc90 con 0x7f0a74109090 2024-03-04T07:39:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a78a00700 1 -- 172.21.15.116:0/1553250332 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a60004500 con 0x7f0a74109090 2024-03-04T07:39:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a741a4120 con 0x7f0a74109090 2024-03-04T07:39:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.001+0000 7f0a78a00700 1 -- 172.21.15.116:0/1553250332 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60010430 con 0x7f0a74109090 2024-03-04T07:39:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.002+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a741ab9a0 con 0x7f0a74109090 2024-03-04T07:39:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.003+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a74068d80 con 0x7f0a74109090 2024-03-04T07:39:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.004+0000 7f0a78a00700 1 -- 172.21.15.116:0/1553250332 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f0a60003f70 con 0x7f0a74109090 2024-03-04T07:39:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.006+0000 7f0a78a00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0a580a1e10 0x7f0a580a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.006+0000 7f0a78a00700 1 -- 172.21.15.116:0/1553250332 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0a600ce160 con 0x7f0a74109090 2024-03-04T07:39:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.006+0000 7f0a72a00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0a580a1e10 0x7f0a580a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.006+0000 7f0a72a00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0a580a1e10 0x7f0a580a42d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009be0 tx=0x7f0a64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:37.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.010+0000 7f0a78a00700 1 -- 172.21.15.116:0/1553250332 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0a6009b550 con 0x7f0a74109090 2024-03-04T07:39:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:36 smithi152 ceph-mon[37154]: pgmap v366: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:37.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:36 smithi133 ceph-mon[36505]: pgmap v366: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.276+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0a741a0180 con 0x7f0a74109090 2024-03-04T07:39:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.276+0000 7f0a78a00700 1 -- 172.21.15.116:0/1553250332 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f0a741a0180 con 0x7f0a74109090 2024-03-04T07:39:37.277 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:37.277 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T07:39:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0a580a1e10 msgr2=0x7f0a580a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0a580a1e10 0x7f0a580a42d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009be0 tx=0x7f0a64009260 comp rx=0 tx=0).stop 2024-03-04T07:39:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a74109090 msgr2=0x7f0a7419f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a74109090 0x7f0a7419f360 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000eb30 tx=0x7f0a6000ee40 comp rx=0 tx=0).stop 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 shutdown_connections 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a741086d0 0x7f0a7419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0a580a1e10 0x7f0a580a42d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a7410e6d0 0x7f0a741a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 --2- 172.21.15.116:0/1553250332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0a74109090 0x7f0a7419f360 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.279+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 >> 172.21.15.116:0/1553250332 conn(0x7f0a74075890 msgr2=0x7f0a74075d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.280+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 shutdown_connections 2024-03-04T07:39:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:37.280+0000 7f0a79e00700 1 -- 172.21.15.116:0/1553250332 wait complete. 2024-03-04T07:39:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:37 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1553250332' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T07:39:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:37 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1553250332' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T07:39:38.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:37 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1553250332' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T07:39:38.265 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-04T07:39:38.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:39:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 -- 172.21.15.116:0/1353662330 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 msgr2=0x7f917010ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 --2- 172.21.15.116:0/1353662330 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f917010ba60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9168009900 tx=0x7f916802bc30 comp rx=0 tx=0).stop 2024-03-04T07:39:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 -- 172.21.15.116:0/1353662330 shutdown_connections 2024-03-04T07:39:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 --2- 172.21.15.116:0/1353662330 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f917010ba60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 --2- 172.21.15.116:0/1353662330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9170104810 0x7f9170104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 --2- 172.21.15.116:0/1353662330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170103620 0x7f9170103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 -- 172.21.15.116:0/1353662330 >> 172.21.15.116:0/1353662330 conn(0x7f91700febb0 msgr2=0x7f9170100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 -- 172.21.15.116:0/1353662330 shutdown_connections 2024-03-04T07:39:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.484+0000 7f9177000640 1 -- 172.21.15.116:0/1353662330 wait complete. 2024-03-04T07:39:38.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.485+0000 7f9177000640 1 Processor -- start 2024-03-04T07:39:38.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.485+0000 7f9177000640 1 -- start start 2024-03-04T07:39:38.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9177000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9170103620 0x7f91701998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9177000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170104810 0x7f9170199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9177000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f91701a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9177000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917010f300 con 0x7f9170104810 2024-03-04T07:39:38.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9177000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917010f180 con 0x7f91701051d0 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9177000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917010f480 con 0x7f9170103620 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f91701a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9175200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170104810 0x7f9170199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f91701a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34780/0 (socket says 172.21.15.116:34780) 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 -- 172.21.15.116:0/1055957986 learned_addr learned my addr 172.21.15.116:0/1055957986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9175c00640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9170103620 0x7f91701998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 -- 172.21.15.116:0/1055957986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9170103620 msgr2=0x7f91701998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9170103620 0x7f91701998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 -- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170104810 msgr2=0x7f9170199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170104810 0x7f9170199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.486+0000 7f9176600640 1 -- 172.21.15.116:0/1055957986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91701a14d0 con 0x7f91701051d0 2024-03-04T07:39:38.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.487+0000 7f9176600640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f91701a0e60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f916802c8e0 tx=0x7f916802c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.487+0000 7f916e000640 1 -- 172.21.15.116:0/1055957986 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168043020 con 0x7f91701051d0 2024-03-04T07:39:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.487+0000 7f916e000640 1 -- 172.21.15.116:0/1055957986 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91680054e0 con 0x7f91701051d0 2024-03-04T07:39:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.487+0000 7f916e000640 1 -- 172.21.15.116:0/1055957986 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9168003ce0 con 0x7f91701051d0 2024-03-04T07:39:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.487+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91701a1760 con 0x7f91701051d0 2024-03-04T07:39:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.487+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91701a1c60 con 0x7f91701051d0 2024-03-04T07:39:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.488+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917010a940 con 0x7f91701051d0 2024-03-04T07:39:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.490+0000 7f916e000640 1 -- 172.21.15.116:0/1055957986 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9168046020 con 0x7f91701051d0 2024-03-04T07:39:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.491+0000 7f916e000640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91580a1e00 0x7f91580a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.491+0000 7f916e000640 1 -- 172.21.15.116:0/1055957986 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f91680efe00 con 0x7f91701051d0 2024-03-04T07:39:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.492+0000 7f9175c00640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91580a1e00 0x7f91580a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.492+0000 7f9175c00640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91580a1e00 0x7f91580a42c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9170104670 tx=0x7f9164009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:38.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.496+0000 7f916e000640 1 -- 172.21.15.116:0/1055957986 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f916804d020 con 0x7f91701051d0 2024-03-04T07:39:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.695+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9170062270 con 0x7f91701051d0 2024-03-04T07:39:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.696+0000 7f916e000640 1 -- 172.21.15.116:0/1055957986 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f91680bd310 con 0x7f91701051d0 2024-03-04T07:39:38.698 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:38.698 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:39:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91580a1e00 msgr2=0x7f91580a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91580a1e00 0x7f91580a42c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9170104670 tx=0x7f9164009670 comp rx=0 tx=0).stop 2024-03-04T07:39:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 msgr2=0x7f91701a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f91701a0e60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f916802c8e0 tx=0x7f916802c6a0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 shutdown_connections 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f91701051d0 0x7f91701a0e60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9170104810 0x7f9170199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91580a1e00 0x7f91580a42c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 --2- 172.21.15.116:0/1055957986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9170103620 0x7f91701998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 >> 172.21.15.116:0/1055957986 conn(0x7f91700febb0 msgr2=0x7f9170107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.699+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 shutdown_connections 2024-03-04T07:39:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.700+0000 7f9177000640 1 -- 172.21.15.116:0/1055957986 wait complete. 2024-03-04T07:39:38.721 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:39:38.721 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:39:38.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.927+0000 7f5ce8600640 1 -- 172.21.15.116:0/93185835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 msgr2=0x7f5ce0109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.927+0000 7f5ce8600640 1 --2- 172.21.15.116:0/93185835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce0109260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009900 tx=0x7f5ccc02bc30 comp rx=0 tx=0).stop 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.927+0000 7f5ce8600640 1 -- 172.21.15.116:0/93185835 shutdown_connections 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.927+0000 7f5ce8600640 1 --2- 172.21.15.116:0/93185835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce01097a0 0x7f5ce0110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.927+0000 7f5ce8600640 1 --2- 172.21.15.116:0/93185835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce0109260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.927+0000 7f5ce8600640 1 --2- 172.21.15.116:0/93185835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ce0107bf0 0x7f5ce0108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.928+0000 7f5ce8600640 1 -- 172.21.15.116:0/93185835 >> 172.21.15.116:0/93185835 conn(0x7f5ce0076b90 msgr2=0x7f5ce0078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.928+0000 7f5ce8600640 1 -- 172.21.15.116:0/93185835 shutdown_connections 2024-03-04T07:39:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.928+0000 7f5ce8600640 1 -- 172.21.15.116:0/93185835 wait complete. 2024-03-04T07:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 Processor -- start 2024-03-04T07:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 -- start start 2024-03-04T07:39:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce0107bf0 0x7f5ce019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ce01097a0 0x7f5ce01a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce01138c0 con 0x7f5ce0107bf0 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce0113740 con 0x7f5ce0108de0 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.929+0000 7f5ce8600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce0113a40 con 0x7f5ce01097a0 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce019e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34804/0 (socket says 172.21.15.116:34804) 2024-03-04T07:39:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 -- 172.21.15.116:0/2268427826 learned_addr learned my addr 172.21.15.116:0/2268427826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5c00640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce0107bf0 0x7f5ce019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce6600640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ce01097a0 0x7f5ce01a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 -- 172.21.15.116:0/2268427826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ce01097a0 msgr2=0x7f5ce01a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ce01097a0 0x7f5ce01a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 -- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce0107bf0 msgr2=0x7f5ce019df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce0107bf0 0x7f5ce019df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 -- 172.21.15.116:0/2268427826 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce01a5b30 con 0x7f5ce0108de0 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5c00640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce0107bf0 0x7f5ce019df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5ce5200640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce019e450 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc000c00 tx=0x7f5ccc02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.930+0000 7f5cde000640 1 -- 172.21.15.116:0/2268427826 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc043020 con 0x7f5ce0108de0 2024-03-04T07:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.931+0000 7f5cde000640 1 -- 172.21.15.116:0/2268427826 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ccc0054e0 con 0x7f5ce0108de0 2024-03-04T07:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.931+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce01a5dc0 con 0x7f5ce0108de0 2024-03-04T07:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.931+0000 7f5cde000640 1 -- 172.21.15.116:0/2268427826 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc003c40 con 0x7f5ce0108de0 2024-03-04T07:39:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.931+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce01a6290 con 0x7f5ce0108de0 2024-03-04T07:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.932+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce010ef10 con 0x7f5ce0108de0 2024-03-04T07:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.933+0000 7f5cde000640 1 -- 172.21.15.116:0/2268427826 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5ccc046020 con 0x7f5ce0108de0 2024-03-04T07:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.935+0000 7f5cde000640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5cc80a1e00 0x7f5cc80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.935+0000 7f5cde000640 1 -- 172.21.15.116:0/2268427826 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5ccc0efe50 con 0x7f5ce0108de0 2024-03-04T07:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.935+0000 7f5ce5c00640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5cc80a1e00 0x7f5cc80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.935+0000 7f5ce5c00640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5cc80a1e00 0x7f5cc80a42c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0108c40 tx=0x7f5cd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:38.938+0000 7f5cde000640 1 -- 172.21.15.116:0/2268427826 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5ccc04d020 con 0x7f5ce0108de0 2024-03-04T07:39:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:38 smithi116 ceph-mon[32150]: pgmap v367: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:38 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1055957986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:38 smithi152 ceph-mon[37154]: pgmap v367: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:38 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1055957986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:38 smithi133 ceph-mon[36505]: pgmap v367: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:38 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1055957986' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:39.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.142+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ce002cff0 con 0x7f5ce0108de0 2024-03-04T07:39:39.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.142+0000 7f5cde000640 1 -- 172.21.15.116:0/2268427826 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ccc0bd240 con 0x7f5ce0108de0 2024-03-04T07:39:39.143 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:39:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5cc80a1e00 msgr2=0x7f5cc80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5cc80a1e00 0x7f5cc80a42c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0108c40 tx=0x7f5cd4005f90 comp rx=0 tx=0).stop 2024-03-04T07:39:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 msgr2=0x7f5ce019e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce019e450 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc000c00 tx=0x7f5ccc02c650 comp rx=0 tx=0).stop 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 shutdown_connections 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5cc80a1e00 0x7f5cc80a42c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ce01097a0 0x7f5ce01a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ce0108de0 0x7f5ce019e450 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 --2- 172.21.15.116:0/2268427826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce0107bf0 0x7f5ce019df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.145+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 >> 172.21.15.116:0/2268427826 conn(0x7f5ce0076b90 msgr2=0x7f5ce010c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.146+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 shutdown_connections 2024-03-04T07:39:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.146+0000 7f5ce8600640 1 -- 172.21.15.116:0/2268427826 wait complete. 2024-03-04T07:39:39.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:39:39.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 -- 172.21.15.116:0/1403331622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 msgr2=0x7f78f0110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1403331622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 0x7f78f0110030 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009be0 tx=0x7f78e802bef0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 -- 172.21.15.116:0/1403331622 shutdown_connections 2024-03-04T07:39:39.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1403331622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 0x7f78f0110030 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1403331622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0108de0 0x7f78f0109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1403331622 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0107bf0 0x7f78f0108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 -- 172.21.15.116:0/1403331622 >> 172.21.15.116:0/1403331622 conn(0x7f78f0076b90 msgr2=0x7f78f0078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:39.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 -- 172.21.15.116:0/1403331622 shutdown_connections 2024-03-04T07:39:39.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.363+0000 7f78f7a00640 1 -- 172.21.15.116:0/1403331622 wait complete. 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.364+0000 7f78f7a00640 1 Processor -- start 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.364+0000 7f78f7a00640 1 -- start start 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f7a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0107bf0 0x7f78f019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f7a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0108de0 0x7f78f019e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f7a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 0x7f78f01a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f7a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0113850 con 0x7f78f0107bf0 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f7a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01136d0 con 0x7f78f0108de0 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f7a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01139d0 con 0x7f78f01097a0 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0108de0 0x7f78f019e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f5000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0107bf0 0x7f78f019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f5a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 0x7f78f01a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0108de0 0x7f78f019e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34818/0 (socket says 172.21.15.116:34818) 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 -- 172.21.15.116:0/1737988733 learned_addr learned my addr 172.21.15.116:0/1737988733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 -- 172.21.15.116:0/1737988733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 msgr2=0x7f78f01a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 0x7f78f01a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 -- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0107bf0 msgr2=0x7f78f019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0107bf0 0x7f78f019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78e7e00640 1 -- 172.21.15.116:0/1737988733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f01a5a90 con 0x7f78f0108de0 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f5a00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 0x7f78f01a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.365+0000 7f78f5000640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0107bf0 0x7f78f019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.366+0000 7f78e7e00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0108de0 0x7f78f019e3b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f78d800e9c0 tx=0x7f78d800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.366+0000 7f78ee000640 1 -- 172.21.15.116:0/1737988733 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d800ccc0 con 0x7f78f0108de0 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.366+0000 7f78ee000640 1 -- 172.21.15.116:0/1737988733 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78d8005500 con 0x7f78f0108de0 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.366+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f01a5d80 con 0x7f78f0108de0 2024-03-04T07:39:39.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.366+0000 7f78ee000640 1 -- 172.21.15.116:0/1737988733 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8010430 con 0x7f78f0108de0 2024-03-04T07:39:39.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.366+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f01a6270 con 0x7f78f0108de0 2024-03-04T07:39:39.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.367+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f010ef10 con 0x7f78f0108de0 2024-03-04T07:39:39.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.369+0000 7f78ee000640 1 -- 172.21.15.116:0/1737988733 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f78d80046c0 con 0x7f78f0108de0 2024-03-04T07:39:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.370+0000 7f78ee000640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1d80 0x7f78d40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.370+0000 7f78ee000640 1 -- 172.21.15.116:0/1737988733 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f78d80ce080 con 0x7f78f0108de0 2024-03-04T07:39:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.371+0000 7f78f5000640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1d80 0x7f78d40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.371+0000 7f78f5000640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1d80 0x7f78d40a4240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f78f0108c40 tx=0x7f78e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:39.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.375+0000 7f78ee000640 1 -- 172.21.15.116:0/1737988733 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f78d809b4b0 con 0x7f78f0108de0 2024-03-04T07:39:39.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.564+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78f0062270 con 0x7f78f0108de0 2024-03-04T07:39:39.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.566+0000 7f78ee000640 1 -- 172.21.15.116:0/1737988733 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f78d809adb0 con 0x7f78f0108de0 2024-03-04T07:39:39.566 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:39:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1d80 msgr2=0x7f78d40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1d80 0x7f78d40a4240 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f78f0108c40 tx=0x7f78e0005f90 comp rx=0 tx=0).stop 2024-03-04T07:39:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0108de0 msgr2=0x7f78f019e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0108de0 0x7f78f019e3b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f78d800e9c0 tx=0x7f78d800ee80 comp rx=0 tx=0).stop 2024-03-04T07:39:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 shutdown_connections 2024-03-04T07:39:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1d80 0x7f78d40a4240 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01097a0 0x7f78f01a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f0108de0 0x7f78f019e3b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 --2- 172.21.15.116:0/1737988733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0107bf0 0x7f78f019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 >> 172.21.15.116:0/1737988733 conn(0x7f78f0076b90 msgr2=0x7f78f010c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 shutdown_connections 2024-03-04T07:39:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.567+0000 7f78f7a00640 1 -- 172.21.15.116:0/1737988733 wait complete. 2024-03-04T07:39:39.585 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:39:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 -- 172.21.15.116:0/695515774 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 msgr2=0x7f34a0101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 --2- 172.21.15.116:0/695515774 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0101a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3490009ca0 tx=0x7f349002c0f0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 -- 172.21.15.116:0/695515774 shutdown_connections 2024-03-04T07:39:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 --2- 172.21.15.116:0/695515774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34a0103190 0x7f34a0109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 --2- 172.21.15.116:0/695515774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a01027d0 0x7f34a0102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 --2- 172.21.15.116:0/695515774 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0101a00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 -- 172.21.15.116:0/695515774 >> 172.21.15.116:0/695515774 conn(0x7f34a00fcb70 msgr2=0x7f34a00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 -- 172.21.15.116:0/695515774 shutdown_connections 2024-03-04T07:39:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.797+0000 7f34a7a00640 1 -- 172.21.15.116:0/695515774 wait complete. 2024-03-04T07:39:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.798+0000 7f34a7a00640 1 Processor -- start 2024-03-04T07:39:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.798+0000 7f34a7a00640 1 -- start start 2024-03-04T07:39:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a7a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a7a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34a01027d0 0x7f34a0197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a7a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a0103190 0x7f34a019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a7a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a010d250 con 0x7f34a0103190 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a7a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a010d0d0 con 0x7f34a01015e0 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a7a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a010d3d0 con 0x7f34a01027d0 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34832/0 (socket says 172.21.15.116:34832) 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 -- 172.21.15.116:0/1561158986 learned_addr learned my addr 172.21.15.116:0/1561158986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a0103190 0x7f34a019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f3497e00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34a01027d0 0x7f34a0197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 -- 172.21.15.116:0/1561158986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34a01027d0 msgr2=0x7f34a0197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34a01027d0 0x7f34a0197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 -- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a0103190 msgr2=0x7f34a019eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a0103190 0x7f34a019eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5000640 1 -- 172.21.15.116:0/1561158986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a019f520 con 0x7f34a01015e0 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.799+0000 7f34a5a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a0103190 0x7f34a019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.800+0000 7f34a5000640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0197900 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34900324c0 tx=0x7f3490032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.800+0000 7f349e000640 1 -- 172.21.15.116:0/1561158986 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490041020 con 0x7f34a01015e0 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.800+0000 7f349e000640 1 -- 172.21.15.116:0/1561158986 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3490034cd0 con 0x7f34a01015e0 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.800+0000 7f349e000640 1 -- 172.21.15.116:0/1561158986 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490039040 con 0x7f34a01015e0 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.800+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a019f7b0 con 0x7f34a01015e0 2024-03-04T07:39:39.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.800+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a019fc50 con 0x7f34a01015e0 2024-03-04T07:39:39.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.801+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a0108970 con 0x7f34a01015e0 2024-03-04T07:39:39.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.803+0000 7f349e000640 1 -- 172.21.15.116:0/1561158986 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3490044020 con 0x7f34a01015e0 2024-03-04T07:39:39.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.804+0000 7f349e000640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f34840a1e00 0x7f34840a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:39.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.804+0000 7f349e000640 1 -- 172.21.15.116:0/1561158986 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f34900efd00 con 0x7f34a01015e0 2024-03-04T07:39:39.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.804+0000 7f3497e00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f34840a1e00 0x7f34840a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:39.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.805+0000 7f3497e00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f34840a1e00 0x7f34840a42c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f34a0198e10 tx=0x7f348c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:39.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:39.812+0000 7f349e000640 1 -- 172.21.15.116:0/1561158986 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f349004b020 con 0x7f34a01015e0 2024-03-04T07:39:40.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:39 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2268427826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:39:40.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:39 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1737988733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:39:40.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.010+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34a00621d0 con 0x7f34a01015e0 2024-03-04T07:39:40.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.011+0000 7f349e000640 1 -- 172.21.15.116:0/1561158986 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34900bd1c0 con 0x7f34a01015e0 2024-03-04T07:39:40.011 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:39:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f34840a1e00 msgr2=0x7f34840a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f34840a1e00 0x7f34840a42c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f34a0198e10 tx=0x7f348c005f90 comp rx=0 tx=0).stop 2024-03-04T07:39:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 msgr2=0x7f34a0197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0197900 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34900324c0 tx=0x7f3490032f40 comp rx=0 tx=0).stop 2024-03-04T07:39:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 shutdown_connections 2024-03-04T07:39:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a0103190 0x7f34a019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f34840a1e00 0x7f34840a42c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34a01027d0 0x7f34a0197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 --2- 172.21.15.116:0/1561158986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f34a01015e0 0x7f34a0197900 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 >> 172.21.15.116:0/1561158986 conn(0x7f34a00fcb70 msgr2=0x7f34a0105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:40.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 shutdown_connections 2024-03-04T07:39:40.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.014+0000 7f34a7a00640 1 -- 172.21.15.116:0/1561158986 wait complete. 2024-03-04T07:39:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:39 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2268427826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:39:40.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:39 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1737988733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:39:40.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:39:40.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:39 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2268427826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:39:40.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:39 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1737988733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:39:40.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 -- 172.21.15.116:0/140380443 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 msgr2=0x7fea68101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 --2- 172.21.15.116:0/140380443 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68101850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009900 tx=0x7fea5c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:39:40.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 -- 172.21.15.116:0/140380443 shutdown_connections 2024-03-04T07:39:40.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 --2- 172.21.15.116:0/140380443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea68102fe0 0x7fea68109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 --2- 172.21.15.116:0/140380443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea68102620 0x7fea68102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 --2- 172.21.15.116:0/140380443 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68101850 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 -- 172.21.15.116:0/140380443 >> 172.21.15.116:0/140380443 conn(0x7fea680fc9a0 msgr2=0x7fea680fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:40.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 -- 172.21.15.116:0/140380443 shutdown_connections 2024-03-04T07:39:40.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.236+0000 7fea6e800640 1 -- 172.21.15.116:0/140380443 wait complete. 2024-03-04T07:39:40.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.237+0000 7fea6e800640 1 Processor -- start 2024-03-04T07:39:40.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.237+0000 7fea6e800640 1 -- start start 2024-03-04T07:39:40.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea68102620 0x7fea68199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea68102fe0 0x7fea681a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6810d1d0 con 0x7fea68102fe0 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6e800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6810d050 con 0x7fea68101430 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6e800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6810d350 con 0x7fea68102620 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34844/0 (socket says 172.21.15.116:34844) 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 -- 172.21.15.116:0/2758275213 learned_addr learned my addr 172.21.15.116:0/2758275213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6de00640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea68102fe0 0x7fea681a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6ca00640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea68102620 0x7fea68199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 -- 172.21.15.116:0/2758275213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea68102620 msgr2=0x7fea68199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea68102620 0x7fea68199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 -- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea68102fe0 msgr2=0x7fea681a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea68102fe0 0x7fea681a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.238+0000 7fea6d400640 1 -- 172.21.15.116:0/2758275213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea681a1680 con 0x7fea68101430 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.239+0000 7fea6d400640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68199ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fea5c000c00 tx=0x7fea5c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.239+0000 7fea66000640 1 -- 172.21.15.116:0/2758275213 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c043020 con 0x7fea68101430 2024-03-04T07:39:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.239+0000 7fea66000640 1 -- 172.21.15.116:0/2758275213 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea5c004440 con 0x7fea68101430 2024-03-04T07:39:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.239+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea681a1910 con 0x7fea68101430 2024-03-04T07:39:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.239+0000 7fea66000640 1 -- 172.21.15.116:0/2758275213 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c03e450 con 0x7fea68101430 2024-03-04T07:39:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.239+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea681a1e60 con 0x7fea68101430 2024-03-04T07:39:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.240+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea681a2030 con 0x7fea68101430 2024-03-04T07:39:40.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.242+0000 7fea66000640 1 -- 172.21.15.116:0/2758275213 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fea5c046020 con 0x7fea68101430 2024-03-04T07:39:40.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.243+0000 7fea66000640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fea500a1d80 0x7fea500a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.243+0000 7fea66000640 1 -- 172.21.15.116:0/2758275213 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fea5c0f0300 con 0x7fea68101430 2024-03-04T07:39:40.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.244+0000 7fea6ca00640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fea500a1d80 0x7fea500a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.244+0000 7fea6ca00640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fea500a1d80 0x7fea500a4240 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea6819af70 tx=0x7fea54008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.250+0000 7fea66000640 1 -- 172.21.15.116:0/2758275213 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fea5c04d020 con 0x7fea68101430 2024-03-04T07:39:40.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.483+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea681a2200 con 0x7fea68101430 2024-03-04T07:39:40.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.484+0000 7fea66000640 1 -- 172.21.15.116:0/2758275213 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fea5c0bd6f0 con 0x7fea68101430 2024-03-04T07:39:40.484 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:40.484 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:39:40.484 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:39:40.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fea500a1d80 msgr2=0x7fea500a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fea500a1d80 0x7fea500a4240 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea6819af70 tx=0x7fea54008040 comp rx=0 tx=0).stop 2024-03-04T07:39:40.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 msgr2=0x7fea68199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68199ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fea5c000c00 tx=0x7fea5c004040 comp rx=0 tx=0).stop 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 shutdown_connections 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea68102fe0 0x7fea681a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fea500a1d80 0x7fea500a4240 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea68102620 0x7fea68199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 --2- 172.21.15.116:0/2758275213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fea68101430 0x7fea68199ab0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.486+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 >> 172.21.15.116:0/2758275213 conn(0x7fea680fc9a0 msgr2=0x7fea68073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.487+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 shutdown_connections 2024-03-04T07:39:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.487+0000 7fea6e800640 1 -- 172.21.15.116:0/2758275213 wait complete. 2024-03-04T07:39:40.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:39:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 -- 172.21.15.116:0/947606840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 msgr2=0x7f3248103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 --2- 172.21.15.116:0/947606840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248103c30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3240009900 tx=0x7f324002bc40 comp rx=0 tx=0).stop 2024-03-04T07:39:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 -- 172.21.15.116:0/947606840 shutdown_connections 2024-03-04T07:39:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 --2- 172.21.15.116:0/947606840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32481053c0 0x7f324810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 --2- 172.21.15.116:0/947606840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 0x7f3248104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 --2- 172.21.15.116:0/947606840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248103c30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 -- 172.21.15.116:0/947606840 >> 172.21.15.116:0/947606840 conn(0x7f32480fedc0 msgr2=0x7f32481011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:40.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 -- 172.21.15.116:0/947606840 shutdown_connections 2024-03-04T07:39:40.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.709+0000 7f324f000640 1 -- 172.21.15.116:0/947606840 wait complete. 2024-03-04T07:39:40.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 Processor -- start 2024-03-04T07:39:40.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 -- start start 2024-03-04T07:39:40.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 0x7f324819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32481053c0 0x7f32481a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324810f4e0 con 0x7f3248104a00 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324810f360 con 0x7f3248103810 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.710+0000 7f324f000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324810f660 con 0x7f32481053c0 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 0x7f324819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34864/0 (socket says 172.21.15.116:34864) 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 -- 172.21.15.116:0/1829590358 learned_addr learned my addr 172.21.15.116:0/1829590358 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 0x7f324819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44996/0 (socket says 172.21.15.116:44996) 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f324d000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32481053c0 0x7f32481a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 -- 172.21.15.116:0/1829590358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32481053c0 msgr2=0x7f32481a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32481053c0 0x7f32481a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 -- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 msgr2=0x7f324819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 0x7f324819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 -- 172.21.15.116:0/1829590358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32481a1740 con 0x7f3248103810 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f324d000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32481053c0 0x7f32481a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247400640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 0x7f324819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3247e00640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248199b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f324002efd0 tx=0x7f3240005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.711+0000 7f3244c00640 1 -- 172.21.15.116:0/1829590358 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240041070 con 0x7f3248103810 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.712+0000 7f3244c00640 1 -- 172.21.15.116:0/1829590358 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32400044d0 con 0x7f3248103810 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.712+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32481a19d0 con 0x7f3248103810 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.712+0000 7f3244c00640 1 -- 172.21.15.116:0/1829590358 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240034440 con 0x7f3248103810 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.712+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32481a1e90 con 0x7f3248103810 2024-03-04T07:39:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.713+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324810aad0 con 0x7f3248103810 2024-03-04T07:39:40.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.714+0000 7f3244c00640 1 -- 172.21.15.116:0/1829590358 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3240044070 con 0x7f3248103810 2024-03-04T07:39:40.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.716+0000 7f3244c00640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f322c0a1d80 0x7f322c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:40.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.716+0000 7f3244c00640 1 -- 172.21.15.116:0/1829590358 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f32400efa10 con 0x7f3248103810 2024-03-04T07:39:40.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.716+0000 7f3247400640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f322c0a1d80 0x7f322c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:40.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.717+0000 7f3247400640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f322c0a1d80 0x7f322c0a4240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f32300099a0 tx=0x7f3230009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:40.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.720+0000 7f3244c00640 1 -- 172.21.15.116:0/1829590358 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f32400bce40 con 0x7f3248103810 2024-03-04T07:39:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:40 smithi116 ceph-mon[32150]: pgmap v368: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:40 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1561158986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:39:40.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:40 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2758275213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:40.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.911+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3248062270 con 0x7f3248103810 2024-03-04T07:39:40.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.914+0000 7f3244c00640 1 -- 172.21.15.116:0/1829590358 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f32400bc740 con 0x7f3248103810 2024-03-04T07:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f322c0a1d80 msgr2=0x7f322c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f322c0a1d80 0x7f322c0a4240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f32300099a0 tx=0x7f3230009500 comp rx=0 tx=0).stop 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 msgr2=0x7f3248199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248199b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f324002efd0 tx=0x7f3240005d50 comp rx=0 tx=0).stop 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 shutdown_connections 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f322c0a1d80 0x7f322c0a4240 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32481053c0 0x7f32481a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3248104a00 0x7f324819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 --2- 172.21.15.116:0/1829590358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3248103810 0x7f3248199b20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 >> 172.21.15.116:0/1829590358 conn(0x7f32480fedc0 msgr2=0x7f3248107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 shutdown_connections 2024-03-04T07:39:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:40.917+0000 7f324f000640 1 -- 172.21.15.116:0/1829590358 wait complete. 2024-03-04T07:39:40.934 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:39:40.935 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:39:41.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:40 smithi152 ceph-mon[37154]: pgmap v368: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:41.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:40 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1561158986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:39:41.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:40 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2758275213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:40 smithi133 ceph-mon[36505]: pgmap v368: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:40 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1561158986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:39:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:40 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2758275213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:41.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.143+0000 7f755a800640 1 -- 172.21.15.116:0/1648692475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 msgr2=0x7f7554100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.143+0000 7f755a800640 1 --2- 172.21.15.116:0/1648692475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 0x7f7554100b10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7540009900 tx=0x7f754002bc30 comp rx=0 tx=0).stop 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.144+0000 7f755a800640 1 -- 172.21.15.116:0/1648692475 shutdown_connections 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.144+0000 7f755a800640 1 --2- 172.21.15.116:0/1648692475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7554101050 0x7f75541014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.144+0000 7f755a800640 1 --2- 172.21.15.116:0/1648692475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 0x7f7554100b10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.144+0000 7f755a800640 1 --2- 172.21.15.116:0/1648692475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f75540ffd50 0x7f7554100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.144+0000 7f755a800640 1 -- 172.21.15.116:0/1648692475 >> 172.21.15.116:0/1648692475 conn(0x7f75540fb950 msgr2=0x7f75540fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.144+0000 7f755a800640 1 -- 172.21.15.116:0/1648692475 shutdown_connections 2024-03-04T07:39:41.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.144+0000 7f755a800640 1 -- 172.21.15.116:0/1648692475 wait complete. 2024-03-04T07:39:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 Processor -- start 2024-03-04T07:39:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 -- start start 2024-03-04T07:39:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75540ffd50 0x7f75541999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 0x7f7554199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7554101050 0x7f75541a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755410f320 con 0x7f75540ffd50 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755410f1a0 con 0x7f75541006b0 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f755a800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755410f4a0 con 0x7f7554101050 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f7559400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75540ffd50 0x7f75541999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f7558a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 0x7f7554199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f7559400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75540ffd50 0x7f75541999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45022/0 (socket says 172.21.15.116:45022) 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f7559400640 1 -- 172.21.15.116:0/513828062 learned_addr learned my addr 172.21.15.116:0/513828062 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.145+0000 7f7559e00640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7554101050 0x7f75541a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7559400640 1 -- 172.21.15.116:0/513828062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7554101050 msgr2=0x7f75541a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7559400640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7554101050 0x7f75541a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7559400640 1 -- 172.21.15.116:0/513828062 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 msgr2=0x7f7554199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7559400640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 0x7f7554199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7559400640 1 -- 172.21.15.116:0/513828062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75541a15c0 con 0x7f75540ffd50 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7559e00640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7554101050 0x7f75541a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:39:41.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7558a00640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 0x7f7554199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7559400640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75540ffd50 0x7f75541999a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f754800d950 tx=0x7f754800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7552000640 1 -- 172.21.15.116:0/513828062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548017030 con 0x7f75540ffd50 2024-03-04T07:39:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75541a18b0 con 0x7f75540ffd50 2024-03-04T07:39:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7552000640 1 -- 172.21.15.116:0/513828062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7548004020 con 0x7f75540ffd50 2024-03-04T07:39:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f7552000640 1 -- 172.21.15.116:0/513828062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75480047a0 con 0x7f75540ffd50 2024-03-04T07:39:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.146+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75541a1e20 con 0x7f75540ffd50 2024-03-04T07:39:41.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.148+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755410ac50 con 0x7f75540ffd50 2024-03-04T07:39:41.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.149+0000 7f7552000640 1 -- 172.21.15.116:0/513828062 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f75480041d0 con 0x7f75540ffd50 2024-03-04T07:39:41.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.151+0000 7f7552000640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f753c0a1d80 0x7f753c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.151+0000 7f7552000640 1 -- 172.21.15.116:0/513828062 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f75480cd850 con 0x7f75540ffd50 2024-03-04T07:39:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.151+0000 7f7558a00640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f753c0a1d80 0x7f753c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.152+0000 7f7558a00640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f753c0a1d80 0x7f753c0a4240 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f754002c7b0 tx=0x7f7540005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:41.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.157+0000 7f7552000640 1 -- 172.21.15.116:0/513828062 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f754809ac40 con 0x7f75540ffd50 2024-03-04T07:39:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.355+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75540621d0 con 0x7f75540ffd50 2024-03-04T07:39:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.356+0000 7f7552000640 1 -- 172.21.15.116:0/513828062 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f754809a540 con 0x7f75540ffd50 2024-03-04T07:39:41.355 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:39:41.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f753c0a1d80 msgr2=0x7f753c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f753c0a1d80 0x7f753c0a4240 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f754002c7b0 tx=0x7f7540005fd0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75540ffd50 msgr2=0x7f75541999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75540ffd50 0x7f75541999a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f754800d950 tx=0x7f754800de10 comp rx=0 tx=0).stop 2024-03-04T07:39:41.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 shutdown_connections 2024-03-04T07:39:41.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f753c0a1d80 0x7f753c0a4240 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7554101050 0x7f75541a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75541006b0 0x7f7554199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 --2- 172.21.15.116:0/513828062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75540ffd50 0x7f75541999a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.358+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 >> 172.21.15.116:0/513828062 conn(0x7f75540fb950 msgr2=0x7f75540fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:41.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.359+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 shutdown_connections 2024-03-04T07:39:41.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.359+0000 7f755a800640 1 -- 172.21.15.116:0/513828062 wait complete. 2024-03-04T07:39:41.378 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:39:41.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 -- 172.21.15.116:0/889497652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 msgr2=0x7f7e7010ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 --2- 172.21.15.116:0/889497652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 0x7f7e7010ffa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009930 tx=0x7f7e6002bc30 comp rx=0 tx=0).stop 2024-03-04T07:39:41.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 -- 172.21.15.116:0/889497652 shutdown_connections 2024-03-04T07:39:41.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 --2- 172.21.15.116:0/889497652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 0x7f7e7010ffa0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 --2- 172.21.15.116:0/889497652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e70074030 0x7f7e700744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 --2- 172.21.15.116:0/889497652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70075610 0x7f7e70073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 -- 172.21.15.116:0/889497652 >> 172.21.15.116:0/889497652 conn(0x7f7e700fcb70 msgr2=0x7f7e700fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:41.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 -- 172.21.15.116:0/889497652 shutdown_connections 2024-03-04T07:39:41.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.603+0000 7f7e74a00640 1 -- 172.21.15.116:0/889497652 wait complete. 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.604+0000 7f7e74a00640 1 Processor -- start 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.604+0000 7f7e74a00640 1 -- start start 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.604+0000 7f7e74a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70074030 0x7f7e7019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e74a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e70075610 0x7f7e7019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e74a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 0x7f7e701a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e74a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e701138a0 con 0x7f7e70074030 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e74a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70113720 con 0x7f7e7010dbb0 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e74a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70113a20 con 0x7f7e70075610 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70074030 0x7f7e7019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70074030 0x7f7e7019def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45038/0 (socket says 172.21.15.116:45038) 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 -- 172.21.15.116:0/377055655 learned_addr learned my addr 172.21.15.116:0/377055655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6e800640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 0x7f7e701a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6d400640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e70075610 0x7f7e7019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 -- 172.21.15.116:0/377055655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e70075610 msgr2=0x7f7e7019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e70075610 0x7f7e7019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 -- 172.21.15.116:0/377055655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 msgr2=0x7f7e701a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 0x7f7e701a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 -- 172.21.15.116:0/377055655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e701a5b10 con 0x7f7e70074030 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6e800640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 0x7f7e701a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.605+0000 7f7e6de00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70074030 0x7f7e7019def0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800ebc0 tx=0x7f7e6800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.606+0000 7f7e66a00640 1 -- 172.21.15.116:0/377055655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68010630 con 0x7f7e70074030 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.606+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e701a5e00 con 0x7f7e70074030 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.606+0000 7f7e66a00640 1 -- 172.21.15.116:0/377055655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e68010cb0 con 0x7f7e70074030 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.606+0000 7f7e66a00640 1 -- 172.21.15.116:0/377055655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68018bd0 con 0x7f7e70074030 2024-03-04T07:39:41.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.606+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e701a62c0 con 0x7f7e70074030 2024-03-04T07:39:41.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.607+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e70103d80 con 0x7f7e70074030 2024-03-04T07:39:41.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.609+0000 7f7e66a00640 1 -- 172.21.15.116:0/377055655 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f7e680107d0 con 0x7f7e70074030 2024-03-04T07:39:41.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.610+0000 7f7e66a00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7e540a1e00 0x7f7e540a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:41.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.610+0000 7f7e66a00640 1 -- 172.21.15.116:0/377055655 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7e680ce3f0 con 0x7f7e70074030 2024-03-04T07:39:41.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.610+0000 7f7e6d400640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7e540a1e00 0x7f7e540a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:41.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.611+0000 7f7e6d400640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7e540a1e00 0x7f7e540a42c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7e7019f400 tx=0x7f7e58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:41.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.614+0000 7f7e66a00640 1 -- 172.21.15.116:0/377055655 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7e6809b7e0 con 0x7f7e70074030 2024-03-04T07:39:41.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.813+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e70062270 con 0x7f7e70074030 2024-03-04T07:39:41.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.813+0000 7f7e66a00640 1 -- 172.21.15.116:0/377055655 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7e6809b0e0 con 0x7f7e70074030 2024-03-04T07:39:41.813 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:39:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7e540a1e00 msgr2=0x7f7e540a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7e540a1e00 0x7f7e540a42c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7e7019f400 tx=0x7f7e58005f90 comp rx=0 tx=0).stop 2024-03-04T07:39:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70074030 msgr2=0x7f7e7019def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70074030 0x7f7e7019def0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800ebc0 tx=0x7f7e6800c6a0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 shutdown_connections 2024-03-04T07:39:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7e7010dbb0 0x7f7e701a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7e540a1e00 0x7f7e540a42c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e70075610 0x7f7e7019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 --2- 172.21.15.116:0/377055655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e70074030 0x7f7e7019def0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 >> 172.21.15.116:0/377055655 conn(0x7f7e700fcb70 msgr2=0x7f7e70100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 shutdown_connections 2024-03-04T07:39:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:41.816+0000 7f7e74a00640 1 -- 172.21.15.116:0/377055655 wait complete. 2024-03-04T07:39:41.836 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:39:42.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:41 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1829590358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:42.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:41 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/513828062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:39:42.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:41 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1829590358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:42.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:41 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/513828062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 -- 172.21.15.116:0/1693525293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 msgr2=0x7f9360106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 --2- 172.21.15.116:0/1693525293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 0x7f9360106d80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f934c009900 tx=0x7f934c02bc50 comp rx=0 tx=0).stop 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 -- 172.21.15.116:0/1693525293 shutdown_connections 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 --2- 172.21.15.116:0/1693525293 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93601072c0 0x7f9360109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 --2- 172.21.15.116:0/1693525293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 0x7f9360106d80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 --2- 172.21.15.116:0/1693525293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 0x7f9360104450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 -- 172.21.15.116:0/1693525293 >> 172.21.15.116:0/1693525293 conn(0x7f93600fba70 msgr2=0x7f93600fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 -- 172.21.15.116:0/1693525293 shutdown_connections 2024-03-04T07:39:42.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.047+0000 7f9368800640 1 -- 172.21.15.116:0/1693525293 wait complete. 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.048+0000 7f9368800640 1 Processor -- start 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.048+0000 7f9368800640 1 -- start start 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9368800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 0x7f9360197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9368800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 0x7f9360197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9368800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93601072c0 0x7f936019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9368800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936010d290 con 0x7f9360102060 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9368800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936010d110 con 0x7f9360104990 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9368800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936010d410 con 0x7f93601072c0 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 0x7f9360197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 0x7f9360197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 0x7f9360197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45042/0 (socket says 172.21.15.116:45042) 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365e00640 1 -- 172.21.15.116:0/180879411 learned_addr learned my addr 172.21.15.116:0/180879411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9366800640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93601072c0 0x7f936019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365400640 1 -- 172.21.15.116:0/180879411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93601072c0 msgr2=0x7f936019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365400640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93601072c0 0x7f936019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365400640 1 -- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 msgr2=0x7f9360197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365400640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 0x7f9360197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365400640 1 -- 172.21.15.116:0/180879411 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936019f560 con 0x7f9360104990 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.049+0000 7f9365e00640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 0x7f9360197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.050+0000 7f9365400640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 0x7f9360197e80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f934c009230 tx=0x7f934c02c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.050+0000 7f935aa00640 1 -- 172.21.15.116:0/180879411 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c041070 con 0x7f9360104990 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.050+0000 7f935aa00640 1 -- 172.21.15.116:0/180879411 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f934c0054e0 con 0x7f9360104990 2024-03-04T07:39:42.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.050+0000 7f935aa00640 1 -- 172.21.15.116:0/180879411 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c034440 con 0x7f9360104990 2024-03-04T07:39:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.050+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936019f7f0 con 0x7f9360104990 2024-03-04T07:39:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.050+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936019fcc0 con 0x7f9360104990 2024-03-04T07:39:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.051+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93600fd560 con 0x7f9360104990 2024-03-04T07:39:42.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.053+0000 7f935aa00640 1 -- 172.21.15.116:0/180879411 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f934c044070 con 0x7f9360104990 2024-03-04T07:39:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.054+0000 7f935aa00640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93480a1d80 0x7f93480a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.054+0000 7f935aa00640 1 -- 172.21.15.116:0/180879411 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f934c0ef7a0 con 0x7f9360104990 2024-03-04T07:39:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.054+0000 7f9365e00640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93480a1d80 0x7f93480a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.055+0000 7f9365e00640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93480a1d80 0x7f93480a4240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9354011550 tx=0x7f9354005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.058+0000 7f935aa00640 1 -- 172.21.15.116:0/180879411 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f934c0bc580 con 0x7f9360104990 2024-03-04T07:39:42.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:41 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1829590358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:41 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/513828062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:39:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.245+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93600621d0 con 0x7f9360104990 2024-03-04T07:39:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.245+0000 7f935aa00640 1 -- 172.21.15.116:0/180879411 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f934c0dc0c0 con 0x7f9360104990 2024-03-04T07:39:42.245 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:39:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93480a1d80 msgr2=0x7f93480a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93480a1d80 0x7f93480a4240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9354011550 tx=0x7f9354005f90 comp rx=0 tx=0).stop 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 msgr2=0x7f9360197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 0x7f9360197e80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f934c009230 tx=0x7f934c02c6e0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 shutdown_connections 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f93480a1d80 0x7f93480a4240 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93601072c0 0x7f936019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9360104990 0x7f9360197e80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 --2- 172.21.15.116:0/180879411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9360102060 0x7f9360197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.248+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 >> 172.21.15.116:0/180879411 conn(0x7f93600fba70 msgr2=0x7f9360102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.249+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 shutdown_connections 2024-03-04T07:39:42.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.249+0000 7f9368800640 1 -- 172.21.15.116:0/180879411 wait complete. 2024-03-04T07:39:42.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 -- 172.21.15.116:0/446741831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb8107c10 msgr2=0x7f9cb8108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 --2- 172.21.15.116:0/446741831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb8107c10 0x7f9cb8108030 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009900 tx=0x7f9ca802bc30 comp rx=0 tx=0).stop 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 -- 172.21.15.116:0/446741831 shutdown_connections 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 --2- 172.21.15.116:0/446741831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9cb81097c0 0x7f9cb8110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 --2- 172.21.15.116:0/446741831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8108e00 0x7f9cb8109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 --2- 172.21.15.116:0/446741831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb8107c10 0x7f9cb8108030 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 -- 172.21.15.116:0/446741831 >> 172.21.15.116:0/446741831 conn(0x7f9cb8076be0 msgr2=0x7f9cb8079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:42.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 -- 172.21.15.116:0/446741831 shutdown_connections 2024-03-04T07:39:42.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.471+0000 7f9cbf400640 1 -- 172.21.15.116:0/446741831 wait complete. 2024-03-04T07:39:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 Processor -- start 2024-03-04T07:39:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 -- start start 2024-03-04T07:39:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8107c10 0x7f9cb819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9cb8108e00 0x7f9cb819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb81097c0 0x7f9cb81a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb81138b0 con 0x7f9cb8107c10 2024-03-04T07:39:42.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8113730 con 0x7f9cb81097c0 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.472+0000 7f9cbf400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8113a30 con 0x7f9cb8108e00 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8107c10 0x7f9cb819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb81097c0 0x7f9cb81a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb81097c0 0x7f9cb81a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34900/0 (socket says 172.21.15.116:34900) 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8107c10 0x7f9cb819def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45046/0 (socket says 172.21.15.116:45046) 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 -- 172.21.15.116:0/3520892726 learned_addr learned my addr 172.21.15.116:0/3520892726 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cafe00640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9cb8108e00 0x7f9cb819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 -- 172.21.15.116:0/3520892726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9cb8108e00 msgr2=0x7f9cb819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9cb8108e00 0x7f9cb819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 -- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8107c10 msgr2=0x7f9cb819def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8107c10 0x7f9cb819def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 -- 172.21.15.116:0/3520892726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb81a5b10 con 0x7f9cb81097c0 2024-03-04T07:39:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbca00640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8107c10 0x7f9cb819def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.473+0000 7f9cbd400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb81097c0 0x7f9cb81a54a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000eaa0 tx=0x7f9cb000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.474+0000 7f9cb6000640 1 -- 172.21.15.116:0/3520892726 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000cdb0 con 0x7f9cb81097c0 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.474+0000 7f9cb6000640 1 -- 172.21.15.116:0/3520892726 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cb0005020 con 0x7f9cb81097c0 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.474+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb81a5e00 con 0x7f9cb81097c0 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.474+0000 7f9cb6000640 1 -- 172.21.15.116:0/3520892726 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb00116b0 con 0x7f9cb81097c0 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.474+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb81a6270 con 0x7f9cb81097c0 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.475+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb810ef30 con 0x7f9cb81097c0 2024-03-04T07:39:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.477+0000 7f9cb6000640 1 -- 172.21.15.116:0/3520892726 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9cb0005560 con 0x7f9cb81097c0 2024-03-04T07:39:42.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.478+0000 7f9cb6000640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9c9c0a1d80 0x7f9c9c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.478+0000 7f9cb6000640 1 -- 172.21.15.116:0/3520892726 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9cb00cdfc0 con 0x7f9cb81097c0 2024-03-04T07:39:42.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.478+0000 7f9cbca00640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9c9c0a1d80 0x7f9c9c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.479+0000 7f9cbca00640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9c9c0a1d80 0x7f9c9c0a4240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9ca802c170 tx=0x7f9ca802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:42.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.482+0000 7f9cb6000640 1 -- 172.21.15.116:0/3520892726 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9cb0016030 con 0x7f9cb81097c0 2024-03-04T07:39:42.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.719+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb802cff0 con 0x7f9cb81097c0 2024-03-04T07:39:42.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.720+0000 7f9cb6000640 1 -- 172.21.15.116:0/3520892726 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f9cb009ad60 con 0x7f9cb81097c0 2024-03-04T07:39:42.720 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:42.720 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:39:42.721 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:39:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9c9c0a1d80 msgr2=0x7f9c9c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9c9c0a1d80 0x7f9c9c0a4240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9ca802c170 tx=0x7f9ca802df90 comp rx=0 tx=0).stop 2024-03-04T07:39:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb81097c0 msgr2=0x7f9cb81a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb81097c0 0x7f9cb81a54a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000eaa0 tx=0x7f9cb000ef60 comp rx=0 tx=0).stop 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 shutdown_connections 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9cb81097c0 0x7f9cb81a54a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9c9c0a1d80 0x7f9c9c0a4240 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9cb8108e00 0x7f9cb819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 --2- 172.21.15.116:0/3520892726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb8107c10 0x7f9cb819def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 >> 172.21.15.116:0/3520892726 conn(0x7f9cb8076be0 msgr2=0x7f9cb810c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 shutdown_connections 2024-03-04T07:39:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.723+0000 7f9cbf400640 1 -- 172.21.15.116:0/3520892726 wait complete. 2024-03-04T07:39:42.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.949+0000 7f5d8f000640 1 -- 172.21.15.116:0/113257960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 msgr2=0x7f5d8810bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.949+0000 7f5d8f000640 1 --2- 172.21.15.116:0/113257960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d8810bc50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009ca0 tx=0x7f5d8002c100 comp rx=0 tx=0).stop 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.950+0000 7f5d8f000640 1 -- 172.21.15.116:0/113257960 shutdown_connections 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.950+0000 7f5d8f000640 1 --2- 172.21.15.116:0/113257960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d8810bc50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.950+0000 7f5d8f000640 1 --2- 172.21.15.116:0/113257960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d88104a00 0x7f5d88104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.950+0000 7f5d8f000640 1 --2- 172.21.15.116:0/113257960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d88103810 0x7f5d88103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.950+0000 7f5d8f000640 1 -- 172.21.15.116:0/113257960 >> 172.21.15.116:0/113257960 conn(0x7f5d880fed80 msgr2=0x7f5d881011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:42.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.950+0000 7f5d8f000640 1 -- 172.21.15.116:0/113257960 shutdown_connections 2024-03-04T07:39:42.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.950+0000 7f5d8f000640 1 -- 172.21.15.116:0/113257960 wait complete. 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 Processor -- start 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 -- start start 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d88103810 0x7f5d88199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d88104a00 0x7f5d88199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d881a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8810f490 con 0x7f5d88104a00 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8810f310 con 0x7f5d881053c0 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8f000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8810f610 con 0x7f5d88103810 2024-03-04T07:39:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d7fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d88104a00 0x7f5d88199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d881a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d881a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34906/0 (socket says 172.21.15.116:34906) 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d7fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d88104a00 0x7f5d88199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45058/0 (socket says 172.21.15.116:45058) 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d8d000640 1 -- 172.21.15.116:0/3867401089 learned_addr learned my addr 172.21.15.116:0/3867401089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.951+0000 7f5d87e00640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d88103810 0x7f5d88199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8d000640 1 -- 172.21.15.116:0/3867401089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d88103810 msgr2=0x7f5d88199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8d000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d88103810 0x7f5d88199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8d000640 1 -- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d88104a00 msgr2=0x7f5d88199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8d000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d88104a00 0x7f5d88199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8d000640 1 -- 172.21.15.116:0/3867401089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d881a16d0 con 0x7f5d881053c0 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8d000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d881a1060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d80037040 tx=0x7f5d80035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d85600640 1 -- 172.21.15.116:0/3867401089 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80039040 con 0x7f5d881053c0 2024-03-04T07:39:42.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d85600640 1 -- 172.21.15.116:0/3867401089 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d80044070 con 0x7f5d881053c0 2024-03-04T07:39:42.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d881a1960 con 0x7f5d881053c0 2024-03-04T07:39:42.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d85600640 1 -- 172.21.15.116:0/3867401089 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80004500 con 0x7f5d881053c0 2024-03-04T07:39:42.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.952+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d881a1e50 con 0x7f5d881053c0 2024-03-04T07:39:42.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.953+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8810aad0 con 0x7f5d881053c0 2024-03-04T07:39:42.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.955+0000 7f5d85600640 1 -- 172.21.15.116:0/3867401089 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5d800046a0 con 0x7f5d881053c0 2024-03-04T07:39:42.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.956+0000 7f5d85600640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d6c0a1d80 0x7f5d6c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:42.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.957+0000 7f5d85600640 1 -- 172.21.15.116:0/3867401089 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5d800f04f0 con 0x7f5d881053c0 2024-03-04T07:39:42.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.957+0000 7f5d87e00640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d6c0a1d80 0x7f5d6c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:42.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.957+0000 7f5d87e00640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d6c0a1d80 0x7f5d6c0a4240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5d88104860 tx=0x7f5d78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:42.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:42.960+0000 7f5d85600640 1 -- 172.21.15.116:0/3867401089 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5d800bd8e0 con 0x7f5d881053c0 2024-03-04T07:39:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:42 smithi116 ceph-mon[32150]: pgmap v369: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:42 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/377055655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:39:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:42 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/180879411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:39:43.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:42 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3520892726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:42 smithi152 ceph-mon[37154]: pgmap v369: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:42 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/377055655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:39:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:42 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/180879411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:39:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:42 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3520892726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:43.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:42 smithi133 ceph-mon[36505]: pgmap v369: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:43.126 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:42 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/377055655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:39:43.127 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:42 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/180879411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:39:43.127 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:42 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3520892726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:43.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.159+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d880621d0 con 0x7f5d881053c0 2024-03-04T07:39:43.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.160+0000 7f5d85600640 1 -- 172.21.15.116:0/3867401089 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f5d800bd1e0 con 0x7f5d881053c0 2024-03-04T07:39:43.162 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:43.162 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:39:43.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.163+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d6c0a1d80 msgr2=0x7f5d6c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.163+0000 7f5d8f000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d6c0a1d80 0x7f5d6c0a4240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5d88104860 tx=0x7f5d78005f90 comp rx=0 tx=0).stop 2024-03-04T07:39:43.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.163+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 msgr2=0x7f5d881a1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.163+0000 7f5d8f000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d881a1060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d80037040 tx=0x7f5d80035d90 comp rx=0 tx=0).stop 2024-03-04T07:39:43.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 shutdown_connections 2024-03-04T07:39:43.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d881053c0 0x7f5d881a1060 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d88104a00 0x7f5d88199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d6c0a1d80 0x7f5d6c0a4240 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 --2- 172.21.15.116:0/3867401089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d88103810 0x7f5d88199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 >> 172.21.15.116:0/3867401089 conn(0x7f5d880fed80 msgr2=0x7f5d88107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 shutdown_connections 2024-03-04T07:39:43.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.164+0000 7f5d8f000640 1 -- 172.21.15.116:0/3867401089 wait complete. 2024-03-04T07:39:43.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 -- 172.21.15.116:0/4028535275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 msgr2=0x7fab681089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 --2- 172.21.15.116:0/4028535275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 0x7fab681089b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fab60009900 tx=0x7fab6002bc50 comp rx=0 tx=0).stop 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 -- 172.21.15.116:0/4028535275 shutdown_connections 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 --2- 172.21.15.116:0/4028535275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 0x7fab681089b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 --2- 172.21.15.116:0/4028535275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab68101760 0x7fab68101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 --2- 172.21.15.116:0/4028535275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 0x7fab68100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 -- 172.21.15.116:0/4028535275 >> 172.21.15.116:0/4028535275 conn(0x7fab680fbb00 msgr2=0x7fab680fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 -- 172.21.15.116:0/4028535275 shutdown_connections 2024-03-04T07:39:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.397+0000 7fab6f600640 1 -- 172.21.15.116:0/4028535275 wait complete. 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 Processor -- start 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 -- start start 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 0x7fab6810aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab68101760 0x7fab68109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 0x7fab68109a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6810d2a0 con 0x7fab68100570 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6810d120 con 0x7fab68102120 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.398+0000 7fab6f600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6810d420 con 0x7fab68101760 2024-03-04T07:39:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 0x7fab6810aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 0x7fab68109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 0x7fab6810aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45084/0 (socket says 172.21.15.116:45084) 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 -- 172.21.15.116:0/849521365 learned_addr learned my addr 172.21.15.116:0/849521365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab67e00640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab68101760 0x7fab68109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 -- 172.21.15.116:0/849521365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab68101760 msgr2=0x7fab68109530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab68101760 0x7fab68109530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 -- 172.21.15.116:0/849521365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 msgr2=0x7fab68109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 0x7fab68109a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 -- 172.21.15.116:0/849521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68109fb0 con 0x7fab68100570 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6d600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 0x7fab68109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.399+0000 7fab6cc00640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 0x7fab6810aef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fab5800ebc0 tx=0x7fab5800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.400+0000 7fab65600640 1 -- 172.21.15.116:0/849521365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58003b70 con 0x7fab68100570 2024-03-04T07:39:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.400+0000 7fab65600640 1 -- 172.21.15.116:0/849521365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab58003d10 con 0x7fab68100570 2024-03-04T07:39:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.400+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6810a2a0 con 0x7fab68100570 2024-03-04T07:39:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.400+0000 7fab65600640 1 -- 172.21.15.116:0/849521365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58019980 con 0x7fab68100570 2024-03-04T07:39:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.400+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab681a6360 con 0x7fab68100570 2024-03-04T07:39:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.401+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab681077c0 con 0x7fab68100570 2024-03-04T07:39:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.403+0000 7fab65600640 1 -- 172.21.15.116:0/849521365 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fab58019b20 con 0x7fab68100570 2024-03-04T07:39:43.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.404+0000 7fab65600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab4c0a1d80 0x7fab4c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.404+0000 7fab65600640 1 -- 172.21.15.116:0/849521365 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fab580ce6a0 con 0x7fab68100570 2024-03-04T07:39:43.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.404+0000 7fab67e00640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab4c0a1d80 0x7fab4c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.405+0000 7fab67e00640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab4c0a1d80 0x7fab4c0a4240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab6810ac60 tx=0x7fab5000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.407+0000 7fab65600640 1 -- 172.21.15.116:0/849521365 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fab5809bb20 con 0x7fab68100570 2024-03-04T07:39:43.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.642+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab6802cff0 con 0x7fab68100570 2024-03-04T07:39:43.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.644+0000 7fab65600640 1 -- 172.21.15.116:0/849521365 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fab5809b420 con 0x7fab68100570 2024-03-04T07:39:43.644 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:39:43.644 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:39:43.644 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:39:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab4c0a1d80 msgr2=0x7fab4c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab4c0a1d80 0x7fab4c0a4240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab6810ac60 tx=0x7fab5000b410 comp rx=0 tx=0).stop 2024-03-04T07:39:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 msgr2=0x7fab6810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 0x7fab6810aef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fab5800ebc0 tx=0x7fab5800c6a0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 shutdown_connections 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab68102120 0x7fab68109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab4c0a1d80 0x7fab4c0a4240 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab68101760 0x7fab68109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 --2- 172.21.15.116:0/849521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100570 0x7fab6810aef0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.646+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 >> 172.21.15.116:0/849521365 conn(0x7fab680fbb00 msgr2=0x7fab68104990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.647+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 shutdown_connections 2024-03-04T07:39:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.647+0000 7fab6f600640 1 -- 172.21.15.116:0/849521365 wait complete. 2024-03-04T07:39:43.667 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-04T07:37:54.573948+0000', 'modified': '2024-03-04T07:37:58.240682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24581}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24553': {'gid': 24553, 'name': 'g', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1000386690', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1000386690}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1000386690}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24581': {'gid': 24581, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.133:6839/1914821860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.133:6838', 'nonce': 1914821860}, {'type': 'v1', 'addr': '172.21.15.133:6839', 'nonce': 1914821860}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24581, 'qdb_cluster': [24581]} 2024-03-04T07:39:43.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T07:39:43.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.877+0000 7fe523000640 1 -- 172.21.15.116:0/3456279419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 msgr2=0x7fe51c110f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.877+0000 7fe523000640 1 --2- 172.21.15.116:0/3456279419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c110f10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe514009ca0 tx=0x7fe51402c100 comp rx=0 tx=0).stop 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.878+0000 7fe523000640 1 -- 172.21.15.116:0/3456279419 shutdown_connections 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.878+0000 7fe523000640 1 --2- 172.21.15.116:0/3456279419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c110f10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.878+0000 7fe523000640 1 --2- 172.21.15.116:0/3456279419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51c10c1f0 0x7fe51c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.878+0000 7fe523000640 1 --2- 172.21.15.116:0/3456279419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51c06a3e0 0x7fe51c108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.878+0000 7fe523000640 1 -- 172.21.15.116:0/3456279419 >> 172.21.15.116:0/3456279419 conn(0x7fe51c06e7d0 msgr2=0x7fe51c070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.878+0000 7fe523000640 1 -- 172.21.15.116:0/3456279419 shutdown_connections 2024-03-04T07:39:43.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.878+0000 7fe523000640 1 -- 172.21.15.116:0/3456279419 wait complete. 2024-03-04T07:39:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 Processor -- start 2024-03-04T07:39:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 -- start start 2024-03-04T07:39:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51c06a3e0 0x7fe51c1a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51c10c1f0 0x7fe51c1a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c1a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c114990 con 0x7fe51c10c1f0 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c114810 con 0x7fe51c10eb20 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe523000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c114b10 con 0x7fe51c06a3e0 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe513400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51c10c1f0 0x7fe51c1a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe521000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c1a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe51be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51c06a3e0 0x7fe51c1a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe513400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51c10c1f0 0x7fe51c1a2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45088/0 (socket says 172.21.15.116:45088) 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe51be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51c06a3e0 0x7fe51c1a22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:46856/0 (socket says 172.21.15.116:46856) 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe513400640 1 -- 172.21.15.116:0/1407627034 learned_addr learned my addr 172.21.15.116:0/1407627034 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.879+0000 7fe521000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c1a9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34926/0 (socket says 172.21.15.116:34926) 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe521000640 1 -- 172.21.15.116:0/1407627034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51c06a3e0 msgr2=0x7fe51c1a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe521000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51c06a3e0 0x7fe51c1a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe521000640 1 -- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51c10c1f0 msgr2=0x7fe51c1a2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe521000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51c10c1f0 0x7fe51c1a2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe521000640 1 -- 172.21.15.116:0/1407627034 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c1a9ee0 con 0x7fe51c10eb20 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe521000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c1a9870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe514037040 tx=0x7fe514035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe519600640 1 -- 172.21.15.116:0/1407627034 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514039040 con 0x7fe51c10eb20 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe519600640 1 -- 172.21.15.116:0/1407627034 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe514044070 con 0x7fe51c10eb20 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51c1aa170 con 0x7fe51c10eb20 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe519600640 1 -- 172.21.15.116:0/1407627034 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe514004350 con 0x7fe51c10eb20 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.880+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51c1aa680 con 0x7fe51c10eb20 2024-03-04T07:39:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.883+0000 7fe519600640 1 -- 172.21.15.116:0/1407627034 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe5140044f0 con 0x7fe51c10eb20 2024-03-04T07:39:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.883+0000 7fe525da7740 1 Processor -- start 2024-03-04T07:39:43.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe525da7740 1 -- start start 2024-03-04T07:39:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe525da7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e041ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe525da7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e04204bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe525da7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042076f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe525da7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04164600 con 0x563e04205260 2024-03-04T07:39:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe525da7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e0410d930 con 0x563e041ff700 2024-03-04T07:39:43.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe525da7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e041606a0 con 0x563e042000f0 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe513400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e041ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe513400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e041ffb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34936/0 (socket says 172.21.15.116:34936) 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe513400640 1 -- 172.21.15.116:0/4041470826 learned_addr learned my addr 172.21.15.116:0/4041470826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe51be00640 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e04204bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe513400640 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.884+0000 7fe519600640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe4fc0a1e00 0x7fe4fc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe513400640 1 -- 172.21.15.116:0/4041470826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 msgr2=0x563e04204bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe513400640 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e04204bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe521000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe4fc0a1e00 0x7fe4fc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe513400640 1 -- 172.21.15.116:0/4041470826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 msgr2=0x563e042076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe513400640 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe519600640 1 -- 172.21.15.116:0/1407627034 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe5140f02c0 con 0x7fe51c10eb20 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe513400640 1 -- 172.21.15.116:0/4041470826 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e03e792d0 con 0x563e041ff700 2024-03-04T07:39:43.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe513400640 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e041ffb20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe504005bc0 tx=0x7fe504009540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58f9ad7d7bf4c0a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe521000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe4fc0a1e00 0x7fe4fc0a42c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe5140324c0 tx=0x7fe514045c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe511600640 1 -- 172.21.15.116:0/4041470826 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50400f7c0 con 0x563e041ff700 2024-03-04T07:39:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe511600640 1 -- 172.21.15.116:0/4041470826 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe50400fe40 con 0x563e041ff700 2024-03-04T07:39:43.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.885+0000 7fe511600640 1 -- 172.21.15.116:0/4041470826 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50401fe30 con 0x563e041ff700 2024-03-04T07:39:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 -- 172.21.15.116:0/4041470826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 msgr2=0x563e041ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e041ffb20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe504005bc0 tx=0x7fe504009540 comp rx=0 tx=0).stop 2024-03-04T07:39:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 -- 172.21.15.116:0/4041470826 shutdown_connections 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042076f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e04204bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 --2- 172.21.15.116:0/4041470826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e041ffb20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 -- 172.21.15.116:0/4041470826 >> 172.21.15.116:0/4041470826 conn(0x563e041fac70 msgr2=0x563e041fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 -- 172.21.15.116:0/4041470826 shutdown_connections 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.886+0000 7fe525da7740 1 -- 172.21.15.116:0/4041470826 wait complete. 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 -- start start 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 Processor -- start 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e041ff700 0x563e04209d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e04205260 0x563e04211270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04208210 con 0x563e041ff700 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04120d90 con 0x563e04205260 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04208390 con 0x563e042000f0 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.887+0000 7fe525da7740 10 client.0 _ll_register_callbacks cb 0x563e0415c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e041ff700 0x563e04209d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e041ff700 0x563e04209d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45100/0 (socket says 172.21.15.116:45100) 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 -- 172.21.15.116:0/3289669896 learned_addr learned my addr 172.21.15.116:0/3289669896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe51be00640 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e04205260 0x563e04211270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 -- 172.21.15.116:0/3289669896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 msgr2=0x563e0420a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 -- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e04205260 msgr2=0x563e04211270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e04205260 0x563e04211270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 -- 172.21.15.116:0/3289669896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e03f436b0 con 0x563e041ff700 2024-03-04T07:39:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe513400640 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e041ff700 0x563e04209d60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe504005b90 tx=0x7fe504011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61aa1c4747a62df2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.888+0000 7fe50a000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe50a000640 1 -- 172.21.15.116:0/3289669896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50401f440 con 0x563e041ff700 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe50a000640 1 -- 172.21.15.116:0/3289669896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe504004020 con 0x563e041ff700 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe50400fe40 con 0x563e041ff700 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe50a000640 1 -- 172.21.15.116:0/3289669896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe504033050 con 0x563e041ff700 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe525da7740 10 client.14874 fetch_fsmap learned FSMap version 19 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563e042145e0 con 0x563e041ff700 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe50a000640 1 -- 172.21.15.116:0/3289669896 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe50400fe40 con 0x563e041ff700 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.889+0000 7fe50a000640 1 -- 172.21.15.116:0/3289669896 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe50401d020 con 0x563e041ff700 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe525da7740 10 client.14874 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe525da7740 10 client.14874 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe525da7740 20 client.14874 populate_metadata read hostname 'smithi116' 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe525da7740 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04215a40 0x563e0421ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe525da7740 4 client.14874 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:39:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e04171b40 con 0x563e04215a40 2024-03-04T07:39:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe51be00640 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04215a40 0x563e0421ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe51be00640 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04215a40 0x563e0421ab50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a08fbab67619cc7e server_cookie=285bbb1f1b45a0a8 in_seq=0 out_seq=0 2024-03-04T07:39:43.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.890+0000 7fe50a000640 10 client.14874 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.891+0000 7fe50a000640 1 -- 172.21.15.116:0/3289669896 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563e04171b40 con 0x563e04215a40 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.891+0000 7fe50a000640 10 client.14874 handle_command_reply: tid=0 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.891+0000 7fe525da7740 1 client.14874 shutdown 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e041ff700 msgr2=0x563e04209d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e041ff700 0x563e04209d60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe504005b90 tx=0x7fe504011660 comp rx=0 tx=0).stop 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 shutdown_connections 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04215a40 0x563e0421ab50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e04205260 0x563e04211270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 --2- 172.21.15.116:0/3289669896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e041ff700 0x563e04209d60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe50a000640 0 client.14874 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:39:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 >> 172.21.15.116:0/3289669896 conn(0x563e041fac70 msgr2=0x563e041f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 shutdown_connections 2024-03-04T07:39:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 1 -- 172.21.15.116:0/3289669896 wait complete. 2024-03-04T07:39:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.892+0000 7fe525da7740 20 client.14874 trim_cache size 0 max 16384 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 Processor -- start 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 -- start start 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e041ff700 0x563e041f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e042000f0 0x563e04211200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e041fb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04215ef0 con 0x563e04205260 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04216550 con 0x563e042000f0 2024-03-04T07:39:43.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe525da7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04216bb0 con 0x563e041ff700 2024-03-04T07:39:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe51be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e042000f0 0x563e04211200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe51be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e042000f0 0x563e04211200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34948/0 (socket says 172.21.15.116:34948) 2024-03-04T07:39:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe51be00640 1 -- 172.21.15.116:0/271287805 learned_addr learned my addr 172.21.15.116:0/271287805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.910+0000 7fe513400640 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e041fb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe513400640 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e041ff700 0x563e041f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe51be00640 1 -- 172.21.15.116:0/271287805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e041ff700 msgr2=0x563e041f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe51be00640 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e041ff700 0x563e041f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe51be00640 1 -- 172.21.15.116:0/271287805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 msgr2=0x563e041fb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe51be00640 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e041fb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe51be00640 1 -- 172.21.15.116:0/271287805 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e042049d0 con 0x563e042000f0 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe513400640 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e041ff700 0x563e041f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe51be00640 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e042000f0 0x563e04211200 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x563e04200b60 tx=0x7fe50c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30d04f5434cf9fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe510c00640 1 -- 172.21.15.116:0/271287805 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c023070 con 0x563e042000f0 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe510c00640 1 -- 172.21.15.116:0/271287805 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe50c005500 con 0x563e042000f0 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.911+0000 7fe510c00640 1 -- 172.21.15.116:0/271287805 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c00fd40 con 0x563e042000f0 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 -- 172.21.15.116:0/271287805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e042000f0 msgr2=0x563e04211200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e042000f0 0x563e04211200 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x563e04200b60 tx=0x7fe50c005f90 comp rx=0 tx=0).stop 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 -- 172.21.15.116:0/271287805 shutdown_connections 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e041fb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e042000f0 0x563e04211200 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 --2- 172.21.15.116:0/271287805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e041ff700 0x563e041f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 -- 172.21.15.116:0/271287805 >> 172.21.15.116:0/271287805 conn(0x563e041fac70 msgr2=0x563e04207650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 -- 172.21.15.116:0/271287805 shutdown_connections 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 -- 172.21.15.116:0/271287805 wait complete. 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.912+0000 7fe525da7740 1 -- start start 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 1 Processor -- start 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e0420cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04217870 con 0x563e04205260 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04217210 con 0x563e041ff700 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e04217ed0 con 0x563e042000f0 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe525da7740 10 client.0 _ll_register_callbacks cb 0x7fe504011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe513400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e0420cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe513400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e0420cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34964/0 (socket says 172.21.15.116:34964) 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe513400640 1 -- 172.21.15.116:0/1609807613 learned_addr learned my addr 172.21.15.116:0/1609807613 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe513400640 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.913+0000 7fe51be00640 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe513400640 1 -- 172.21.15.116:0/1609807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 msgr2=0x563e0420d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe513400640 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe513400640 1 -- 172.21.15.116:0/1609807613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 msgr2=0x563e042100f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe513400640 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042100f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe513400640 1 -- 172.21.15.116:0/1609807613 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e042106e0 con 0x563e041ff700 2024-03-04T07:39:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe513400640 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e0420cb90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5040051c0 tx=0x7fe504009dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5711fccb5fb8d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe50a000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe50a000640 1 -- 172.21.15.116:0/1609807613 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50401f940 con 0x563e041ff700 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe50a000640 1 -- 172.21.15.116:0/1609807613 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe504009370 con 0x563e041ff700 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe50c005500 con 0x563e041ff700 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.914+0000 7fe50a000640 1 -- 172.21.15.116:0/1609807613 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe504048050 con 0x563e041ff700 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 10 client.24986 fetch_fsmap learned FSMap version 19 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563e04271e30 con 0x563e041ff700 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe50a000640 1 -- 172.21.15.116:0/1609807613 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe50c005500 con 0x563e041ff700 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe50a000640 1 -- 172.21.15.116:0/1609807613 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe50401d070 con 0x563e041ff700 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 10 client.24986 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 10 client.24986 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 20 client.24986 populate_metadata read hostname 'smithi116' 2024-03-04T07:39:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04276400 0x563e04272000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 4 client.24986 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T07:39:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563e0416be90 con 0x563e04276400 2024-03-04T07:39:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.915+0000 7fe51be00640 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04276400 0x563e04272000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe51be00640 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04276400 0x563e04272000 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78fa8af1ad87c44b server_cookie=37f94ea1414e1be6 in_seq=0 out_seq=0 2024-03-04T07:39:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe50a000640 10 client.24986 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:39:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe50a000640 1 -- 172.21.15.116:0/1609807613 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x563e0416be90 con 0x563e04276400 2024-03-04T07:39:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe50a000640 10 client.24986 handle_command_reply: tid=0 2024-03-04T07:39:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 client.24986 shutdown 2024-03-04T07:39:43.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 msgr2=0x563e0420cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e0420cb90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe5040051c0 tx=0x7fe504009dd0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 shutdown_connections 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563e04276400 0x563e04272000 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e04205260 0x563e042100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e042000f0 0x563e0420d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe50a000640 0 client.24986 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 --2- 172.21.15.116:0/1609807613 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563e041ff700 0x563e0420cb90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 >> 172.21.15.116:0/1609807613 conn(0x563e041fac70 msgr2=0x563e041fb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 shutdown_connections 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 1 -- 172.21.15.116:0/1609807613 wait complete. 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.916+0000 7fe525da7740 20 client.24986 trim_cache size 0 max 16384 2024-03-04T07:39:43.921 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "af2ccd48-d9f8-11ee-95c6-87774f69a715", 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24581, 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 108.770427845, 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 19, 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 116.137905393 2024-03-04T07:39:43.922 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe4fc0a1e00 msgr2=0x7fe4fc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe4fc0a1e00 0x7fe4fc0a42c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe5140324c0 tx=0x7fe514045c40 comp rx=0 tx=0).stop 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 msgr2=0x7fe51c1a9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c1a9870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe514037040 tx=0x7fe514035d10 comp rx=0 tx=0).stop 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 shutdown_connections 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe51c10eb20 0x7fe51c1a9870 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe51c10c1f0 0x7fe51c1a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe4fc0a1e00 0x7fe4fc0a42c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 --2- 172.21.15.116:0/1407627034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51c06a3e0 0x7fe51c1a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 >> 172.21.15.116:0/1407627034 conn(0x7fe51c06e7d0 msgr2=0x7fe51c109550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 shutdown_connections 2024-03-04T07:39:43.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:43.917+0000 7fe523000640 1 -- 172.21.15.116:0/1407627034 wait complete. 2024-03-04T07:39:43.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:43 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3867401089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:43.932 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:43 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/849521365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:43.934 INFO:tasks.cephadm:Setup complete, yielding 2024-03-04T07:39:43.935 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T07:39:43.943 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T07:39:43.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- bash -c 'ceph orch status' 2024-03-04T07:39:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:43 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3867401089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:43 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/849521365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:44.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:43 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3867401089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:39:44.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:43 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/849521365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:39:44.375 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:44.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:44 smithi116 ceph-mon[32150]: pgmap v370: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:44 smithi152 ceph-mon[37154]: pgmap v370: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:45.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:44 smithi133 ceph-mon[36505]: pgmap v370: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:39:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.881+0000 7f0f77600700 1 -- 172.21.15.116:0/1808111138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103300 msgr2=0x7f0f70103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.881+0000 7f0f77600700 1 --2- 172.21.15.116:0/1808111138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103300 0x7f0f70103760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0099c0 tx=0x7f0f5c009cd0 comp rx=0 tx=0).stop 2024-03-04T07:39:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.881+0000 7f0f77600700 1 -- 172.21.15.116:0/1808111138 shutdown_connections 2024-03-04T07:39:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.881+0000 7f0f77600700 1 --2- 172.21.15.116:0/1808111138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70102100 0x7f0f70102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.881+0000 7f0f77600700 1 --2- 172.21.15.116:0/1808111138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f70103ca0 0x7f0f7010a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.881+0000 7f0f77600700 1 --2- 172.21.15.116:0/1808111138 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103300 0x7f0f70103760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.881+0000 7f0f77600700 1 -- 172.21.15.116:0/1808111138 >> 172.21.15.116:0/1808111138 conn(0x7f0f700fd660 msgr2=0x7f0f700ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.882+0000 7f0f77600700 1 -- 172.21.15.116:0/1808111138 shutdown_connections 2024-03-04T07:39:46.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:46 smithi116 ceph-mon[32150]: pgmap v371: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.882+0000 7f0f77600700 1 -- 172.21.15.116:0/1808111138 wait complete. 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.883+0000 7f0f77600700 1 Processor -- start 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.883+0000 7f0f77600700 1 -- start start 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f77600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70102100 0x7f0f7019a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f77600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f70103300 0x7f0f7019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f77600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103ca0 0x7f0f701a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f77600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7010dd20 con 0x7f0f70102100 2024-03-04T07:39:46.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f77600700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7010dba0 con 0x7f0f70103ca0 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f77600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7010dea0 con 0x7f0f70103300 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f74c00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70102100 0x7f0f7019a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f6fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f70103300 0x7f0f7019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f75600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103ca0 0x7f0f701a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f6fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f70103300 0x7f0f7019aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43490/0 (socket says 172.21.15.116:43490) 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.884+0000 7f0f6fe00700 1 -- 172.21.15.116:0/890011022 learned_addr learned my addr 172.21.15.116:0/890011022 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f75600700 1 -- 172.21.15.116:0/890011022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f70103300 msgr2=0x7f0f7019aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f75600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f70103300 0x7f0f7019aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f75600700 1 -- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70102100 msgr2=0x7f0f7019a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f75600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70102100 0x7f0f7019a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:46.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f75600700 1 -- 172.21.15.116:0/890011022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f701a21d0 con 0x7f0f70103ca0 2024-03-04T07:39:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f75600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103ca0 0x7f0f701a1b30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f68007f10 tx=0x7f0f6800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f6d600700 1 -- 172.21.15.116:0/890011022 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f68004db0 con 0x7f0f70103ca0 2024-03-04T07:39:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f6d600700 1 -- 172.21.15.116:0/890011022 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f68003a00 con 0x7f0f70103ca0 2024-03-04T07:39:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f6d600700 1 -- 172.21.15.116:0/890011022 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f68004680 con 0x7f0f70103ca0 2024-03-04T07:39:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.885+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f701a24c0 con 0x7f0f70103ca0 2024-03-04T07:39:46.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.886+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f701a29b0 con 0x7f0f70103ca0 2024-03-04T07:39:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.887+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f70068d80 con 0x7f0f70103ca0 2024-03-04T07:39:46.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.889+0000 7f0f6d600700 1 -- 172.21.15.116:0/890011022 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f0f6801c020 con 0x7f0f70103ca0 2024-03-04T07:39:46.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.891+0000 7f0f6d600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0f580a1e90 0x7f0f580a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:46.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.891+0000 7f0f74c00700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0f580a1e90 0x7f0f580a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:46.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.891+0000 7f0f6d600700 1 -- 172.21.15.116:0/890011022 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0f680cd620 con 0x7f0f70103ca0 2024-03-04T07:39:46.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.892+0000 7f0f74c00700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0f580a1e90 0x7f0f580a4350 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009680 tx=0x7f0f64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:46.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:46.895+0000 7f0f6d600700 1 -- 172.21.15.116:0/890011022 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0f68010070 con 0x7f0f70103ca0 2024-03-04T07:39:47.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:46 smithi152 ceph-mon[37154]: pgmap v371: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:46 smithi133 ceph-mon[36505]: pgmap v371: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.141+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f70075a10 con 0x7f0f580a1e90 2024-03-04T07:39:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.143+0000 7f0f6d600700 1 -- 172.21.15.116:0/890011022 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0f70075a10 con 0x7f0f580a1e90 2024-03-04T07:39:47.143 INFO:teuthology.orchestra.run.smithi116.stdout:Backend: cephadm 2024-03-04T07:39:47.144 INFO:teuthology.orchestra.run.smithi116.stdout:Available: Yes 2024-03-04T07:39:47.144 INFO:teuthology.orchestra.run.smithi116.stdout:Paused: No 2024-03-04T07:39:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0f580a1e90 msgr2=0x7f0f580a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0f580a1e90 0x7f0f580a4350 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009680 tx=0x7f0f64009260 comp rx=0 tx=0).stop 2024-03-04T07:39:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103ca0 msgr2=0x7f0f701a1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103ca0 0x7f0f701a1b30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f68007f10 tx=0x7f0f6800d3b0 comp rx=0 tx=0).stop 2024-03-04T07:39:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 shutdown_connections 2024-03-04T07:39:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f70102100 0x7f0f7019a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:47.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0f580a1e90 0x7f0f580a4350 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:47.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0f70103ca0 0x7f0f701a1b30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:47.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 --2- 172.21.15.116:0/890011022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f70103300 0x7f0f7019aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:47.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 >> 172.21.15.116:0/890011022 conn(0x7f0f700fd660 msgr2=0x7f0f700742f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:47.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.147+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 shutdown_connections 2024-03-04T07:39:47.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:47.148+0000 7f0f77600700 1 -- 172.21.15.116:0/890011022 wait complete. 2024-03-04T07:39:48.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-04T07:39:48.508 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:48.998 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:48 smithi116 ceph-mon[32150]: from='client.24992 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:48.999 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:48 smithi116 ceph-mon[32150]: pgmap v372: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:49.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:48 smithi152 ceph-mon[37154]: from='client.24992 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:49.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:48 smithi152 ceph-mon[37154]: pgmap v372: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:48 smithi133 ceph-mon[36505]: from='client.24992 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:48 smithi133 ceph-mon[36505]: pgmap v372: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 -- 172.21.15.116:0/36558126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105440 msgr2=0x7fed701058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:50.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 --2- 172.21.15.116:0/36558126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105440 0x7fed701058a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0099c0 tx=0x7fed5c009cd0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 -- 172.21.15.116:0/36558126 shutdown_connections 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 --2- 172.21.15.116:0/36558126 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed70105de0 0x7fed7010c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 --2- 172.21.15.116:0/36558126 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70104240 0x7fed70104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 --2- 172.21.15.116:0/36558126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105440 0x7fed701058a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 -- 172.21.15.116:0/36558126 >> 172.21.15.116:0/36558126 conn(0x7fed700ff7e0 msgr2=0x7fed70101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 -- 172.21.15.116:0/36558126 shutdown_connections 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.682+0000 7fed6fe00700 1 -- 172.21.15.116:0/36558126 wait complete. 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.683+0000 7fed6fe00700 1 Processor -- start 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.683+0000 7fed6fe00700 1 -- start start 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed70104240 0x7fed7019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70105440 0x7fed7019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6fe00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105de0 0x7fed701a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:50.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6fe00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7010fe70 con 0x7fed70105de0 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6fe00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7010fcf0 con 0x7fed70105440 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6fe00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7010fff0 con 0x7fed70104240 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70105440 0x7fed7019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70105440 0x7fed7019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35694/0 (socket says 172.21.15.116:35694) 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 -- 172.21.15.116:0/3148884529 learned_addr learned my addr 172.21.15.116:0/3148884529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6f400700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105de0 0x7fed701a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6ea00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed70104240 0x7fed7019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 -- 172.21.15.116:0/3148884529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed70104240 msgr2=0x7fed7019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:50.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed70104240 0x7fed7019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 -- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105de0 msgr2=0x7fed701a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105de0 0x7fed701a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.684+0000 7fed6e000700 1 -- 172.21.15.116:0/3148884529 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed701a2060 con 0x7fed70105440 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.685+0000 7fed6e000700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70105440 0x7fed7019a940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fed5c003790 tx=0x7fed5c00bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.685+0000 7fed67400700 1 -- 172.21.15.116:0/3148884529 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c009e60 con 0x7fed70105440 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.685+0000 7fed67400700 1 -- 172.21.15.116:0/3148884529 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed5c00f460 con 0x7fed70105440 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.685+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed701a22f0 con 0x7fed70105440 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.685+0000 7fed67400700 1 -- 172.21.15.116:0/3148884529 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c017650 con 0x7fed70105440 2024-03-04T07:39:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.685+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701a27a0 con 0x7fed70105440 2024-03-04T07:39:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.686+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed70068d80 con 0x7fed70105440 2024-03-04T07:39:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.688+0000 7fed67400700 1 -- 172.21.15.116:0/3148884529 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fed5c00f980 con 0x7fed70105440 2024-03-04T07:39:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.689+0000 7fed67400700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed540a1e90 0x7fed540a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.689+0000 7fed67400700 1 -- 172.21.15.116:0/3148884529 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed5c0ceaa0 con 0x7fed70105440 2024-03-04T07:39:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.690+0000 7fed6ea00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed540a1e90 0x7fed540a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.690+0000 7fed6ea00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed540a1e90 0x7fed540a4350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fed701052a0 tx=0x7fed60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:50.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.693+0000 7fed67400700 1 -- 172.21.15.116:0/3148884529 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fed5c013070 con 0x7fed70105440 2024-03-04T07:39:50.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.929+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fed70062500 con 0x7fed540a1e90 2024-03-04T07:39:50.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.955+0000 7fed67400700 1 -- 172.21.15.116:0/3148884529 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7fed70062500 con 0x7fed540a1e90 2024-03-04T07:39:50.955 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T07:39:50.955 INFO:teuthology.orchestra.run.smithi116.stdout:mds.a smithi116 running (2m) 24s ago 2m 12.2M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a e387a3fde331 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.b smithi133 running (2m) 54s ago 2m 25.6M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a 104f6c82bd6c 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.c smithi152 running (2m) 54s ago 2m 15.3M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a 44d4152152c3 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.d smithi116 running (2m) 24s ago 2m 12.3M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a dcb27b9d2112 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.e smithi133 running (2m) 54s ago 2m 13.7M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a e8b7fc0b65d6 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.f smithi152 running (2m) 54s ago 2m 16.2M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a 15259a754df5 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.g smithi116 running (2m) 24s ago 2m 16.6M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a 6f4a49fbf911 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.h smithi133 running (2m) 54s ago 2m 13.6M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a fca8568dd783 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.i smithi152 running (2m) 54s ago 2m 16.2M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a f793c284997a 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.j smithi116 running (2m) 24s ago 2m 12.4M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a 61a6923b3de4 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.k smithi133 running (2m) 54s ago 2m 16.6M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a da4094da1ead 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mds.l smithi152 running (2m) 54s ago 2m 13.8M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a 48ac649e0e10 2024-03-04T07:39:50.956 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi116 *:9283,8765 running (10m) 24s ago 10m 512M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a dff4e95798d3 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi133 *:8443,8765 running (8m) 54s ago 8m 454M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a e84d6d702ead 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.z smithi152 *:8443,8765 running (8m) 54s ago 8m 452M - 19.0.0-1739-g1bc250c7 f88ffb32aa6a b76a315aee73 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (10m) 24s ago 11m 47.0M 2048M 19.0.0-1739-g1bc250c7 f88ffb32aa6a e0bdece57fe4 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi133 running (8m) 54s ago 8m 42.0M 2048M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 55bc537d1fec 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi152 running (8m) 54s ago 8m 42.9M 2048M 19.0.0-1739-g1bc250c7 f88ffb32aa6a d5d2c117cb36 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (8m) 24s ago 8m 54.2M 4272M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 05fe9353c209 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi133 running (7m) 54s ago 7m 56.8M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 648aba921096 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi152 running (7m) 54s ago 7m 56.9M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 1c35311fb1ee 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (6m) 24s ago 6m 42.1M 4272M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 9dc47f048854 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi133 running (6m) 54s ago 6m 56.4M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a c49399a54bcb 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi152 running (5m) 54s ago 5m 45.7M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 4db1d6eaf80b 2024-03-04T07:39:50.957 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi116 running (5m) 24s ago 5m 50.5M 4272M 19.0.0-1739-g1bc250c7 f88ffb32aa6a c74a69216f07 2024-03-04T07:39:50.958 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi133 running (4m) 54s ago 4m 43.8M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 0a60e45a29ef 2024-03-04T07:39:50.958 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 smithi152 running (4m) 54s ago 4m 42.2M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 7195222f906e 2024-03-04T07:39:50.958 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 smithi116 running (3m) 24s ago 3m 40.4M 4272M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 9f24c844307a 2024-03-04T07:39:50.958 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 smithi133 running (3m) 54s ago 3m 43.0M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 0613f3f15cb9 2024-03-04T07:39:50.958 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 smithi152 running (3m) 54s ago 3m 50.0M 4270M 19.0.0-1739-g1bc250c7 f88ffb32aa6a 82fef648b6ec 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed540a1e90 msgr2=0x7fed540a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed540a1e90 0x7fed540a4350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fed701052a0 tx=0x7fed60009260 comp rx=0 tx=0).stop 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70105440 msgr2=0x7fed7019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70105440 0x7fed7019a940 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fed5c003790 tx=0x7fed5c00bee0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 shutdown_connections 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed70105de0 0x7fed701a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed540a1e90 0x7fed540a4350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed70104240 0x7fed7019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 --2- 172.21.15.116:0/3148884529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed70105440 0x7fed7019a940 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 >> 172.21.15.116:0/3148884529 conn(0x7fed700ff7e0 msgr2=0x7fed701086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:50.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 shutdown_connections 2024-03-04T07:39:50.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:50.958+0000 7fed6fe00700 1 -- 172.21.15.116:0/3148884529 wait complete. 2024-03-04T07:39:51.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:50 smithi133 ceph-mon[36505]: pgmap v373: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:50 smithi116 ceph-mon[32150]: pgmap v373: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:50 smithi152 ceph-mon[37154]: pgmap v373: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:51.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-04T07:39:52.285 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:53.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:52 smithi133 ceph-mon[36505]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:53.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:52 smithi133 ceph-mon[36505]: pgmap v374: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:52 smithi116 ceph-mon[32150]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:52 smithi116 ceph-mon[32150]: pgmap v374: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:52 smithi152 ceph-mon[37154]: from='client.24998 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:52 smithi152 ceph-mon[37154]: pgmap v374: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 -- 172.21.15.116:0/2398116233 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 msgr2=0x7fd2b81078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 --2- 172.21.15.116:0/2398116233 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b81078a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009990 tx=0x7fd2a4009ca0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 -- 172.21.15.116:0/2398116233 shutdown_connections 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 --2- 172.21.15.116:0/2398116233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2b8102b80 0x7fd2b8104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 --2- 172.21.15.116:0/2398116233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2b8107de0 0x7fd2b810a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 --2- 172.21.15.116:0/2398116233 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b81078a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 -- 172.21.15.116:0/2398116233 >> 172.21.15.116:0/2398116233 conn(0x7fd2b80fc590 msgr2=0x7fd2b80fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 -- 172.21.15.116:0/2398116233 shutdown_connections 2024-03-04T07:39:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.523+0000 7fd2bf600700 1 -- 172.21.15.116:0/2398116233 wait complete. 2024-03-04T07:39:54.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 Processor -- start 2024-03-04T07:39:54.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 -- start start 2024-03-04T07:39:54.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2b8102b80 0x7fd2b81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2b8107de0 0x7fd2b819f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b810dd10 con 0x7fd2b8107de0 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b810db90 con 0x7fd2b81054b0 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.525+0000 7fd2bf600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b810de90 con 0x7fd2b8102b80 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b8198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35714/0 (socket says 172.21.15.116:35714) 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 -- 172.21.15.116:0/3929552578 learned_addr learned my addr 172.21.15.116:0/3929552578 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2bd600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2b8107de0 0x7fd2b819f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2bcc00700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2b8102b80 0x7fd2b81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 -- 172.21.15.116:0/3929552578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2b8102b80 msgr2=0x7fd2b81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2b8102b80 0x7fd2b81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 -- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2b8107de0 msgr2=0x7fd2b819f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2b8107de0 0x7fd2b819f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 -- 172.21.15.116:0/3929552578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b81a0020 con 0x7fd2b81054b0 2024-03-04T07:39:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2bd600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2b8107de0 0x7fd2b819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:39:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.526+0000 7fd2b7e00700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b8198900 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400b2e0 tx=0x7fd2a400bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.527+0000 7fd2b5600700 1 -- 172.21.15.116:0/3929552578 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4026070 con 0x7fd2b81054b0 2024-03-04T07:39:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.527+0000 7fd2b5600700 1 -- 172.21.15.116:0/3929552578 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2a4003b20 con 0x7fd2b81054b0 2024-03-04T07:39:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.527+0000 7fd2b5600700 1 -- 172.21.15.116:0/3929552578 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4021650 con 0x7fd2b81054b0 2024-03-04T07:39:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.527+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b81a02b0 con 0x7fd2b81054b0 2024-03-04T07:39:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.527+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b81a06b0 con 0x7fd2b81054b0 2024-03-04T07:39:54.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.528+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b8068d80 con 0x7fd2b81054b0 2024-03-04T07:39:54.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.530+0000 7fd2b5600700 1 -- 172.21.15.116:0/3929552578 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd2a4029070 con 0x7fd2b81054b0 2024-03-04T07:39:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.531+0000 7fd2b5600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd2a00a1e90 0x7fd2a00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.531+0000 7fd2b5600700 1 -- 172.21.15.116:0/3929552578 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd2a40ced90 con 0x7fd2b81054b0 2024-03-04T07:39:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.531+0000 7fd2bcc00700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd2a00a1e90 0x7fd2a00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:54.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.532+0000 7fd2bcc00700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd2a00a1e90 0x7fd2a00a4350 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009bf0 tx=0x7fd2ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.538+0000 7fd2b5600700 1 -- 172.21.15.116:0/3929552578 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd2a405d280 con 0x7fd2b81054b0 2024-03-04T07:39:54.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.769+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2b8062500 con 0x7fd2a00a1e90 2024-03-04T07:39:54.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.781+0000 7fd2b5600700 1 -- 172.21.15.116:0/3929552578 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fd2b8062500 con 0x7fd2a00a1e90 2024-03-04T07:39:54.782 INFO:teuthology.orchestra.run.smithi116.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-04T07:39:54.782 INFO:teuthology.orchestra.run.smithi116.stdout:mds.all 12/12 58s ago 2m smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi133=b;smithi133=e;smithi133=h;smithi133=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-04T07:39:54.782 INFO:teuthology.orchestra.run.smithi116.stdout:mgr 3/3 58s ago 8m smithi116=x;smithi133=y;smithi152=z;count:3 2024-03-04T07:39:54.782 INFO:teuthology.orchestra.run.smithi116.stdout:mon 3/3 58s ago 9m smithi116:172.21.15.116=a;smithi133:172.21.15.133=b;smithi152:172.21.15.152=c;count:3 2024-03-04T07:39:54.782 INFO:teuthology.orchestra.run.smithi116.stdout:osd 12 58s ago - 2024-03-04T07:39:54.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd2a00a1e90 msgr2=0x7fd2a00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd2a00a1e90 0x7fd2a00a4350 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009bf0 tx=0x7fd2ac009260 comp rx=0 tx=0).stop 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 msgr2=0x7fd2b8198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b8198900 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400b2e0 tx=0x7fd2a400bc60 comp rx=0 tx=0).stop 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 shutdown_connections 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2b8107de0 0x7fd2b819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd2a00a1e90 0x7fd2a00a4350 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2b8102b80 0x7fd2b81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 --2- 172.21.15.116:0/3929552578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd2b81054b0 0x7fd2b8198900 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.784+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 >> 172.21.15.116:0/3929552578 conn(0x7fd2b80fc590 msgr2=0x7fd2b81034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.785+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 shutdown_connections 2024-03-04T07:39:54.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:54.785+0000 7fd2bf600700 1 -- 172.21.15.116:0/3929552578 wait complete. 2024-03-04T07:39:55.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:54 smithi133 ceph-mon[36505]: pgmap v375: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:55.177 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:54 smithi116 ceph-mon[32150]: pgmap v375: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:54 smithi152 ceph-mon[37154]: pgmap v375: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:55.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-04T07:39:56.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:55 smithi116 ceph-mon[32150]: from='client.25004 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:56.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:55 smithi133 ceph-mon[36505]: from='client.25004 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:56.104 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:39:56.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:55 smithi152 ceph-mon[37154]: from='client.25004 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:39:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:56 smithi116 ceph-mon[32150]: pgmap v376: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:57.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:56 smithi133 ceph-mon[36505]: pgmap v376: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:56 smithi152 ceph-mon[37154]: pgmap v376: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:39:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.367+0000 7f7d77000700 1 -- 172.21.15.116:0/3961780343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 msgr2=0x7f7d70102530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.367+0000 7f7d77000700 1 --2- 172.21.15.116:0/3961780343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d70102530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7d600099c0 tx=0x7f7d60009cd0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.367+0000 7f7d77000700 1 -- 172.21.15.116:0/3961780343 shutdown_connections 2024-03-04T07:39:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.367+0000 7f7d77000700 1 --2- 172.21.15.116:0/3961780343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70103cb0 0x7f7d7010a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.367+0000 7f7d77000700 1 --2- 172.21.15.116:0/3961780343 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d70102530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.367+0000 7f7d77000700 1 --2- 172.21.15.116:0/3961780343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d70103310 0x7f7d70103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.367+0000 7f7d77000700 1 -- 172.21.15.116:0/3961780343 >> 172.21.15.116:0/3961780343 conn(0x7f7d700fd670 msgr2=0x7f7d700ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.368+0000 7f7d77000700 1 -- 172.21.15.116:0/3961780343 shutdown_connections 2024-03-04T07:39:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.368+0000 7f7d77000700 1 -- 172.21.15.116:0/3961780343 wait complete. 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 Processor -- start 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 -- start start 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d7019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70103310 0x7f7d7019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d70103cb0 0x7f7d701a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7010dce0 con 0x7f7d70103310 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7010db60 con 0x7f7d70102110 2024-03-04T07:39:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d77000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7010de60 con 0x7f7d70103cb0 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.369+0000 7f7d6fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d7019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d7019a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36666/0 (socket says 172.21.15.116:36666) 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3679727405 learned_addr learned my addr 172.21.15.116:0/3679727405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6f400700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70103310 0x7f7d7019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d75000700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d70103cb0 0x7f7d701a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3679727405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d70103cb0 msgr2=0x7f7d701a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d70103cb0 0x7f7d701a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70103310 msgr2=0x7f7d7019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70103310 0x7f7d7019aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 -- 172.21.15.116:0/3679727405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d701a2160 con 0x7f7d70102110 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.370+0000 7f7d6fe00700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d7019a500 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000bf60 tx=0x7f7d6000bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.371+0000 7f7d6cc00700 1 -- 172.21.15.116:0/3679727405 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60009e60 con 0x7f7d70102110 2024-03-04T07:39:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.371+0000 7f7d6cc00700 1 -- 172.21.15.116:0/3679727405 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d6000f460 con 0x7f7d70102110 2024-03-04T07:39:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.371+0000 7f7d6cc00700 1 -- 172.21.15.116:0/3679727405 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60017630 con 0x7f7d70102110 2024-03-04T07:39:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.371+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d701a23f0 con 0x7f7d70102110 2024-03-04T07:39:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.371+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d701a2920 con 0x7f7d70102110 2024-03-04T07:39:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.372+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d70068d80 con 0x7f7d70102110 2024-03-04T07:39:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.374+0000 7f7d6cc00700 1 -- 172.21.15.116:0/3679727405 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f7d6000f610 con 0x7f7d70102110 2024-03-04T07:39:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.375+0000 7f7d6cc00700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7d540a1e90 0x7f7d540a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:39:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.375+0000 7f7d6cc00700 1 -- 172.21.15.116:0/3679727405 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7d600ceaa0 con 0x7f7d70102110 2024-03-04T07:39:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.375+0000 7f7d6f400700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7d540a1e90 0x7f7d540a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:39:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.376+0000 7f7d6f400700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7d540a1e90 0x7f7d540a4350 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c005950 tx=0x7f7d5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:39:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.379+0000 7f7d6cc00700 1 -- 172.21.15.116:0/3679727405 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7d60013070 con 0x7f7d70102110 2024-03-04T07:39:58.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.614+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d70075a10 con 0x7f7d540a1e90 2024-03-04T07:39:58.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.616+0000 7f7d6cc00700 1 -- 172.21.15.116:0/3679727405 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f7d70075a10 con 0x7f7d540a1e90 2024-03-04T07:39:58.616 INFO:teuthology.orchestra.run.smithi116.stdout:HOST ADDR LABELS STATUS 2024-03-04T07:39:58.616 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 172.21.15.116 2024-03-04T07:39:58.616 INFO:teuthology.orchestra.run.smithi116.stdout:smithi133 172.21.15.133 2024-03-04T07:39:58.616 INFO:teuthology.orchestra.run.smithi116.stdout:smithi152 172.21.15.152 2024-03-04T07:39:58.616 INFO:teuthology.orchestra.run.smithi116.stdout:3 hosts in cluster 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7d540a1e90 msgr2=0x7f7d540a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7d540a1e90 0x7f7d540a4350 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c005950 tx=0x7f7d5c00b410 comp rx=0 tx=0).stop 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 msgr2=0x7f7d7019a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d7019a500 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000bf60 tx=0x7f7d6000bf90 comp rx=0 tx=0).stop 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 shutdown_connections 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7d540a1e90 0x7f7d540a4350 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d70103cb0 0x7f7d701a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7d70102110 0x7f7d7019a500 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 --2- 172.21.15.116:0/3679727405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70103310 0x7f7d7019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 >> 172.21.15.116:0/3679727405 conn(0x7f7d700fd670 msgr2=0x7f7d700742f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:39:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 shutdown_connections 2024-03-04T07:39:58.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:39:58.619+0000 7f7d77000700 1 -- 172.21.15.116:0/3679727405 wait complete. 2024-03-04T07:39:58.990 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:58 smithi116 ceph-mon[32150]: pgmap v377: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:59.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:58 smithi133 ceph-mon[36505]: pgmap v377: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:58 smithi152 ceph-mon[37154]: pgmap v377: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:39:59.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-04T07:39:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:40:00.078 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:39:59 smithi116 ceph-mon[32150]: from='client.25010 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:40:00.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:39:59 smithi133 ceph-mon[36505]: from='client.25010 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:40:00.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:39:59 smithi152 ceph-mon[37154]: from='client.25010 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:40:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:00 smithi133 ceph-mon[36505]: pgmap v378: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:01.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:00 smithi133 ceph-mon[36505]: overall HEALTH_OK 2024-03-04T07:40:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:00 smithi116 ceph-mon[32150]: pgmap v378: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:00 smithi116 ceph-mon[32150]: overall HEALTH_OK 2024-03-04T07:40:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:00 smithi152 ceph-mon[37154]: pgmap v378: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:00 smithi152 ceph-mon[37154]: overall HEALTH_OK 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.137+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3782292369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 msgr2=0x7ff7ec110a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.137+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3782292369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec110a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400b0a0 tx=0x7ff7e400b3b0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.138+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3782292369 shutdown_connections 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.138+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3782292369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ec0760f0 0x7ff7ec074550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.138+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3782292369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec110a00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.138+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3782292369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7ec074b20 0x7ff7ec074fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.138+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3782292369 >> 172.21.15.116:0/3782292369 conn(0x7ff7ec0fd5d0 msgr2=0x7ff7ec0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.138+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3782292369 shutdown_connections 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.138+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3782292369 wait complete. 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 Processor -- start 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 -- start start 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ec074b20 0x7ff7ec19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7ec0760f0 0x7ff7ec19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec1a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec114290 con 0x7ff7ec074b20 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec114110 con 0x7ff7ec10e610 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.139+0000 7ff7f0a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec114410 con 0x7ff7ec0760f0 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec1a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7ea800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ec074b20 0x7ff7ec19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec1a5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36688/0 (socket says 172.21.15.116:36688) 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 -- 172.21.15.116:0/3822479396 learned_addr learned my addr 172.21.15.116:0/3822479396 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7e9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7ec0760f0 0x7ff7ec19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 -- 172.21.15.116:0/3822479396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7ec0760f0 msgr2=0x7ff7ec19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7ec0760f0 0x7ff7ec19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 -- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ec074b20 msgr2=0x7ff7ec19e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ec074b20 0x7ff7ec19e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 -- 172.21.15.116:0/3822479396 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec1a6460 con 0x7ff7ec10e610 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7eb200700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec1a5dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009bc0 tx=0x7ff7e4004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7e3400700 1 -- 172.21.15.116:0/3822479396 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4021070 con 0x7ff7ec10e610 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7e3400700 1 -- 172.21.15.116:0/3822479396 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7e4004e50 con 0x7ff7ec10e610 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7e3400700 1 -- 172.21.15.116:0/3822479396 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e40047d0 con 0x7ff7ec10e610 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.140+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ec1a66f0 con 0x7ff7ec10e610 2024-03-04T07:40:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.141+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec1a6ba0 con 0x7ff7ec10e610 2024-03-04T07:40:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.142+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec068d80 con 0x7ff7ec10e610 2024-03-04T07:40:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.144+0000 7ff7e3400700 1 -- 172.21.15.116:0/3822479396 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff7e4003a00 con 0x7ff7ec10e610 2024-03-04T07:40:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.145+0000 7ff7e3400700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7d00a1e90 0x7ff7d00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.146+0000 7ff7e3400700 1 -- 172.21.15.116:0/3822479396 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff7e40cec60 con 0x7ff7ec10e610 2024-03-04T07:40:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.146+0000 7ff7ea800700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7d00a1e90 0x7ff7d00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.146+0000 7ff7ea800700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7d00a1e90 0x7ff7d00a4350 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0764d0 tx=0x7ff7dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.149+0000 7ff7e3400700 1 -- 172.21.15.116:0/3822479396 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff7e409c0e0 con 0x7ff7ec10e610 2024-03-04T07:40:02.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.379+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7ec062500 con 0x7ff7d00a1e90 2024-03-04T07:40:02.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.384+0000 7ff7e3400700 1 -- 172.21.15.116:0/3822479396 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7ff7ec062500 con 0x7ff7d00a1e90 2024-03-04T07:40:02.384 INFO:teuthology.orchestra.run.smithi116.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-04T07:40:02.384 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-04T07:40:02.384 INFO:teuthology.orchestra.run.smithi116.stdout:smithi133 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-04T07:40:02.384 INFO:teuthology.orchestra.run.smithi116.stdout:smithi152 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JT400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7d00a1e90 msgr2=0x7ff7d00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7d00a1e90 0x7ff7d00a4350 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0764d0 tx=0x7ff7dc009260 comp rx=0 tx=0).stop 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 msgr2=0x7ff7ec1a5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec1a5dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009bc0 tx=0x7ff7e4004c60 comp rx=0 tx=0).stop 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 shutdown_connections 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ec074b20 0x7ff7ec19e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7d00a1e90 0x7ff7d00a4350 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7ec10e610 0x7ff7ec1a5dc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 --2- 172.21.15.116:0/3822479396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7ec0760f0 0x7ff7ec19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 >> 172.21.15.116:0/3822479396 conn(0x7ff7ec0fd5d0 msgr2=0x7ff7ec1019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:02.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 shutdown_connections 2024-03-04T07:40:02.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:02.387+0000 7ff7f0a00700 1 -- 172.21.15.116:0/3822479396 wait complete. 2024-03-04T07:40:03.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:02 smithi133 ceph-mon[36505]: pgmap v379: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:03.198 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:02 smithi116 ceph-mon[32150]: pgmap v379: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:03.269 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T07:40:03.278 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T07:40:03.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-04T07:40:03.279 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:02 smithi152 ceph-mon[37154]: pgmap v379: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:03.731 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:40:04.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:03 smithi116 ceph-mon[32150]: from='client.25016 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:40:04.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:03 smithi133 ceph-mon[36505]: from='client.25016 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:40:04.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:03 smithi152 ceph-mon[37154]: from='client.25016 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T07:40:05.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:04 smithi133 ceph-mon[36505]: pgmap v380: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:05.207 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:04 smithi116 ceph-mon[32150]: pgmap v380: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:04 smithi152 ceph-mon[37154]: pgmap v380: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:06.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 -- 172.21.15.116:0/2538311165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 msgr2=0x7fbef8108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:06.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 --2- 172.21.15.116:0/2538311165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef8108b60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbee80099c0 tx=0x7fbee8009cd0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 -- 172.21.15.116:0/2538311165 shutdown_connections 2024-03-04T07:40:06.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 --2- 172.21.15.116:0/2538311165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbef810a2e0 0x7fbef8110b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 --2- 172.21.15.116:0/2538311165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef8108b60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 --2- 172.21.15.116:0/2538311165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbef8109940 0x7fbef8109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 -- 172.21.15.116:0/2538311165 >> 172.21.15.116:0/2538311165 conn(0x7fbef8077710 msgr2=0x7fbef8079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 -- 172.21.15.116:0/2538311165 shutdown_connections 2024-03-04T07:40:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.054+0000 7fbefe800700 1 -- 172.21.15.116:0/2538311165 wait complete. 2024-03-04T07:40:06.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.055+0000 7fbefe800700 1 Processor -- start 2024-03-04T07:40:06.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbefe800700 1 -- start start 2024-03-04T07:40:06.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbefe800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbefe800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbef8109940 0x7fbef819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbefe800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbef810a2e0 0x7fbef81a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbefe800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef8114350 con 0x7fbef810a2e0 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbefe800700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef81141d0 con 0x7fbef8108740 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbefe800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef81144d0 con 0x7fbef8109940 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbef7400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbef7400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef819e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35626/0 (socket says 172.21.15.116:35626) 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbef7400700 1 -- 172.21.15.116:0/321831572 learned_addr learned my addr 172.21.15.116:0/321831572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbeefe00700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbef8109940 0x7fbef819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.056+0000 7fbef7e00700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbef810a2e0 0x7fbef81a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef7400700 1 -- 172.21.15.116:0/321831572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbef8109940 msgr2=0x7fbef819ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef7400700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbef8109940 0x7fbef819ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef7400700 1 -- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbef810a2e0 msgr2=0x7fbef81a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:06.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef7400700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbef810a2e0 0x7fbef81a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef7400700 1 -- 172.21.15.116:0/321831572 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef81a6590 con 0x7fbef8108740 2024-03-04T07:40:06.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbeefe00700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbef8109940 0x7fbef819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:06.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef7400700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef819e930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbee80098e0 tx=0x7fbee8009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:06.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef4c00700 1 -- 172.21.15.116:0/321831572 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8027070 con 0x7fbef8108740 2024-03-04T07:40:06.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef4c00700 1 -- 172.21.15.116:0/321831572 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbee8022470 con 0x7fbef8108740 2024-03-04T07:40:06.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef81a6820 con 0x7fbef8108740 2024-03-04T07:40:06.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbef4c00700 1 -- 172.21.15.116:0/321831572 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8009e60 con 0x7fbef8108740 2024-03-04T07:40:06.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.057+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef81a6cd0 con 0x7fbef8108740 2024-03-04T07:40:06.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.058+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef8068d80 con 0x7fbef8108740 2024-03-04T07:40:06.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.060+0000 7fbef4c00700 1 -- 172.21.15.116:0/321831572 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fbee802a070 con 0x7fbef8108740 2024-03-04T07:40:06.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.061+0000 7fbef4c00700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbedc0a1e90 0x7fbedc0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:06.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.062+0000 7fbef4c00700 1 -- 172.21.15.116:0/321831572 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fbee80ce810 con 0x7fbef8108740 2024-03-04T07:40:06.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.062+0000 7fbeefe00700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbedc0a1e90 0x7fbedc0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:06.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.062+0000 7fbeefe00700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbedc0a1e90 0x7fbedc0a4350 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbee0011490 tx=0x7fbee0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:06.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.065+0000 7fbef4c00700 1 -- 172.21.15.116:0/321831572 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbee8013070 con 0x7fbef8108740 2024-03-04T07:40:06.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.334+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbef8062cd0 con 0x7fbef8108740 2024-03-04T07:40:06.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.334+0000 7fbef4c00700 1 -- 172.21.15.116:0/321831572 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+2954 (secure 0 0 0) 0x7fbee809bc90 con 0x7fbef8108740 2024-03-04T07:40:06.335 INFO:teuthology.orchestra.run.smithi116.stdout:e19 2024-03-04T07:40:06.335 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T07:40:06.335 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 19 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T07:37:54.573948+0000 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T07:37:58.240682+0000 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 300 2024-03-04T07:40:06.336 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 0 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 1 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:in 0 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24581} 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-04T07:40:06.337 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-04T07:40:06.338 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-04T07:40:06.338 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-04T07:40:06.338 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-04T07:40:06.338 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-04T07:40:06.339 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-04T07:40:06.339 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-04T07:40:06.339 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 0 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:qdb_cluster leader: 24581 members: 24581 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b{0:24581} state up:active seq 3 addr [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.g{0:24553} state up:standby-replay seq 1 addr [v2:172.21.15.116:6838/1000386690,v1:172.21.15.116:6839/1000386690] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.116:6834/666560370,v1:172.21.15.116:6835/666560370] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a{-1:14463} state up:standby seq 1 addr [v2:172.21.15.116:6840/18076009,v1:172.21.15.116:6841/18076009] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.l{-1:24445} state up:standby seq 1 addr [v2:172.21.15.152:6832/2796423979,v1:172.21.15.152:6833/2796423979] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.k{-1:24455} state up:standby seq 1 addr [v2:172.21.15.133:6832/20614029,v1:172.21.15.133:6833/20614029] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d{-1:24475} state up:standby seq 1 addr [v2:172.21.15.116:6836/3275850329,v1:172.21.15.116:6837/3275850329] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.e{-1:24497} state up:standby seq 1 addr [v2:172.21.15.133:6834/331803361,v1:172.21.15.133:6835/331803361] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.f{-1:24499} state up:standby seq 1 addr [v2:172.21.15.152:6834/3438829914,v1:172.21.15.152:6835/3438829914] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.341 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c{-1:24511} state up:standby seq 1 addr [v2:172.21.15.152:6836/603594525,v1:172.21.15.152:6837/603594525] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.341 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.h{-1:24524} state up:standby seq 1 addr [v2:172.21.15.133:6836/336393981,v1:172.21.15.133:6837/336393981] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.341 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.i{-1:24535} state up:standby seq 1 addr [v2:172.21.15.152:6838/3832975468,v1:172.21.15.152:6839/3832975468] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T07:40:06.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbedc0a1e90 msgr2=0x7fbedc0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbedc0a1e90 0x7fbedc0a4350 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbee0011490 tx=0x7fbee0009260 comp rx=0 tx=0).stop 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 msgr2=0x7fbef819e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef819e930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbee80098e0 tx=0x7fbee8009580 comp rx=0 tx=0).stop 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 shutdown_connections 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbef810a2e0 0x7fbef81a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbedc0a1e90 0x7fbedc0a4350 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbef8108740 0x7fbef819e930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 --2- 172.21.15.116:0/321831572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbef8109940 0x7fbef819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.337+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 >> 172.21.15.116:0/321831572 conn(0x7fbef8077710 msgr2=0x7fbef810cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.338+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 shutdown_connections 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:06.338+0000 7fbefe800700 1 -- 172.21.15.116:0/321831572 wait complete. 2024-03-04T07:40:06.342 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:40:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:06 smithi116 ceph-mon[32150]: pgmap v381: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:06 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/321831572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T07:40:07.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:06 smithi133 ceph-mon[36505]: pgmap v381: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:07.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:06 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/321831572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T07:40:07.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-04T07:40:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:06 smithi152 ceph-mon[37154]: pgmap v381: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:06 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/321831572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T07:40:07.586 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:40:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:08 smithi116 ceph-mon[32150]: pgmap v382: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:09.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:08 smithi133 ceph-mon[36505]: pgmap v382: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:08 smithi152 ceph-mon[37154]: pgmap v382: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 -- 172.21.15.116:0/3728259314 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 msgr2=0x7fe69c1059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 --2- 172.21.15.116:0/3728259314 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c1059a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe6880099c0 tx=0x7fe688009cd0 comp rx=0 tx=0).stop 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 -- 172.21.15.116:0/3728259314 shutdown_connections 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 --2- 172.21.15.116:0/3728259314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c104340 0x7fe69c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 --2- 172.21.15.116:0/3728259314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe69c105ee0 0x7fe69c10c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 --2- 172.21.15.116:0/3728259314 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c1059a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 -- 172.21.15.116:0/3728259314 >> 172.21.15.116:0/3728259314 conn(0x7fe69c0ff8e0 msgr2=0x7fe69c101d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:09.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.847+0000 7fe6a4600700 1 -- 172.21.15.116:0/3728259314 shutdown_connections 2024-03-04T07:40:09.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.848+0000 7fe6a4600700 1 -- 172.21.15.116:0/3728259314 wait complete. 2024-03-04T07:40:09.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 Processor -- start 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 -- start start 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c104340 0x7fe69c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe69c105ee0 0x7fe69c1a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c10ff80 con 0x7fe69c104340 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c10fe00 con 0x7fe69c105540 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.849+0000 7fe6a4600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c110100 con 0x7fe69c105ee0 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c19aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35648/0 (socket says 172.21.15.116:35648) 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 -- 172.21.15.116:0/1916342036 learned_addr learned my addr 172.21.15.116:0/1916342036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1c00700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c104340 0x7fe69c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a2600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe69c105ee0 0x7fe69c1a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:09.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 -- 172.21.15.116:0/1916342036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe69c105ee0 msgr2=0x7fe69c1a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe69c105ee0 0x7fe69c1a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 -- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c104340 msgr2=0x7fe69c19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c104340 0x7fe69c19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 -- 172.21.15.116:0/1916342036 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c1a2190 con 0x7fe69c105540 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1c00700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c104340 0x7fe69c19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a2600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe69c105ee0 0x7fe69c1a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe6a1200700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c19aa70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe68800f690 tx=0x7fe68800f770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.850+0000 7fe69a000700 1 -- 172.21.15.116:0/1916342036 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688027070 con 0x7fe69c105540 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.851+0000 7fe69a000700 1 -- 172.21.15.116:0/1916342036 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe68800fc90 con 0x7fe69c105540 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.851+0000 7fe69a000700 1 -- 172.21.15.116:0/1916342036 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688017650 con 0x7fe69c105540 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.851+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69c1a2420 con 0x7fe69c105540 2024-03-04T07:40:09.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.851+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c1a2820 con 0x7fe69c105540 2024-03-04T07:40:09.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.852+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69c068d80 con 0x7fe69c105540 2024-03-04T07:40:09.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.854+0000 7fe69a000700 1 -- 172.21.15.116:0/1916342036 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe68802a070 con 0x7fe69c105540 2024-03-04T07:40:09.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.855+0000 7fe69a000700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6840a1e90 0x7fe6840a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:09.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.855+0000 7fe69a000700 1 -- 172.21.15.116:0/1916342036 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe6880cebe0 con 0x7fe69c105540 2024-03-04T07:40:09.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.855+0000 7fe6a1c00700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6840a1e90 0x7fe6840a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:09.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.856+0000 7fe6a1c00700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6840a1e90 0x7fe6840a4350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe69c1053a0 tx=0x7fe690009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:09.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:09.859+0000 7fe69a000700 1 -- 172.21.15.116:0/1916342036 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe68805e280 con 0x7fe69c105540 2024-03-04T07:40:10.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:09 smithi116 ceph-mon[32150]: pgmap v383: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.084+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fe69c02d9b0 con 0x7fe69c105540 2024-03-04T07:40:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.085+0000 7fe69a000700 1 -- 172.21.15.116:0/1916342036 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+5340 (secure 0 0 0) 0x7fe688013070 con 0x7fe69c105540 2024-03-04T07:40:10.087 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 87 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-04T07:28:48.960816+0000 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-04T07:38:58.798253+0000 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:crush_version 26 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:full_ratio 0.95 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:backfillfull_ratio 0.9 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:nearfull_ratio 0.85 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:require_min_compat_client luminous 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:min_compat_client luminous 2024-03-04T07:40:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:require_osd_release squid 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:stretch_mode_enabled false 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:max_osd 12 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6802/3034556591,v1:172.21.15.116:6803/3034556591] [v2:172.21.15.116:6804/3034556591,v1:172.21.15.116:6805/3034556591] exists,up 914e00c8-058b-4b8a-8432-b88bb60d516b 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.133:6800/1626213243,v1:172.21.15.133:6801/1626213243] [v2:172.21.15.133:6802/1626213243,v1:172.21.15.133:6803/1626213243] exists,up 4809c734-3113-4aa9-b228-80c2e0c2dd5b 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6800/4177028869,v1:172.21.15.152:6801/4177028869] [v2:172.21.15.152:6802/4177028869,v1:172.21.15.152:6803/4177028869] exists,up 5e50d8f2-dbf0-4066-b2cb-df0821866acd 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 up in weight 1 up_from 25 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6810/2894752093,v1:172.21.15.116:6811/2894752093] [v2:172.21.15.116:6812/2894752093,v1:172.21.15.116:6813/2894752093] exists,up 3acba8d9-b204-41eb-8f35-936f0acb0f87 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.133:6808/3217315189,v1:172.21.15.133:6809/3217315189] [v2:172.21.15.133:6810/3217315189,v1:172.21.15.133:6811/3217315189] exists,up 60bcd05b-b11c-4cca-948b-7dd432c6cd2c 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6808/2293273292,v1:172.21.15.152:6809/2293273292] [v2:172.21.15.152:6810/2293273292,v1:172.21.15.152:6811/2293273292] exists,up c0fa4743-f949-4748-9772-3bb1292ebb01 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6818/836391582,v1:172.21.15.116:6819/836391582] [v2:172.21.15.116:6820/836391582,v1:172.21.15.116:6821/836391582] exists,up 2442f5fc-a522-4f08-8a68-c0bc903427b4 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.133:6816/3238142404,v1:172.21.15.133:6817/3238142404] [v2:172.21.15.133:6818/3238142404,v1:172.21.15.133:6819/3238142404] exists,up f1dbc539-c51f-4ae3-9239-8c032ef60b4f 2024-03-04T07:40:10.089 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6816/1470448295,v1:172.21.15.152:6817/1470448295] [v2:172.21.15.152:6818/1470448295,v1:172.21.15.152:6819/1470448295] exists,up a9bc6819-8f22-4e6d-83e1-96690fc82524 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6826/3236031777,v1:172.21.15.116:6827/3236031777] [v2:172.21.15.116:6828/3236031777,v1:172.21.15.116:6829/3236031777] exists,up 2931b69e-19e6-4739-8f27-b1d0b83f96d5 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 up in weight 1 up_from 72 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.133:6824/2680230905,v1:172.21.15.133:6825/2680230905] [v2:172.21.15.133:6826/2680230905,v1:172.21.15.133:6827/2680230905] exists,up 4111372f-edde-4015-beb6-fb5895afe8d7 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6824/1192411911,v1:172.21.15.152:6825/1192411911] [v2:172.21.15.152:6826/1192411911,v1:172.21.15.152:6827/1192411911] exists,up 9c4a6c93-8d7c-448c-99cb-a0ede94976a9 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.8 [2,11] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.b [8,11] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.10 [2,4] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.0 [4,8] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.2 [5,7] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.3 [4,9] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.c [3,9] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.e [4,6] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.1d [5,9] 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/1782984174 expires 2024-03-05T07:29:57.331810+0000 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3439724466 expires 2024-03-05T07:29:57.331810+0000 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2846507920 expires 2024-03-05T07:29:57.331810+0000 2024-03-04T07:40:10.090 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3742476358 expires 2024-03-05T07:29:22.911402+0000 2024-03-04T07:40:10.091 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/3218338096 expires 2024-03-05T07:29:57.331810+0000 2024-03-04T07:40:10.091 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2843178631 expires 2024-03-05T07:29:22.911402+0000 2024-03-04T07:40:10.091 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/3218338096 expires 2024-03-05T07:29:57.331810+0000 2024-03-04T07:40:10.091 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/1181502267 expires 2024-03-05T07:29:22.911402+0000 2024-03-04T07:40:10.091 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/1181502267 expires 2024-03-05T07:29:22.911402+0000 2024-03-04T07:40:10.091 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/740422381 expires 2024-03-05T07:29:22.911402+0000 2024-03-04T07:40:10.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.088+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6840a1e90 msgr2=0x7fe6840a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:10.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.088+0000 7fe6a4600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6840a1e90 0x7fe6840a4350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe69c1053a0 tx=0x7fe690009260 comp rx=0 tx=0).stop 2024-03-04T07:40:10.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.088+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 msgr2=0x7fe69c19aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:10.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.088+0000 7fe6a4600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c19aa70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe68800f690 tx=0x7fe68800f770 comp rx=0 tx=0).stop 2024-03-04T07:40:10.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.088+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 shutdown_connections 2024-03-04T07:40:10.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.088+0000 7fe6a4600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c104340 0x7fe69c19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:10.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.088+0000 7fe6a4600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6840a1e90 0x7fe6840a4350 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:10.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.089+0000 7fe6a4600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe69c105ee0 0x7fe69c1a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:10.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.089+0000 7fe6a4600700 1 --2- 172.21.15.116:0/1916342036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe69c105540 0x7fe69c19aa70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:10.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.089+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 >> 172.21.15.116:0/1916342036 conn(0x7fe69c0ff8e0 msgr2=0x7fe69c108840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:10.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.089+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 shutdown_connections 2024-03-04T07:40:10.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:10.089+0000 7fe6a4600700 1 -- 172.21.15.116:0/1916342036 wait complete. 2024-03-04T07:40:10.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:09 smithi152 ceph-mon[37154]: pgmap v383: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:10.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:09 smithi133 ceph-mon[36505]: pgmap v383: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:10.861 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-04T07:40:10.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 -- 172.21.15.116:0/3791586202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954101410 msgr2=0x7f1954101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 --2- 172.21.15.116:0/3791586202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954101410 0x7f1954101830 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1944009900 tx=0x7f194402bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 -- 172.21.15.116:0/3791586202 shutdown_connections 2024-03-04T07:40:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 --2- 172.21.15.116:0/3791586202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1954102fc0 0x7f1954109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 --2- 172.21.15.116:0/3791586202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1954102600 0x7f1954102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 --2- 172.21.15.116:0/3791586202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954101410 0x7f1954101830 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 -- 172.21.15.116:0/3791586202 >> 172.21.15.116:0/3791586202 conn(0x7f19540fc960 msgr2=0x7f19540fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 -- 172.21.15.116:0/3791586202 shutdown_connections 2024-03-04T07:40:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.087+0000 7f1953200640 1 -- 172.21.15.116:0/3791586202 wait complete. 2024-03-04T07:40:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.088+0000 7f1953200640 1 Processor -- start 2024-03-04T07:40:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.088+0000 7f1953200640 1 -- start start 2024-03-04T07:40:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1953200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1954101410 0x7f19541999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1953200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954102600 0x7f1954199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1953200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1954102fc0 0x7f19541a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1953200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195410d0f0 con 0x7f1954101410 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1953200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195410cf70 con 0x7f1954102600 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1953200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195410d270 con 0x7f1954102fc0 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1954101410 0x7f19541999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954102600 0x7f1954199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954102600 0x7f1954199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35666/0 (socket says 172.21.15.116:35666) 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 -- 172.21.15.116:0/3131831107 learned_addr learned my addr 172.21.15.116:0/3131831107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1952800640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1954102fc0 0x7f19541a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 -- 172.21.15.116:0/3131831107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1954102fc0 msgr2=0x7f19541a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1954102fc0 0x7f19541a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 -- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1954101410 msgr2=0x7f19541999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1954101410 0x7f19541999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1951400640 1 -- 172.21.15.116:0/3131831107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19541a15c0 con 0x7f1954102600 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.089+0000 7f1952800640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1954102fc0 0x7f19541a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.090+0000 7f1951400640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954102600 0x7f1954199f30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f194000e9f0 tx=0x7f194000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.090+0000 7f194aa00640 1 -- 172.21.15.116:0/3131831107 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194000cd80 con 0x7f1954102600 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.090+0000 7f194aa00640 1 -- 172.21.15.116:0/3131831107 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1940005500 con 0x7f1954102600 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.090+0000 7f194aa00640 1 -- 172.21.15.116:0/3131831107 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940010690 con 0x7f1954102600 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.090+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19541a18b0 con 0x7f1954102600 2024-03-04T07:40:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.090+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19541a1db0 con 0x7f1954102600 2024-03-04T07:40:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.091+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19541a2010 con 0x7f1954102600 2024-03-04T07:40:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.093+0000 7f194aa00640 1 -- 172.21.15.116:0/3131831107 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f19400046c0 con 0x7f1954102600 2024-03-04T07:40:11.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.095+0000 7f194aa00640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19380a1e00 0x7f19380a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.095+0000 7f194aa00640 1 -- 172.21.15.116:0/3131831107 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f19400ce230 con 0x7f1954102600 2024-03-04T07:40:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.095+0000 7f1951e00640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19380a1e00 0x7f19380a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.096+0000 7f1951e00640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19380a1e00 0x7f19380a42c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f194402c170 tx=0x7f194402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.098+0000 7f194aa00640 1 -- 172.21.15.116:0/3131831107 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f194009b620 con 0x7f1954102600 2024-03-04T07:40:11.172 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:10 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1916342036' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T07:40:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:10 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1916342036' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T07:40:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.304+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19541a21e0 con 0x7f1954102600 2024-03-04T07:40:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.305+0000 7f194aa00640 1 -- 172.21.15.116:0/3131831107 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f194009af20 con 0x7f1954102600 2024-03-04T07:40:11.307 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:11.307 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.308+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19380a1e00 msgr2=0x7f19380a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.308+0000 7f1953200640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19380a1e00 0x7f19380a42c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f194402c170 tx=0x7f194402df90 comp rx=0 tx=0).stop 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.308+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954102600 msgr2=0x7f1954199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.308+0000 7f1953200640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954102600 0x7f1954199f30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f194000e9f0 tx=0x7f194000edf0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 shutdown_connections 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19380a1e00 0x7f19380a42c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1954102fc0 0x7f19541a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1954102600 0x7f1954199f30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 --2- 172.21.15.116:0/3131831107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1954101410 0x7f19541999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 >> 172.21.15.116:0/3131831107 conn(0x7f19540fc960 msgr2=0x7f1954073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 shutdown_connections 2024-03-04T07:40:11.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.309+0000 7f1953200640 1 -- 172.21.15.116:0/3131831107 wait complete. 2024-03-04T07:40:11.327 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:40:11.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:40:11.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:10 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1916342036' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T07:40:11.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 -- 172.21.15.116:0/1512829198 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08074030 msgr2=0x7f6e080744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 --2- 172.21.15.116:0/1512829198 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08074030 0x7f6e080744b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009890 tx=0x7f6df002bb70 comp rx=0 tx=0).stop 2024-03-04T07:40:11.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 -- 172.21.15.116:0/1512829198 shutdown_connections 2024-03-04T07:40:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 --2- 172.21.15.116:0/1512829198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0810daa0 0x7f6e0810fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 --2- 172.21.15.116:0/1512829198 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08074030 0x7f6e080744b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 --2- 172.21.15.116:0/1512829198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08075610 0x7f6e08073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 -- 172.21.15.116:0/1512829198 >> 172.21.15.116:0/1512829198 conn(0x7f6e080fca40 msgr2=0x7f6e080fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 -- 172.21.15.116:0/1512829198 shutdown_connections 2024-03-04T07:40:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.543+0000 7f6e07e00640 1 -- 172.21.15.116:0/1512829198 wait complete. 2024-03-04T07:40:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.544+0000 7f6e07e00640 1 Processor -- start 2024-03-04T07:40:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.544+0000 7f6e07e00640 1 -- start start 2024-03-04T07:40:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e07e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08074030 0x7f6e0819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e07e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08075610 0x7f6e0819e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e07e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0810daa0 0x7f6e081a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e07e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08113720 con 0x7f6e08074030 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e07e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081135a0 con 0x7f6e08075610 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e07e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081138a0 con 0x7f6e0810daa0 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08074030 0x7f6e0819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08074030 0x7f6e0819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42326/0 (socket says 172.21.15.116:42326) 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e06a00640 1 -- 172.21.15.116:0/3369301925 learned_addr learned my addr 172.21.15.116:0/3369301925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6dfe000640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08075610 0x7f6e0819e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e07400640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0810daa0 0x7f6e081a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6dfe000640 1 -- 172.21.15.116:0/3369301925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0810daa0 msgr2=0x7f6e081a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6dfe000640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0810daa0 0x7f6e081a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6dfe000640 1 -- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08074030 msgr2=0x7f6e0819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6dfe000640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08074030 0x7f6e0819de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6dfe000640 1 -- 172.21.15.116:0/3369301925 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e081a5a30 con 0x7f6e08075610 2024-03-04T07:40:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6e06a00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08074030 0x7f6e0819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.545+0000 7f6dfe000640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08075610 0x7f6e0819e350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6df002c0b0 tx=0x7f6df0034830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.546+0000 7f6e0ca00640 1 -- 172.21.15.116:0/3369301925 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0041020 con 0x7f6e08075610 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.546+0000 7f6e0ca00640 1 -- 172.21.15.116:0/3369301925 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6df002c9f0 con 0x7f6e08075610 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.546+0000 7f6e0ca00640 1 -- 172.21.15.116:0/3369301925 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0039040 con 0x7f6e08075610 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.546+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e081a5cc0 con 0x7f6e08075610 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.546+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e081a61b0 con 0x7f6e08075610 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.547+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e08103c70 con 0x7f6e08075610 2024-03-04T07:40:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.549+0000 7f6e0ca00640 1 -- 172.21.15.116:0/3369301925 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f6df0044020 con 0x7f6e08075610 2024-03-04T07:40:11.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.550+0000 7f6e0ca00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6dec0a1e00 0x7f6dec0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.550+0000 7f6e0ca00640 1 -- 172.21.15.116:0/3369301925 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f6df00ef7d0 con 0x7f6e08075610 2024-03-04T07:40:11.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.550+0000 7f6e06a00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6dec0a1e00 0x7f6dec0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.551+0000 7f6e06a00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6dec0a1e00 0x7f6dec0a42c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6df80058c0 tx=0x7f6df8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.554+0000 7f6e0ca00640 1 -- 172.21.15.116:0/3369301925 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6df004b020 con 0x7f6e08075610 2024-03-04T07:40:11.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.751+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e0802d210 con 0x7f6e08075610 2024-03-04T07:40:11.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.752+0000 7f6e0ca00640 1 -- 172.21.15.116:0/3369301925 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6df00bcbc0 con 0x7f6e08075610 2024-03-04T07:40:11.753 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:40:11.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6dec0a1e00 msgr2=0x7f6dec0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6dec0a1e00 0x7f6dec0a42c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6df80058c0 tx=0x7f6df8009260 comp rx=0 tx=0).stop 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08075610 msgr2=0x7f6e0819e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08075610 0x7f6e0819e350 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6df002c0b0 tx=0x7f6df0034830 comp rx=0 tx=0).stop 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 shutdown_connections 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6dec0a1e00 0x7f6dec0a42c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0810daa0 0x7f6e081a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6e08075610 0x7f6e0819e350 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 --2- 172.21.15.116:0/3369301925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e08074030 0x7f6e0819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 >> 172.21.15.116:0/3369301925 conn(0x7f6e080fca40 msgr2=0x7f6e08100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.754+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 shutdown_connections 2024-03-04T07:40:11.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.755+0000 7f6e07e00640 1 -- 172.21.15.116:0/3369301925 wait complete. 2024-03-04T07:40:11.774 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.987+0000 7f11ef000640 1 -- 172.21.15.116:0/3182196949 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100ff0 msgr2=0x7f11e8101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.987+0000 7f11ef000640 1 --2- 172.21.15.116:0/3182196949 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100ff0 0x7f11e8101870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009ca0 tx=0x7f11e002c100 comp rx=0 tx=0).stop 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.988+0000 7f11ef000640 1 -- 172.21.15.116:0/3182196949 shutdown_connections 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.988+0000 7f11ef000640 1 --2- 172.21.15.116:0/3182196949 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100ff0 0x7f11e8101870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.988+0000 7f11ef000640 1 --2- 172.21.15.116:0/3182196949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8100630 0x7f11e8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.988+0000 7f11ef000640 1 --2- 172.21.15.116:0/3182196949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11e80ff8d0 0x7f11e80ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.988+0000 7f11ef000640 1 -- 172.21.15.116:0/3182196949 >> 172.21.15.116:0/3182196949 conn(0x7f11e80fb200 msgr2=0x7f11e80fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.988+0000 7f11ef000640 1 -- 172.21.15.116:0/3182196949 shutdown_connections 2024-03-04T07:40:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.988+0000 7f11ef000640 1 -- 172.21.15.116:0/3182196949 wait complete. 2024-03-04T07:40:11.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 Processor -- start 2024-03-04T07:40:11.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 -- start start 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80ff8d0 0x7f11e8199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100630 0x7f11e8199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11e8100ff0 0x7f11e81a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e810f4b0 con 0x7f11e80ff8d0 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e810f330 con 0x7f11e8100630 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ef000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e810f630 con 0x7f11e8100ff0 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11e7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80ff8d0 0x7f11e8199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11e7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80ff8d0 0x7f11e8199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42344/0 (socket says 172.21.15.116:42344) 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11e7e00640 1 -- 172.21.15.116:0/3014132435 learned_addr learned my addr 172.21.15.116:0/3014132435 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11e7400640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100630 0x7f11e8199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.989+0000 7f11ed000640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11e8100ff0 0x7f11e81a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e7e00640 1 -- 172.21.15.116:0/3014132435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11e8100ff0 msgr2=0x7f11e81a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e7e00640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11e8100ff0 0x7f11e81a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e7e00640 1 -- 172.21.15.116:0/3014132435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100630 msgr2=0x7f11e8199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e7e00640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100630 0x7f11e8199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e7e00640 1 -- 172.21.15.116:0/3014132435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e81a1690 con 0x7f11e80ff8d0 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e7400640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100630 0x7f11e8199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e7e00640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80ff8d0 0x7f11e8199a70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f11d800e9f0 tx=0x7f11d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e4c00640 1 -- 172.21.15.116:0/3014132435 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d800cd80 con 0x7f11e80ff8d0 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e4c00640 1 -- 172.21.15.116:0/3014132435 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11d8005500 con 0x7f11e80ff8d0 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e81a1980 con 0x7f11e80ff8d0 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.990+0000 7f11e4c00640 1 -- 172.21.15.116:0/3014132435 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8010690 con 0x7f11e80ff8d0 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.991+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e81a1ed0 con 0x7f11e80ff8d0 2024-03-04T07:40:11.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.992+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e810b530 con 0x7f11e80ff8d0 2024-03-04T07:40:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.994+0000 7f11e4c00640 1 -- 172.21.15.116:0/3014132435 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f11d80046c0 con 0x7f11e80ff8d0 2024-03-04T07:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.995+0000 7f11e4c00640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11cc0a1e00 0x7f11cc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.995+0000 7f11e4c00640 1 -- 172.21.15.116:0/3014132435 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f11d80ce180 con 0x7f11e80ff8d0 2024-03-04T07:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.995+0000 7f11e7400640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11cc0a1e00 0x7f11cc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.996+0000 7f11e7400640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11cc0a1e00 0x7f11cc0a42c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f11e819af80 tx=0x7f11d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:11.999+0000 7f11e4c00640 1 -- 172.21.15.116:0/3014132435 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f11d809b570 con 0x7f11e80ff8d0 2024-03-04T07:40:12.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.198+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11e80621d0 con 0x7f11e80ff8d0 2024-03-04T07:40:12.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.199+0000 7f11e4c00640 1 -- 172.21.15.116:0/3014132435 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f11d809ae70 con 0x7f11e80ff8d0 2024-03-04T07:40:12.200 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:40:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11cc0a1e00 msgr2=0x7f11cc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11cc0a1e00 0x7f11cc0a42c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f11e819af80 tx=0x7f11d4005f90 comp rx=0 tx=0).stop 2024-03-04T07:40:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80ff8d0 msgr2=0x7f11e8199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80ff8d0 0x7f11e8199a70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f11d800e9f0 tx=0x7f11d800edf0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 shutdown_connections 2024-03-04T07:40:12.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f11cc0a1e00 0x7f11cc0a42c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11e8100ff0 0x7f11e81a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f11e8100630 0x7f11e8199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 --2- 172.21.15.116:0/3014132435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80ff8d0 0x7f11e8199a70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 >> 172.21.15.116:0/3014132435 conn(0x7f11e80fb200 msgr2=0x7f11e80fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 shutdown_connections 2024-03-04T07:40:12.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.202+0000 7f11ef000640 1 -- 172.21.15.116:0/3014132435 wait complete. 2024-03-04T07:40:12.220 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3131831107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:12.220 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:11 smithi116 ceph-mon[32150]: pgmap v384: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:12.220 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3369301925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:12.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:40:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:11 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3131831107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:11 smithi152 ceph-mon[37154]: pgmap v384: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:12.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:11 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3369301925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:12.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:11 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3131831107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:12.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:11 smithi133 ceph-mon[36505]: pgmap v384: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:12.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:11 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3369301925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:12.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.436+0000 7fe150200640 1 -- 172.21.15.116:0/16714229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1481020c0 msgr2=0x7fe1481044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.436+0000 7fe150200640 1 --2- 172.21.15.116:0/16714229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1481020c0 0x7fe1481044b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009900 tx=0x7fe13c02bc50 comp rx=0 tx=0).stop 2024-03-04T07:40:12.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.437+0000 7fe150200640 1 -- 172.21.15.116:0/16714229 shutdown_connections 2024-03-04T07:40:12.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.437+0000 7fe150200640 1 --2- 172.21.15.116:0/16714229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe148107320 0x7fe148109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.437+0000 7fe150200640 1 --2- 172.21.15.116:0/16714229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1481049f0 0x7fe148106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.437+0000 7fe150200640 1 --2- 172.21.15.116:0/16714229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1481020c0 0x7fe1481044b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.437+0000 7fe150200640 1 -- 172.21.15.116:0/16714229 >> 172.21.15.116:0/16714229 conn(0x7fe1480fba70 msgr2=0x7fe1480fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:12.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.437+0000 7fe150200640 1 -- 172.21.15.116:0/16714229 shutdown_connections 2024-03-04T07:40:12.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.437+0000 7fe150200640 1 -- 172.21.15.116:0/16714229 wait complete. 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 Processor -- start 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 -- start start 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1481020c0 0x7fe148199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1481049f0 0x7fe14819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe148107320 0x7fe1481a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14810d2c0 con 0x7fe1481020c0 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14810d140 con 0x7fe148107320 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe150200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14810d440 con 0x7fe1481049f0 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe14d800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1481020c0 0x7fe148199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe14e200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe148107320 0x7fe1481a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe14e200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe148107320 0x7fe1481a10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35716/0 (socket says 172.21.15.116:35716) 2024-03-04T07:40:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe14e200640 1 -- 172.21.15.116:0/1740131898 learned_addr learned my addr 172.21.15.116:0/1740131898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.438+0000 7fe14ce00640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1481049f0 0x7fe14819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe14e200640 1 -- 172.21.15.116:0/1740131898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1481049f0 msgr2=0x7fe14819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe14e200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1481049f0 0x7fe14819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe14e200640 1 -- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1481020c0 msgr2=0x7fe148199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe14e200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1481020c0 0x7fe148199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe14e200640 1 -- 172.21.15.116:0/1740131898 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1481a1750 con 0x7fe148107320 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe14e200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe148107320 0x7fe1481a10e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe14000eb10 tx=0x7fe14000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe146000640 1 -- 172.21.15.116:0/1740131898 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140004b20 con 0x7fe148107320 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe146000640 1 -- 172.21.15.116:0/1740131898 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe140004cc0 con 0x7fe148107320 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1481a1a40 con 0x7fe148107320 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe146000640 1 -- 172.21.15.116:0/1740131898 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140010760 con 0x7fe148107320 2024-03-04T07:40:12.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.439+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1481a1fe0 con 0x7fe148107320 2024-03-04T07:40:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.440+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe148108e10 con 0x7fe148107320 2024-03-04T07:40:12.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.442+0000 7fe146000640 1 -- 172.21.15.116:0/1740131898 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe140005020 con 0x7fe148107320 2024-03-04T07:40:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.443+0000 7fe146000640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1300a1e00 0x7fe1300a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.444+0000 7fe146000640 1 -- 172.21.15.116:0/1740131898 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe1400ce400 con 0x7fe148107320 2024-03-04T07:40:12.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.444+0000 7fe14d800640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1300a1e00 0x7fe1300a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.444+0000 7fe14d800640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1300a1e00 0x7fe1300a42c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe13c02c260 tx=0x7fe13c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:12.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.447+0000 7fe146000640 1 -- 172.21.15.116:0/1740131898 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe14009b830 con 0x7fe148107320 2024-03-04T07:40:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.646+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe1480621d0 con 0x7fe148107320 2024-03-04T07:40:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.647+0000 7fe146000640 1 -- 172.21.15.116:0/1740131898 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe14009b130 con 0x7fe148107320 2024-03-04T07:40:12.648 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1300a1e00 msgr2=0x7fe1300a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1300a1e00 0x7fe1300a42c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe13c02c260 tx=0x7fe13c000bc0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe148107320 msgr2=0x7fe1481a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe148107320 0x7fe1481a10e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe14000eb10 tx=0x7fe14000c2d0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 shutdown_connections 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe148107320 0x7fe1481a10e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1300a1e00 0x7fe1300a42c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1481049f0 0x7fe14819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 --2- 172.21.15.116:0/1740131898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1481020c0 0x7fe148199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 >> 172.21.15.116:0/1740131898 conn(0x7fe1480fba70 msgr2=0x7fe148101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 shutdown_connections 2024-03-04T07:40:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.650+0000 7fe150200640 1 -- 172.21.15.116:0/1740131898 wait complete. 2024-03-04T07:40:12.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:40:12.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 -- 172.21.15.116:0/2730973639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc1097c0 msgr2=0x7f1cdc110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/2730973639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc110050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009ca0 tx=0x7f1cd402c100 comp rx=0 tx=0).stop 2024-03-04T07:40:12.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 -- 172.21.15.116:0/2730973639 shutdown_connections 2024-03-04T07:40:12.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/2730973639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc110050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/2730973639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cdc108e00 0x7f1cdc109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/2730973639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc107c10 0x7f1cdc108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 -- 172.21.15.116:0/2730973639 >> 172.21.15.116:0/2730973639 conn(0x7f1cdc076be0 msgr2=0x7f1cdc079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:12.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 -- 172.21.15.116:0/2730973639 shutdown_connections 2024-03-04T07:40:12.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.885+0000 7f1ce1e00640 1 -- 172.21.15.116:0/2730973639 wait complete. 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 Processor -- start 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 -- start start 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc107c10 0x7f1cdc19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cdc108e00 0x7f1cdc19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc1a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc113870 con 0x7f1cdc1097c0 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc1136f0 con 0x7f1cdc107c10 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1ce1e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc1139f0 con 0x7f1cdc108e00 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1cdb400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc1a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1cdaa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc107c10 0x7f1cdc19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1cdaa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc107c10 0x7f1cdc19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35732/0 (socket says 172.21.15.116:35732) 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1cdaa00640 1 -- 172.21.15.116:0/4062022697 learned_addr learned my addr 172.21.15.116:0/4062022697 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:12.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.886+0000 7f1cdb400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc1a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42392/0 (socket says 172.21.15.116:42392) 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cda000640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cdc108e00 0x7f1cdc19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cdaa00640 1 -- 172.21.15.116:0/4062022697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cdc108e00 msgr2=0x7f1cdc19e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cdaa00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cdc108e00 0x7f1cdc19e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cdaa00640 1 -- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc1097c0 msgr2=0x7f1cdc1a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cdaa00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc1a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cdaa00640 1 -- 172.21.15.116:0/4062022697 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc1a5aa0 con 0x7f1cdc107c10 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cdb400640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1cdaa00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc107c10 0x7f1cdc19de80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9c0 tx=0x7f1ccc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1ce0a00640 1 -- 172.21.15.116:0/4062022697 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00cd80 con 0x7f1cdc107c10 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1ce0a00640 1 -- 172.21.15.116:0/4062022697 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ccc0035f0 con 0x7f1cdc107c10 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.887+0000 7f1ce0a00640 1 -- 172.21.15.116:0/4062022697 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc010690 con 0x7f1cdc107c10 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.888+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc1a5d90 con 0x7f1cdc107c10 2024-03-04T07:40:12.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.888+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc1a61d0 con 0x7f1cdc107c10 2024-03-04T07:40:12.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.889+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc10ef30 con 0x7f1cdc107c10 2024-03-04T07:40:12.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.890+0000 7f1ce0a00640 1 -- 172.21.15.116:0/4062022697 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1ccc0037a0 con 0x7f1cdc107c10 2024-03-04T07:40:12.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.892+0000 7f1ce0a00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cc00a1e00 0x7f1cc00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:12.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.892+0000 7f1ce0a00640 1 -- 172.21.15.116:0/4062022697 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1ccc0ce120 con 0x7f1cdc107c10 2024-03-04T07:40:12.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.892+0000 7f1cda000640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cc00a1e00 0x7f1cc00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:12.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.893+0000 7f1cda000640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cc00a1e00 0x7f1cc00a42c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc19f390 tx=0x7f1cc800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:12.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:12.895+0000 7f1ce0a00640 1 -- 172.21.15.116:0/4062022697 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1ccc09b510 con 0x7f1cdc107c10 2024-03-04T07:40:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:12 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3014132435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:12 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1740131898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:13.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.128+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cdc02cff0 con 0x7f1cdc107c10 2024-03-04T07:40:13.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.130+0000 7f1ce0a00640 1 -- 172.21.15.116:0/4062022697 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f1ccc09ae10 con 0x7f1cdc107c10 2024-03-04T07:40:13.131 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:13.131 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:40:13.131 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cc00a1e00 msgr2=0x7f1cc00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cc00a1e00 0x7f1cc00a42c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc19f390 tx=0x7f1cc800b410 comp rx=0 tx=0).stop 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc107c10 msgr2=0x7f1cdc19de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc107c10 0x7f1cdc19de80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9c0 tx=0x7f1ccc00ee80 comp rx=0 tx=0).stop 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 shutdown_connections 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cdc1097c0 0x7f1cdc1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cc00a1e00 0x7f1cc00a42c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cdc108e00 0x7f1cdc19e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 --2- 172.21.15.116:0/4062022697 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1cdc107c10 0x7f1cdc19de80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.132+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 >> 172.21.15.116:0/4062022697 conn(0x7f1cdc076be0 msgr2=0x7f1cdc10c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.133+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 shutdown_connections 2024-03-04T07:40:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.133+0000 7f1ce1e00640 1 -- 172.21.15.116:0/4062022697 wait complete. 2024-03-04T07:40:13.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:40:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:12 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3014132435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:12 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1740131898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:13.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:12 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3014132435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:13.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:12 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1740131898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 -- 172.21.15.116:0/1680779661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc107c10 msgr2=0x7f5edc108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1680779661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc107c10 0x7f5edc108030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009900 tx=0x7f5ecc02bc40 comp rx=0 tx=0).stop 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 -- 172.21.15.116:0/1680779661 shutdown_connections 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1680779661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5edc1097c0 0x7f5edc110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1680779661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5edc108e00 0x7f5edc109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1680779661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc107c10 0x7f5edc108030 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 -- 172.21.15.116:0/1680779661 >> 172.21.15.116:0/1680779661 conn(0x7f5edc076be0 msgr2=0x7f5edc079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 -- 172.21.15.116:0/1680779661 shutdown_connections 2024-03-04T07:40:13.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.360+0000 7f5ee1800640 1 -- 172.21.15.116:0/1680779661 wait complete. 2024-03-04T07:40:13.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.361+0000 7f5ee1800640 1 Processor -- start 2024-03-04T07:40:13.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.361+0000 7f5ee1800640 1 -- start start 2024-03-04T07:40:13.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.361+0000 7f5ee1800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5edc107c10 0x7f5edc19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.361+0000 7f5ee1800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5edc108e00 0x7f5edc19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5ee1800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc1097c0 0x7f5edc1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5ee1800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc1138b0 con 0x7f5edc108e00 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5ee1800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc113730 con 0x7f5edc1097c0 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5ee1800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc113a30 con 0x7f5edc107c10 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5eda000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5edc108e00 0x7f5edc19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc1097c0 0x7f5edc1a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5eda000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5edc108e00 0x7f5edc19e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42418/0 (socket says 172.21.15.116:42418) 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc1097c0 0x7f5edc1a54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35750/0 (socket says 172.21.15.116:35750) 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5eda000640 1 -- 172.21.15.116:0/1621633181 learned_addr learned my addr 172.21.15.116:0/1621633181 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edaa00640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5edc107c10 0x7f5edc19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 -- 172.21.15.116:0/1621633181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5edc107c10 msgr2=0x7f5edc19df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5edc107c10 0x7f5edc19df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 -- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5edc108e00 msgr2=0x7f5edc19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5edc108e00 0x7f5edc19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 -- 172.21.15.116:0/1621633181 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc1a5b40 con 0x7f5edc1097c0 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.362+0000 7f5edb400640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc1097c0 0x7f5edc1a54d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400ec50 tx=0x7f5ed400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.363+0000 7f5ed3400640 1 -- 172.21.15.116:0/1621633181 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4003b70 con 0x7f5edc1097c0 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.363+0000 7f5ed3400640 1 -- 172.21.15.116:0/1621633181 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ed4003d10 con 0x7f5edc1097c0 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.363+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5edc1a5e30 con 0x7f5edc1097c0 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.363+0000 7f5ed3400640 1 -- 172.21.15.116:0/1621633181 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4010890 con 0x7f5edc1097c0 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.363+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5edc1a62a0 con 0x7f5edc1097c0 2024-03-04T07:40:13.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.364+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc10ef30 con 0x7f5edc1097c0 2024-03-04T07:40:13.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.366+0000 7f5ed3400640 1 -- 172.21.15.116:0/1621633181 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5ed4003fc0 con 0x7f5edc1097c0 2024-03-04T07:40:13.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.367+0000 7f5ed3400640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5ec00a1d80 0x7f5ec00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.367+0000 7f5ed3400640 1 -- 172.21.15.116:0/1621633181 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5ed40ce350 con 0x7f5edc1097c0 2024-03-04T07:40:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.367+0000 7f5edaa00640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5ec00a1d80 0x7f5ec00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.368+0000 7f5edaa00640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5ec00a1d80 0x7f5ec00a4240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc02c180 tx=0x7f5ecc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:13.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.371+0000 7f5ed3400640 1 -- 172.21.15.116:0/1621633181 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5ed409b740 con 0x7f5edc1097c0 2024-03-04T07:40:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.555+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5edc062270 con 0x7f5edc1097c0 2024-03-04T07:40:13.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.557+0000 7f5ed3400640 1 -- 172.21.15.116:0/1621633181 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f5ed409b040 con 0x7f5edc1097c0 2024-03-04T07:40:13.559 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:13.559 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:40:13.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5ec00a1d80 msgr2=0x7f5ec00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5ec00a1d80 0x7f5ec00a4240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc02c180 tx=0x7f5ecc02cf90 comp rx=0 tx=0).stop 2024-03-04T07:40:13.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc1097c0 msgr2=0x7f5edc1a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc1097c0 0x7f5edc1a54d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400ec50 tx=0x7f5ed400c6a0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 shutdown_connections 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5edc1097c0 0x7f5edc1a54d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5edc108e00 0x7f5edc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5ec00a1d80 0x7f5ec00a4240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 --2- 172.21.15.116:0/1621633181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5edc107c10 0x7f5edc19df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 >> 172.21.15.116:0/1621633181 conn(0x7f5edc076be0 msgr2=0x7f5edc10c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 shutdown_connections 2024-03-04T07:40:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.560+0000 7f5ee1800640 1 -- 172.21.15.116:0/1621633181 wait complete. 2024-03-04T07:40:13.578 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:40:13.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:40:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 -- 172.21.15.116:0/982619240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1014b0 msgr2=0x7ff79c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/982619240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1014b0 0x7ff79c1018d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009900 tx=0x7ff78c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 -- 172.21.15.116:0/982619240 shutdown_connections 2024-03-04T07:40:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/982619240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff79c103060 0x7ff79c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/982619240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff79c1026a0 0x7ff79c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/982619240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1014b0 0x7ff79c1018d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 -- 172.21.15.116:0/982619240 >> 172.21.15.116:0/982619240 conn(0x7ff79c0fca40 msgr2=0x7ff79c0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 -- 172.21.15.116:0/982619240 shutdown_connections 2024-03-04T07:40:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.790+0000 7ff7a1a00640 1 -- 172.21.15.116:0/982619240 wait complete. 2024-03-04T07:40:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.791+0000 7ff7a1a00640 1 Processor -- start 2024-03-04T07:40:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.791+0000 7ff7a1a00640 1 -- start start 2024-03-04T07:40:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff79c1014b0 0x7ff79c199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1026a0 0x7ff79c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff79c103060 0x7ff79c1a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c10d190 con 0x7ff79c103060 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c10d010 con 0x7ff79c1026a0 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c10d310 con 0x7ff79c1014b0 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff79c103060 0x7ff79c1a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1026a0 0x7ff79c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1026a0 0x7ff79c199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35776/0 (socket says 172.21.15.116:35776) 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 -- 172.21.15.116:0/3987370705 learned_addr learned my addr 172.21.15.116:0/3987370705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79be00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff79c1014b0 0x7ff79c199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 -- 172.21.15.116:0/3987370705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff79c1014b0 msgr2=0x7ff79c199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff79c1014b0 0x7ff79c199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 -- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff79c103060 msgr2=0x7ff79c1a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff79c103060 0x7ff79c1a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff79b400640 1 -- 172.21.15.116:0/3987370705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c1a15d0 con 0x7ff79c1026a0 2024-03-04T07:40:13.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.792+0000 7ff7a1000640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff79c103060 0x7ff79c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.793+0000 7ff79b400640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1026a0 0x7ff79c199f40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff78400e9f0 tx=0x7ff78400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.793+0000 7ff798c00640 1 -- 172.21.15.116:0/3987370705 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78400cd80 con 0x7ff79c1026a0 2024-03-04T07:40:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.793+0000 7ff798c00640 1 -- 172.21.15.116:0/3987370705 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff784010910 con 0x7ff79c1026a0 2024-03-04T07:40:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.793+0000 7ff798c00640 1 -- 172.21.15.116:0/3987370705 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784018990 con 0x7ff79c1026a0 2024-03-04T07:40:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.793+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c1a18c0 con 0x7ff79c1026a0 2024-03-04T07:40:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.793+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c1a1dc0 con 0x7ff79c1026a0 2024-03-04T07:40:13.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.794+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c1a2020 con 0x7ff79c1026a0 2024-03-04T07:40:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.796+0000 7ff798c00640 1 -- 172.21.15.116:0/3987370705 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff784010ac0 con 0x7ff79c1026a0 2024-03-04T07:40:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.798+0000 7ff798c00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7800a1e00 0x7ff7800a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.798+0000 7ff798c00640 1 -- 172.21.15.116:0/3987370705 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff7840ce090 con 0x7ff79c1026a0 2024-03-04T07:40:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.798+0000 7ff79be00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7800a1e00 0x7ff7800a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:13.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.798+0000 7ff79be00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7800a1e00 0x7ff7800a42c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff78c02c170 tx=0x7ff78c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:13.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.806+0000 7ff798c00640 1 -- 172.21.15.116:0/3987370705 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff78409ae30 con 0x7ff79c1026a0 2024-03-04T07:40:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.993+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff79c1a21f0 con 0x7ff79c1026a0 2024-03-04T07:40:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.994+0000 7ff798c00640 1 -- 172.21.15.116:0/3987370705 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff78409eed0 con 0x7ff79c1026a0 2024-03-04T07:40:13.995 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.996+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7800a1e00 msgr2=0x7ff7800a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7800a1e00 0x7ff7800a42c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff78c02c170 tx=0x7ff78c02df90 comp rx=0 tx=0).stop 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1026a0 msgr2=0x7ff79c199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1026a0 0x7ff79c199f40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff78400e9f0 tx=0x7ff78400eeb0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 shutdown_connections 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff79c103060 0x7ff79c1a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff79c1026a0 0x7ff79c199f40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff7800a1e00 0x7ff7800a42c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 --2- 172.21.15.116:0/3987370705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff79c1014b0 0x7ff79c199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 >> 172.21.15.116:0/3987370705 conn(0x7ff79c0fca40 msgr2=0x7ff79c073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 shutdown_connections 2024-03-04T07:40:13.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:13.997+0000 7ff7a1a00640 1 -- 172.21.15.116:0/3987370705 wait complete. 2024-03-04T07:40:14.016 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:13 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4062022697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:13 smithi116 ceph-mon[32150]: pgmap v385: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:13 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1621633181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.265+0000 7f5d43e00640 1 -- 172.21.15.116:0/1201742871 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c074030 msgr2=0x7f5d3c0744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.265+0000 7f5d43e00640 1 --2- 172.21.15.116:0/1201742871 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c074030 0x7f5d3c0744b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009930 tx=0x7f5d2802bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.266+0000 7f5d43e00640 1 -- 172.21.15.116:0/1201742871 shutdown_connections 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.266+0000 7f5d43e00640 1 --2- 172.21.15.116:0/1201742871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d3c10dbb0 0x7f5d3c10ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.266+0000 7f5d43e00640 1 --2- 172.21.15.116:0/1201742871 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c074030 0x7f5d3c0744b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.266+0000 7f5d43e00640 1 --2- 172.21.15.116:0/1201742871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d3c075610 0x7f5d3c073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.266+0000 7f5d43e00640 1 -- 172.21.15.116:0/1201742871 >> 172.21.15.116:0/1201742871 conn(0x7f5d3c0fcb70 msgr2=0x7f5d3c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.266+0000 7f5d43e00640 1 -- 172.21.15.116:0/1201742871 shutdown_connections 2024-03-04T07:40:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.266+0000 7f5d43e00640 1 -- 172.21.15.116:0/1201742871 wait complete. 2024-03-04T07:40:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 Processor -- start 2024-03-04T07:40:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 -- start start 2024-03-04T07:40:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d3c074030 0x7f5d3c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c075610 0x7f5d3c19e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d3c10dbb0 0x7f5d3c1a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c113880 con 0x7f5d3c10dbb0 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c113700 con 0x7f5d3c075610 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d43e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c113a00 con 0x7f5d3c074030 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d41e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d3c10dbb0 0x7f5d3c1a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d40a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c075610 0x7f5d3c19e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d41400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d3c074030 0x7f5d3c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d40a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c075610 0x7f5d3c19e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35806/0 (socket says 172.21.15.116:35806) 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.267+0000 7f5d40a00640 1 -- 172.21.15.116:0/7940385 learned_addr learned my addr 172.21.15.116:0/7940385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d41400640 1 -- 172.21.15.116:0/7940385 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c075610 msgr2=0x7f5d3c19e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d41400640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c075610 0x7f5d3c19e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d41400640 1 -- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d3c10dbb0 msgr2=0x7f5d3c1a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d41400640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d3c10dbb0 0x7f5d3c1a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d41400640 1 -- 172.21.15.116:0/7940385 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c1a5b90 con 0x7f5d3c074030 2024-03-04T07:40:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d40a00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c075610 0x7f5d3c19e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d41400640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d3c074030 0x7f5d3c19df70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00e9f0 tx=0x7f5d2c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d3a000640 1 -- 172.21.15.116:0/7940385 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c00ccc0 con 0x7f5d3c074030 2024-03-04T07:40:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c1a5e80 con 0x7f5d3c074030 2024-03-04T07:40:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d3a000640 1 -- 172.21.15.116:0/7940385 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d2c010910 con 0x7f5d3c074030 2024-03-04T07:40:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d3a000640 1 -- 172.21.15.116:0/7940385 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c018960 con 0x7f5d3c074030 2024-03-04T07:40:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.268+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c1a62e0 con 0x7f5d3c074030 2024-03-04T07:40:14.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:13 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4062022697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:14.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:13 smithi152 ceph-mon[37154]: pgmap v385: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:14.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:13 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1621633181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.270+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c103d80 con 0x7f5d3c074030 2024-03-04T07:40:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.271+0000 7f5d3a000640 1 -- 172.21.15.116:0/7940385 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5d2c010ac0 con 0x7f5d3c074030 2024-03-04T07:40:14.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.273+0000 7f5d3a000640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d200a1d80 0x7f5d200a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.273+0000 7f5d3a000640 1 -- 172.21.15.116:0/7940385 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5d2c0ce150 con 0x7f5d3c074030 2024-03-04T07:40:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.273+0000 7f5d40a00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d200a1d80 0x7f5d200a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.274+0000 7f5d40a00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d200a1d80 0x7f5d200a4240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5d28006010 tx=0x7f5d2802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.277+0000 7f5d3a000640 1 -- 172.21.15.116:0/7940385 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5d2c09aef0 con 0x7f5d3c074030 2024-03-04T07:40:14.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:13 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4062022697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:14.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:13 smithi133 ceph-mon[36505]: pgmap v385: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:14.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:13 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1621633181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:14.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.474+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d3c0622a0 con 0x7f5d3c074030 2024-03-04T07:40:14.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.475+0000 7f5d3a000640 1 -- 172.21.15.116:0/7940385 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5d2c09ef90 con 0x7f5d3c074030 2024-03-04T07:40:14.476 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d200a1d80 msgr2=0x7f5d200a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d200a1d80 0x7f5d200a4240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5d28006010 tx=0x7f5d2802df90 comp rx=0 tx=0).stop 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d3c074030 msgr2=0x7f5d3c19df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d3c074030 0x7f5d3c19df70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00e9f0 tx=0x7f5d2c00eeb0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 shutdown_connections 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d3c10dbb0 0x7f5d3c1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5d3c075610 0x7f5d3c19e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d200a1d80 0x7f5d200a4240 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 --2- 172.21.15.116:0/7940385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d3c074030 0x7f5d3c19df70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 >> 172.21.15.116:0/7940385 conn(0x7f5d3c0fcb70 msgr2=0x7f5d3c100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:14.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 shutdown_connections 2024-03-04T07:40:14.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.478+0000 7f5d43e00640 1 -- 172.21.15.116:0/7940385 wait complete. 2024-03-04T07:40:14.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:40:14.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 -- 172.21.15.116:0/1637666726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241048e0 msgr2=0x7f5624104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 --2- 172.21.15.116:0/1637666726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241048e0 0x7f5624104d60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5610009900 tx=0x7f561002bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 -- 172.21.15.116:0/1637666726 shutdown_connections 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 --2- 172.21.15.116:0/1637666726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56241052a0 0x7f562410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 --2- 172.21.15.116:0/1637666726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241048e0 0x7f5624104d60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 --2- 172.21.15.116:0/1637666726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56241036f0 0x7f5624103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 -- 172.21.15.116:0/1637666726 >> 172.21.15.116:0/1637666726 conn(0x7f56240fec80 msgr2=0x7f56241010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.708+0000 7f5623200640 1 -- 172.21.15.116:0/1637666726 shutdown_connections 2024-03-04T07:40:14.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.709+0000 7f5623200640 1 -- 172.21.15.116:0/1637666726 wait complete. 2024-03-04T07:40:14.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.709+0000 7f5623200640 1 Processor -- start 2024-03-04T07:40:14.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5623200640 1 -- start start 2024-03-04T07:40:14.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5623200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56241036f0 0x7f56241999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5623200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56241048e0 0x7f5624199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5623200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241052a0 0x7f56241a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5623200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562410f3a0 con 0x7f56241036f0 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5623200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562410f220 con 0x7f56241052a0 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5623200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562410f520 con 0x7f56241048e0 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241052a0 0x7f56241a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241052a0 0x7f56241a0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35810/0 (socket says 172.21.15.116:35810) 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 -- 172.21.15.116:0/1890857705 learned_addr learned my addr 172.21.15.116:0/1890857705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5621e00640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56241036f0 0x7f56241999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5621400640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56241048e0 0x7f5624199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 -- 172.21.15.116:0/1890857705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56241048e0 msgr2=0x7f5624199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56241048e0 0x7f5624199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 -- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56241036f0 msgr2=0x7f56241999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56241036f0 0x7f56241999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.710+0000 7f5622800640 1 -- 172.21.15.116:0/1890857705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56241a15e0 con 0x7f56241052a0 2024-03-04T07:40:14.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.711+0000 7f5622800640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241052a0 0x7f56241a0f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f561c00eb10 tx=0x7f561c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:14.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.711+0000 7f561aa00640 1 -- 172.21.15.116:0/1890857705 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c004b20 con 0x7f56241052a0 2024-03-04T07:40:14.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.711+0000 7f561aa00640 1 -- 172.21.15.116:0/1890857705 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f561c004cc0 con 0x7f56241052a0 2024-03-04T07:40:14.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.711+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56241a18d0 con 0x7f56241052a0 2024-03-04T07:40:14.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.711+0000 7f561aa00640 1 -- 172.21.15.116:0/1890857705 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c010760 con 0x7f56241052a0 2024-03-04T07:40:14.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.711+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56241a1df0 con 0x7f56241052a0 2024-03-04T07:40:14.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.712+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562410a9b0 con 0x7f56241052a0 2024-03-04T07:40:14.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.714+0000 7f561aa00640 1 -- 172.21.15.116:0/1890857705 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f561c005020 con 0x7f56241052a0 2024-03-04T07:40:14.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.715+0000 7f561aa00640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56080a1e00 0x7f56080a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:14.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.715+0000 7f561aa00640 1 -- 172.21.15.116:0/1890857705 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f561c0ce2e0 con 0x7f56241052a0 2024-03-04T07:40:14.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.716+0000 7f5621e00640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56080a1e00 0x7f56080a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:14.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.716+0000 7f5621e00640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56080a1e00 0x7f56080a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5624104740 tx=0x7f561400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:14.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.719+0000 7f561aa00640 1 -- 172.21.15.116:0/1890857705 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f561c09b6d0 con 0x7f56241052a0 2024-03-04T07:40:14.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.919+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56240621d0 con 0x7f56241052a0 2024-03-04T07:40:14.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.920+0000 7f561aa00640 1 -- 172.21.15.116:0/1890857705 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f561c09afd0 con 0x7f56241052a0 2024-03-04T07:40:14.921 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.922+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56080a1e00 msgr2=0x7f56080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.922+0000 7f5623200640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56080a1e00 0x7f56080a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5624104740 tx=0x7f561400b390 comp rx=0 tx=0).stop 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.922+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241052a0 msgr2=0x7f56241a0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.922+0000 7f5623200640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241052a0 0x7f56241a0f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f561c00eb10 tx=0x7f561c00c2d0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 shutdown_connections 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56241052a0 0x7f56241a0f70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56080a1e00 0x7f56080a42c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56241048e0 0x7f5624199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 --2- 172.21.15.116:0/1890857705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56241036f0 0x7f56241999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 >> 172.21.15.116:0/1890857705 conn(0x7f56240fec80 msgr2=0x7f5624107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 shutdown_connections 2024-03-04T07:40:14.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:14.923+0000 7f5623200640 1 -- 172.21.15.116:0/1890857705 wait complete. 2024-03-04T07:40:14.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 -- 172.21.15.116:0/1156196617 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd8104a00 msgr2=0x7f7dd8104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 --2- 172.21.15.116:0/1156196617 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd8104a00 0x7f7dd8104e80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009900 tx=0x7f7dc002bc40 comp rx=0 tx=0).stop 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 -- 172.21.15.116:0/1156196617 shutdown_connections 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 --2- 172.21.15.116:0/1156196617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7dd81053c0 0x7f7dd810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 --2- 172.21.15.116:0/1156196617 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd8104a00 0x7f7dd8104e80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 --2- 172.21.15.116:0/1156196617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 0x7f7dd8103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 -- 172.21.15.116:0/1156196617 >> 172.21.15.116:0/1156196617 conn(0x7f7dd80fedc0 msgr2=0x7f7dd81011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 -- 172.21.15.116:0/1156196617 shutdown_connections 2024-03-04T07:40:15.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.155+0000 7f7ddd400640 1 -- 172.21.15.116:0/1156196617 wait complete. 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.156+0000 7f7ddd400640 1 Processor -- start 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.156+0000 7f7ddd400640 1 -- start start 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7ddd400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 0x7f7dd8079380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7ddd400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7dd8104a00 0x7f7dd80798c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7ddd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd81053c0 0x7f7dd8075ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7ddd400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd810f4c0 con 0x7f7dd8103810 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7ddd400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd810f340 con 0x7f7dd81053c0 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7ddd400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd810f640 con 0x7f7dd8104a00 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 0x7f7dd8079380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 0x7f7dd8079380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42486/0 (socket says 172.21.15.116:42486) 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 -- 172.21.15.116:0/3897715518 learned_addr learned my addr 172.21.15.116:0/3897715518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd7200640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd81053c0 0x7f7dd8075ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dcde00640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7dd8104a00 0x7f7dd80798c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 -- 172.21.15.116:0/3897715518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7dd8104a00 msgr2=0x7f7dd80798c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7dd8104a00 0x7f7dd80798c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 -- 172.21.15.116:0/3897715518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd81053c0 msgr2=0x7f7dd8075ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd81053c0 0x7f7dd8075ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd6800640 1 -- 172.21.15.116:0/3897715518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80767b0 con 0x7f7dd8103810 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.157+0000 7f7dd7200640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd81053c0 0x7f7dd8075ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.158+0000 7f7dd6800640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 0x7f7dd8079380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800e9c0 tx=0x7f7dc800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.158+0000 7f7dcfe00640 1 -- 172.21.15.116:0/3897715518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc800cd80 con 0x7f7dd8103810 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.158+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd81a5f20 con 0x7f7dd8103810 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.158+0000 7f7dcfe00640 1 -- 172.21.15.116:0/3897715518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dc80035f0 con 0x7f7dd8103810 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.158+0000 7f7dcfe00640 1 -- 172.21.15.116:0/3897715518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8010690 con 0x7f7dd8103810 2024-03-04T07:40:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.158+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd81a62d0 con 0x7f7dd8103810 2024-03-04T07:40:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.159+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd8068810 con 0x7f7dd8103810 2024-03-04T07:40:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.161+0000 7f7dcfe00640 1 -- 172.21.15.116:0/3897715518 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f7dc80037a0 con 0x7f7dd8103810 2024-03-04T07:40:15.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.163+0000 7f7dcfe00640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7dbc0a1d80 0x7f7dbc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.163+0000 7f7dcfe00640 1 -- 172.21.15.116:0/3897715518 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7dc80ce180 con 0x7f7dd8103810 2024-03-04T07:40:15.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.163+0000 7f7dcde00640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7dbc0a1d80 0x7f7dbc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.164+0000 7f7dcde00640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7dbc0a1d80 0x7f7dbc0a4240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7dc002c250 tx=0x7f7dc002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:15.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.168+0000 7f7dcfe00640 1 -- 172.21.15.116:0/3897715518 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7dc809b570 con 0x7f7dd8103810 2024-03-04T07:40:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3987370705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/7940385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:15.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:14 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3987370705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:15.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:14 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/7940385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:15.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:14 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3987370705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:15.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:14 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/7940385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:15.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.404+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dd802cff0 con 0x7f7dd8103810 2024-03-04T07:40:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.406+0000 7f7dcfe00640 1 -- 172.21.15.116:0/3897715518 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f7dc809ae70 con 0x7f7dd8103810 2024-03-04T07:40:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:15.407 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:40:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.408+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7dbc0a1d80 msgr2=0x7f7dbc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.408+0000 7f7ddd400640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7dbc0a1d80 0x7f7dbc0a4240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7dc002c250 tx=0x7f7dc002cf90 comp rx=0 tx=0).stop 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.408+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 msgr2=0x7f7dd8079380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.408+0000 7f7ddd400640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 0x7f7dd8079380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7dc800e9c0 tx=0x7f7dc800ee80 comp rx=0 tx=0).stop 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 shutdown_connections 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dd81053c0 0x7f7dd8075ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7dbc0a1d80 0x7f7dbc0a4240 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7dd8104a00 0x7f7dd80798c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 --2- 172.21.15.116:0/3897715518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dd8103810 0x7f7dd8079380 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 >> 172.21.15.116:0/3897715518 conn(0x7f7dd80fedc0 msgr2=0x7f7dd8107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 shutdown_connections 2024-03-04T07:40:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.409+0000 7f7ddd400640 1 -- 172.21.15.116:0/3897715518 wait complete. 2024-03-04T07:40:15.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:40:15.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 -- 172.21.15.116:0/1565348511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd58104810 msgr2=0x7fdd58104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 --2- 172.21.15.116:0/1565348511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd58104810 0x7fdd58104c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009900 tx=0x7fdd4002bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:15.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 -- 172.21.15.116:0/1565348511 shutdown_connections 2024-03-04T07:40:15.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 --2- 172.21.15.116:0/1565348511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd581051d0 0x7fdd5810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 --2- 172.21.15.116:0/1565348511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd58104810 0x7fdd58104c90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 --2- 172.21.15.116:0/1565348511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd58103620 0x7fdd58103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 -- 172.21.15.116:0/1565348511 >> 172.21.15.116:0/1565348511 conn(0x7fdd580febb0 msgr2=0x7fdd58100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:15.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 -- 172.21.15.116:0/1565348511 shutdown_connections 2024-03-04T07:40:15.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.633+0000 7fdd57200640 1 -- 172.21.15.116:0/1565348511 wait complete. 2024-03-04T07:40:15.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.634+0000 7fdd57200640 1 Processor -- start 2024-03-04T07:40:15.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd57200640 1 -- start start 2024-03-04T07:40:15.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd57200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd58103620 0x7fdd581998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd57200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd58104810 0x7fdd58199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd57200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd581051d0 0x7fdd581a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd57200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5810f2b0 con 0x7fdd58104810 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd57200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5810f130 con 0x7fdd581051d0 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd57200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5810f430 con 0x7fdd58103620 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd56800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd581051d0 0x7fdd581a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd4fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd58104810 0x7fdd58199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd55e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd58103620 0x7fdd581998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd56800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd581051d0 0x7fdd581a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35838/0 (socket says 172.21.15.116:35838) 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.635+0000 7fdd56800640 1 -- 172.21.15.116:0/3056290145 learned_addr learned my addr 172.21.15.116:0/3056290145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd55e00640 1 -- 172.21.15.116:0/3056290145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd581051d0 msgr2=0x7fdd581a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd55e00640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd581051d0 0x7fdd581a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd55e00640 1 -- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd58104810 msgr2=0x7fdd58199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd55e00640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd58104810 0x7fdd58199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd55e00640 1 -- 172.21.15.116:0/3056290145 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd581a14f0 con 0x7fdd58103620 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd56800640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd581051d0 0x7fdd581a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd55e00640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd58103620 0x7fdd581998d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800e9c0 tx=0x7fdd4800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd4ea00640 1 -- 172.21.15.116:0/3056290145 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4800ccc0 con 0x7fdd58103620 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd581a17e0 con 0x7fdd58103620 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd4ea00640 1 -- 172.21.15.116:0/3056290145 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd480035f0 con 0x7fdd58103620 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd4ea00640 1 -- 172.21.15.116:0/3056290145 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4800ce60 con 0x7fdd58103620 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.636+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd581a1d50 con 0x7fdd58103620 2024-03-04T07:40:15.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.637+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5810a940 con 0x7fdd58103620 2024-03-04T07:40:15.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.640+0000 7fdd4ea00640 1 -- 172.21.15.116:0/3056290145 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fdd480037a0 con 0x7fdd58103620 2024-03-04T07:40:15.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.641+0000 7fdd4ea00640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd3c0a1e00 0x7fdd3c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:15.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.641+0000 7fdd4ea00640 1 -- 172.21.15.116:0/3056290145 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdd480ce180 con 0x7fdd58103620 2024-03-04T07:40:15.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.641+0000 7fdd4fe00640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd3c0a1e00 0x7fdd3c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:15.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.642+0000 7fdd4fe00640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd3c0a1e00 0x7fdd3c0a42c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdd4002c7b0 tx=0x7fdd40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:15.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.643+0000 7fdd4ea00640 1 -- 172.21.15.116:0/3056290145 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdd4809b570 con 0x7fdd58103620 2024-03-04T07:40:15.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.841+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd5802cff0 con 0x7fdd58103620 2024-03-04T07:40:15.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.843+0000 7fdd4ea00640 1 -- 172.21.15.116:0/3056290145 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7fdd4809ae70 con 0x7fdd58103620 2024-03-04T07:40:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:15.845 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:40:15.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd3c0a1e00 msgr2=0x7fdd3c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd3c0a1e00 0x7fdd3c0a42c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdd4002c7b0 tx=0x7fdd40005fd0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd58103620 msgr2=0x7fdd581998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd58103620 0x7fdd581998d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800e9c0 tx=0x7fdd4800ee80 comp rx=0 tx=0).stop 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 shutdown_connections 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd581051d0 0x7fdd581a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd58104810 0x7fdd58199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd3c0a1e00 0x7fdd3c0a42c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 --2- 172.21.15.116:0/3056290145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd58103620 0x7fdd581998d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.846+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 >> 172.21.15.116:0/3056290145 conn(0x7fdd580febb0 msgr2=0x7fdd58107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.847+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 shutdown_connections 2024-03-04T07:40:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:15.847+0000 7fdd57200640 1 -- 172.21.15.116:0/3056290145 wait complete. 2024-03-04T07:40:15.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-04T07:37:54.573948+0000', 'modified': '2024-03-04T07:37:58.240682+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24581}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24553': {'gid': 24553, 'name': 'g', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1000386690', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1000386690}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1000386690}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24581': {'gid': 24581, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.133:6839/1914821860', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.133:6838', 'nonce': 1914821860}, {'type': 'v1', 'addr': '172.21.15.133:6839', 'nonce': 1914821860}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24581, 'qdb_cluster': [24581]} 2024-03-04T07:40:15.867 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T07:40:15.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-04T07:40:16.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 -- 172.21.15.116:0/510034172 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c075f90 msgr2=0x7eff8c0763b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 --2- 172.21.15.116:0/510034172 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c075f90 0x7eff8c0763b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009ca0 tx=0x7eff7c02c0f0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 -- 172.21.15.116:0/510034172 shutdown_connections 2024-03-04T07:40:16.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 --2- 172.21.15.116:0/510034172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c077b70 0x7eff8c078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 --2- 172.21.15.116:0/510034172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff8c0771d0 0x7eff8c077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 --2- 172.21.15.116:0/510034172 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c075f90 0x7eff8c0763b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 -- 172.21.15.116:0/510034172 >> 172.21.15.116:0/510034172 conn(0x7eff8c0fed80 msgr2=0x7eff8c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:16.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.080+0000 7eff91a00640 1 -- 172.21.15.116:0/510034172 shutdown_connections 2024-03-04T07:40:16.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.081+0000 7eff91a00640 1 -- 172.21.15.116:0/510034172 wait complete. 2024-03-04T07:40:16.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.081+0000 7eff91a00640 1 Processor -- start 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff91a00640 1 -- start start 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff91a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c075f90 0x7eff8c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff91a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff8c0771d0 0x7eff8c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff91a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c077b70 0x7eff8c1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff91a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c1138a0 con 0x7eff8c075f90 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff91a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c113720 con 0x7eff8c077b70 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff91a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c113a20 con 0x7eff8c0771d0 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff8aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c075f90 0x7eff8c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff8aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c075f90 0x7eff8c19df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49834/0 (socket says 172.21.15.116:49834) 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff8b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c077b70 0x7eff8c1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff8aa00640 1 -- 172.21.15.116:0/2761798160 learned_addr learned my addr 172.21.15.116:0/2761798160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.082+0000 7eff8a000640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff8c0771d0 0x7eff8c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff8b400640 1 -- 172.21.15.116:0/2761798160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff8c0771d0 msgr2=0x7eff8c19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff8b400640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff8c0771d0 0x7eff8c19e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff8b400640 1 -- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c075f90 msgr2=0x7eff8c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff8b400640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c075f90 0x7eff8c19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff8b400640 1 -- 172.21.15.116:0/2761798160 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8c1a5b70 con 0x7eff8c077b70 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff8aa00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c075f90 0x7eff8c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff8b400640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c077b70 0x7eff8c1a5500 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e970 tx=0x7eff8400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff83400640 1 -- 172.21.15.116:0/2761798160 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400ccc0 con 0x7eff8c077b70 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff83400640 1 -- 172.21.15.116:0/2761798160 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff84005020 con 0x7eff8c077b70 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff83400640 1 -- 172.21.15.116:0/2761798160 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84004a00 con 0x7eff8c077b70 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8c1a5e60 con 0x7eff8c077b70 2024-03-04T07:40:16.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.083+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8c1a62d0 con 0x7eff8c077b70 2024-03-04T07:40:16.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.086+0000 7eff83400640 1 -- 172.21.15.116:0/2761798160 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7eff840051d0 con 0x7eff8c077b70 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.086+0000 7eff94825740 1 Processor -- start 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.086+0000 7eff94825740 1 -- start start 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff94825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597fe700 0x55f4597feb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff94825740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ff0f0 0x55f459803bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff94825740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f4598066f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff94825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f459763600 con 0x55f4597fe700 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff94825740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45970c930 con 0x55f459804260 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff94825740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45975f6a0 con 0x55f4597ff0f0 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff8a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597fe700 0x55f4597feb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff8a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597fe700 0x55f4597feb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49844/0 (socket says 172.21.15.116:49844) 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff8a000640 1 -- 172.21.15.116:0/2669695101 learned_addr learned my addr 172.21.15.116:0/2669695101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff8aa00640 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ff0f0 0x55f459803bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff8a000640 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f4598066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff8a000640 1 -- 172.21.15.116:0/2669695101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ff0f0 msgr2=0x55f459803bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.087+0000 7eff8a000640 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ff0f0 0x55f459803bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff8a000640 1 -- 172.21.15.116:0/2669695101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 msgr2=0x55f4598066f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff8a000640 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f4598066f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff8a000640 1 -- 172.21.15.116:0/2669695101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4594782d0 con 0x55f4597fe700 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff8aa00640 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ff0f0 0x55f459803bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff83400640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff700a1d80 0x7eff700a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff8a000640 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597fe700 0x55f4597feb20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7eff7400be20 tx=0x7eff74004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ce337e93f54511 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff83400640 1 -- 172.21.15.116:0/2761798160 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7eff840cdd50 con 0x7eff8c077b70 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff8aa00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff700a1d80 0x7eff700a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff7be00640 1 -- 172.21.15.116:0/2669695101 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74003e80 con 0x55f4597fe700 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff7be00640 1 -- 172.21.15.116:0/2669695101 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff74005500 con 0x55f4597fe700 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.088+0000 7eff7be00640 1 -- 172.21.15.116:0/2669695101 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74012450 con 0x55f4597fe700 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 -- 172.21.15.116:0/2669695101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597fe700 msgr2=0x55f4597feb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff8aa00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff700a1d80 0x7eff700a4240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7eff7c004af0 tx=0x7eff7c0325b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597fe700 0x55f4597feb20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7eff7400be20 tx=0x7eff74004160 comp rx=0 tx=0).stop 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 -- 172.21.15.116:0/2669695101 shutdown_connections 2024-03-04T07:40:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f4598066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ff0f0 0x55f459803bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 --2- 172.21.15.116:0/2669695101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597fe700 0x55f4597feb20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 -- 172.21.15.116:0/2669695101 >> 172.21.15.116:0/2669695101 conn(0x55f4597f9c70 msgr2=0x55f4597fa070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 -- 172.21.15.116:0/2669695101 shutdown_connections 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.089+0000 7eff94825740 1 -- 172.21.15.116:0/2669695101 wait complete. 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 -- start start 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 Processor -- start 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f459808bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ff0f0 0x55f4598090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f459810130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4598071b0 con 0x55f4597ff0f0 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45971fd90 con 0x55f459804260 2024-03-04T07:40:16.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f459807330 con 0x55f4597fe700 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff94825740 10 client.0 _ll_register_callbacks cb 0x55f45975b350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.090+0000 7eff8a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ff0f0 0x55f4598090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f459810130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f459810130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51120/0 (socket says 172.21.15.116:51120) 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ff0f0 0x55f4598090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49856/0 (socket says 172.21.15.116:49856) 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 -- 172.21.15.116:0/1346553026 learned_addr learned my addr 172.21.15.116:0/1346553026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8aa00640 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f459808bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 -- 172.21.15.116:0/1346553026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 msgr2=0x55f459808bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f459808bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 -- 172.21.15.116:0/1346553026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ff0f0 msgr2=0x55f4598090f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ff0f0 0x55f4598090f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 -- 172.21.15.116:0/1346553026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4595426b0 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff8b400640 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f459810130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff84096ea0 tx=0x7eff8409b200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5619d07e19281d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff78c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff78c00640 1 -- 172.21.15.116:0/1346553026 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff840b3060 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff78c00640 1 -- 172.21.15.116:0/1346553026 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7eff840a2f50 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff74005500 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.091+0000 7eff78c00640 1 -- 172.21.15.116:0/1346553026 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff8401c040 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 10 client.25106 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f459813510 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff78c00640 1 -- 172.21.15.116:0/1346553026 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff74005500 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff78c00640 1 -- 172.21.15.116:0/1346553026 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7eff84099f20 con 0x55f459804260 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 10 client.25106 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 10 client.25106 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T07:40:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 20 client.25106 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4598158b0 0x55f459814980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 4 client.25106 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f459770b40 con 0x55f4598158b0 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.092+0000 7eff8aa00640 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4598158b0 0x55f459814980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.093+0000 7eff8aa00640 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4598158b0 0x55f459814980 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c8255d6cf9b2a3a server_cookie=439da2bab5d7c55d in_seq=0 out_seq=0 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.093+0000 7eff78c00640 10 client.25106 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.093+0000 7eff78c00640 1 -- 172.21.15.116:0/1346553026 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f459770b40 con 0x55f4598158b0 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.093+0000 7eff78c00640 10 client.25106 handle_command_reply: tid=0 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.093+0000 7eff94825740 1 client.25106 shutdown 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 msgr2=0x55f459810130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f459810130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff84096ea0 tx=0x7eff8409b200 comp rx=0 tx=0).stop 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 shutdown_connections 2024-03-04T07:40:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4598158b0 0x55f459814980 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f459804260 0x55f459810130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ff0f0 0x55f4598090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff78c00640 0 client.25106 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 --2- 172.21.15.116:0/1346553026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f459808bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 >> 172.21.15.116:0/1346553026 conn(0x55f4597f9c70 msgr2=0x55f4597f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 shutdown_connections 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 1 -- 172.21.15.116:0/1346553026 wait complete. 2024-03-04T07:40:16.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.094+0000 7eff94825740 20 client.25106 trim_cache size 0 max 16384 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.112+0000 7eff94825740 1 Processor -- start 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.112+0000 7eff94825740 1 -- start start 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff94825740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ffed0 0x55f4597f8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff94825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597f9c70 0x55f4598100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff94825740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597fe700 0x55f4597f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff94825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4598150d0 con 0x55f4597f9c70 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff94825740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f459815730 con 0x55f4597fe700 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff94825740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f459815d90 con 0x55f4597ffed0 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597fe700 0x55f4597f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597fe700 0x55f4597f8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51132/0 (socket says 172.21.15.116:51132) 2024-03-04T07:40:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 -- 172.21.15.116:0/3237060456 learned_addr learned my addr 172.21.15.116:0/3237060456 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8a000640 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597f9c70 0x55f4598100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 -- 172.21.15.116:0/3237060456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ffed0 msgr2=0x55f4597f8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ffed0 0x55f4597f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 -- 172.21.15.116:0/3237060456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597f9c70 msgr2=0x55f4598100f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597f9c70 0x55f4598100f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 -- 172.21.15.116:0/3237060456 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4597f9aa0 con 0x55f4597fe700 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.113+0000 7eff8b400640 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597fe700 0x55f4597f8b60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7eff8409b2f0 tx=0x7eff84096e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52e4c7fb8878dcd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff7b400640 1 -- 172.21.15.116:0/3237060456 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff840a31e0 con 0x55f4597fe700 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff7b400640 1 -- 172.21.15.116:0/3237060456 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff840a2d50 con 0x55f4597fe700 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff7b400640 1 -- 172.21.15.116:0/3237060456 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84004e00 con 0x55f4597fe700 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 -- 172.21.15.116:0/3237060456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597fe700 msgr2=0x55f4597f8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597fe700 0x55f4597f8b60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7eff8409b2f0 tx=0x7eff84096e10 comp rx=0 tx=0).stop 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 -- 172.21.15.116:0/3237060456 shutdown_connections 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597fe700 0x55f4597f8b60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597f9c70 0x55f4598100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 --2- 172.21.15.116:0/3237060456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597ffed0 0x55f4597f8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 -- 172.21.15.116:0/3237060456 >> 172.21.15.116:0/3237060456 conn(0x55f459814cf0 msgr2=0x55f4597fa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 -- 172.21.15.116:0/3237060456 shutdown_connections 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.114+0000 7eff94825740 1 -- 172.21.15.116:0/3237060456 wait complete. 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 -- start start 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 Processor -- start 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597f9c70 0x55f45980fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f45980c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ffed0 0x55f45980cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f459816a50 con 0x55f4597ffed0 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4598163f0 con 0x55f4597f9c70 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4598170b0 con 0x55f4597fe700 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.115+0000 7eff94825740 10 client.0 _ll_register_callbacks cb 0x7eff84096ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597f9c70 0x55f45980fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597f9c70 0x55f45980fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51148/0 (socket says 172.21.15.116:51148) 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 -- 172.21.15.116:0/1569802023 learned_addr learned my addr 172.21.15.116:0/1569802023 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8b400640 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ffed0 0x55f45980cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8a000640 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f45980c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 -- 172.21.15.116:0/1569802023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 msgr2=0x55f45980c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f45980c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 -- 172.21.15.116:0/1569802023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ffed0 msgr2=0x55f45980cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ffed0 0x55f45980cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 -- 172.21.15.116:0/1569802023 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f45980d3d0 con 0x55f4597f9c70 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff8aa00640 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597f9c70 0x55f45980fe00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009a60 tx=0x7eff7c02c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c4773075c89a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff78c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff78c00640 1 -- 172.21.15.116:0/1569802023 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c056020 con 0x55f4597f9c70 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff78c00640 1 -- 172.21.15.116:0/1569802023 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7eff7c0522a0 con 0x55f4597f9c70 2024-03-04T07:40:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.116+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff840a2d50 con 0x55f4597f9c70 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff78c00640 1 -- 172.21.15.116:0/1569802023 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff7c039040 con 0x55f4597f9c70 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 10 client.25112 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f4598719f0 con 0x55f4597f9c70 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff78c00640 1 -- 172.21.15.116:0/1569802023 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff840a2d50 con 0x55f4597f9c70 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff78c00640 1 -- 172.21.15.116:0/1569802023 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7eff7c034a50 con 0x55f4597f9c70 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 10 client.25112 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 10 client.25112 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 20 client.25112 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4597ff0f0 0x55f459871c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 4 client.25112 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f45976ae90 con 0x55f4597ff0f0 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.117+0000 7eff8a000640 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4597ff0f0 0x55f459871c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff8a000640 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4597ff0f0 0x55f459871c80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6170dc78eb75a11 server_cookie=a17be4f093f9f1da in_seq=0 out_seq=0 2024-03-04T07:40:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff78c00640 10 client.25112 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff78c00640 1 -- 172.21.15.116:0/1569802023 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55f45976ae90 con 0x55f4597ff0f0 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff78c00640 10 client.25112 handle_command_reply: tid=0 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 client.25112 shutdown 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597f9c70 msgr2=0x55f45980fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597f9c70 0x55f45980fe00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009a60 tx=0x7eff7c02c0f0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 shutdown_connections 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f4597ff0f0 0x55f459871c80 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4597ffed0 0x55f45980cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff78c00640 0 client.25112 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4597fe700 0x55f45980c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 --2- 172.21.15.116:0/1569802023 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f4597f9c70 0x55f45980fe00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.118+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 >> 172.21.15.116:0/1569802023 conn(0x55f459814cf0 msgr2=0x55f4597fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 shutdown_connections 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff94825740 1 -- 172.21.15.116:0/1569802023 wait complete. 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff94825740 20 client.25112 trim_cache size 0 max 16384 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "af2ccd48-d9f8-11ee-95c6-87774f69a715", 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24581, 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-04T07:40:16.125 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-04T07:40:16.126 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 140.97365159200001, 2024-03-04T07:40:16.126 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 19, 2024-03-04T07:40:16.126 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-04T07:40:16.126 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-04T07:40:16.126 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 148.341129473 2024-03-04T07:40:16.126 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff700a1d80 msgr2=0x7eff700a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff700a1d80 0x7eff700a4240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7eff7c004af0 tx=0x7eff7c0325b0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c077b70 msgr2=0x7eff8c1a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c077b70 0x7eff8c1a5500 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e970 tx=0x7eff8400ee30 comp rx=0 tx=0).stop 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 shutdown_connections 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7eff8c077b70 0x7eff8c1a5500 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7eff700a1d80 0x7eff700a4240 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff8c0771d0 0x7eff8c19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 --2- 172.21.15.116:0/2761798160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff8c075f90 0x7eff8c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 >> 172.21.15.116:0/2761798160 conn(0x7eff8c0fed80 msgr2=0x7eff8c1020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.119+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 shutdown_connections 2024-03-04T07:40:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:16.120+0000 7eff91a00640 1 -- 172.21.15.116:0/2761798160 wait complete. 2024-03-04T07:40:16.134 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1890857705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:16.135 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3897715518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:16.135 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:15 smithi116 ceph-mon[32150]: pgmap v386: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:16.135 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:15 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3056290145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:16.137 INFO:teuthology.run_tasks:Running task kclient... 2024-03-04T07:40:16.160 INFO:tasks.kclient:Mounting kernel clients... 2024-03-04T07:40:16.160 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-04T07:40:16.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:16.160 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-04T07:40:16.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:16.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-04T07:40:16.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1890857705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:16.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3897715518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:16.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:15 smithi152 ceph-mon[37154]: pgmap v386: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:16.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:15 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3056290145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2024-03-04T07:40:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:40:16.305 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-04T07:40:16.305 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T07:40:16.305 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:16.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd shell --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-04T07:40:16.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1890857705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:16.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3897715518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:16.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:15 smithi133 ceph-mon[36505]: pgmap v386: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-04T07:40:16.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:15 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3056290145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:16.756 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T07:40:18.847 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:18 smithi116 ceph-mon[32150]: pgmap v387: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:18 smithi152 ceph-mon[37154]: pgmap v387: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:19.039 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:18 smithi133 ceph-mon[36505]: pgmap v387: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:19.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 -- 172.21.15.116:0/2892728010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 msgr2=0x7fe1dc108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 --2- 172.21.15.116:0/2892728010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc108570 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009990 tx=0x7fe1d0009ca0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 -- 172.21.15.116:0/2892728010 shutdown_connections 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 --2- 172.21.15.116:0/2892728010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc108570 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 --2- 172.21.15.116:0/2892728010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1dc102b00 0x7fe1dc10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 --2- 172.21.15.116:0/2892728010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc102140 0x7fe1dc1025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 -- 172.21.15.116:0/2892728010 >> 172.21.15.116:0/2892728010 conn(0x7fe1dc079210 msgr2=0x7fe1dc079620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 -- 172.21.15.116:0/2892728010 shutdown_connections 2024-03-04T07:40:19.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.175+0000 7fe1e4800700 1 -- 172.21.15.116:0/2892728010 wait complete. 2024-03-04T07:40:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.176+0000 7fe1e4800700 1 Processor -- start 2024-03-04T07:40:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.177+0000 7fe1e4800700 1 -- start start 2024-03-04T07:40:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.177+0000 7fe1e4800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc102140 0x7fe1dc0743c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:19.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.177+0000 7fe1e4800700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1dc102b00 0x7fe1dc074900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.177+0000 7fe1e4800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc074e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.177+0000 7fe1e4800700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc112420 con 0x7fe1dc108190 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.177+0000 7fe1e4800700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1122a0 con 0x7fe1dc102b00 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.177+0000 7fe1e4800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1125a0 con 0x7fe1dc102140 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc074e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1400700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1dc102b00 0x7fe1dc074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e2800700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc074e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49884/0 (socket says 172.21.15.116:49884) 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e2800700 1 -- 172.21.15.116:0/3617581096 learned_addr learned my addr 172.21.15.116:0/3617581096 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:19.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1e00700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc102140 0x7fe1dc0743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1e00700 1 -- 172.21.15.116:0/3617581096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1dc102b00 msgr2=0x7fe1dc074900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1e00700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1dc102b00 0x7fe1dc074900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1e00700 1 -- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 msgr2=0x7fe1dc074e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1e00700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc074e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1e00700 1 -- 172.21.15.116:0/3617581096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc1a6e20 con 0x7fe1dc102140 2024-03-04T07:40:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1400700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1dc102b00 0x7fe1dc074900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:19.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e2800700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc074e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:19.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.178+0000 7fe1e1e00700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc102140 0x7fe1dc0743c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b2e0 tx=0x7fe1d00047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:19.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.179+0000 7fe1d6a00700 1 -- 172.21.15.116:0/3617581096 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0026070 con 0x7fe1dc102140 2024-03-04T07:40:19.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.179+0000 7fe1d6a00700 1 -- 172.21.15.116:0/3617581096 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1d000fde0 con 0x7fe1dc102140 2024-03-04T07:40:19.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.179+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc1a70b0 con 0x7fe1dc102140 2024-03-04T07:40:19.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.179+0000 7fe1d6a00700 1 -- 172.21.15.116:0/3617581096 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00173f0 con 0x7fe1dc102140 2024-03-04T07:40:19.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.179+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc1a7500 con 0x7fe1dc102140 2024-03-04T07:40:19.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.180+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc10a650 con 0x7fe1dc102140 2024-03-04T07:40:19.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.182+0000 7fe1d6a00700 1 -- 172.21.15.116:0/3617581096 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe1d0029070 con 0x7fe1dc102140 2024-03-04T07:40:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.183+0000 7fe1d6a00700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1c40a1e10 0x7fe1c40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.183+0000 7fe1d6a00700 1 -- 172.21.15.116:0/3617581096 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe1d00cf240 con 0x7fe1dc102140 2024-03-04T07:40:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.183+0000 7fe1e1400700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1c40a1e10 0x7fe1c40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:19.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.184+0000 7fe1e1400700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1c40a1e10 0x7fe1c40a42d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc079880 tx=0x7fe1cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.186+0000 7fe1d6a00700 1 -- 172.21.15.116:0/3617581096 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe1d0030070 con 0x7fe1dc102140 2024-03-04T07:40:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.453+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe1dc10a030 con 0x7fe1dc102140 2024-03-04T07:40:19.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.454+0000 7fe1d6a00700 1 -- 172.21.15.116:0/3617581096 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe1d0013070 con 0x7fe1dc102140 2024-03-04T07:40:19.455 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T07:40:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1c40a1e10 msgr2=0x7fe1c40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1c40a1e10 0x7fe1c40a42d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc079880 tx=0x7fe1cc009260 comp rx=0 tx=0).stop 2024-03-04T07:40:19.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc102140 msgr2=0x7fe1dc0743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc102140 0x7fe1dc0743c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b2e0 tx=0x7fe1d00047b0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 shutdown_connections 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1dc108190 0x7fe1dc074e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1c40a1e10 0x7fe1c40a42d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1dc102140 0x7fe1dc0743c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 --2- 172.21.15.116:0/3617581096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe1dc102b00 0x7fe1dc074900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 >> 172.21.15.116:0/3617581096 conn(0x7fe1dc079210 msgr2=0x7fe1dc100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 shutdown_connections 2024-03-04T07:40:19.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:19.458+0000 7fe1e4800700 1 -- 172.21.15.116:0/3617581096 wait complete. 2024-03-04T07:40:19.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:19 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3617581096' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T07:40:20.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:19 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3617581096' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T07:40:20.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:19 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3617581096' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T07:40:20.313 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T07:40:20.313 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T07:40:20.313 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-04T07:40:20.313 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T07:40:20.313 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:20.313 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T07:40:20.313 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T07:40:20.314 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T07:40:20.314 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T07:40:20.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:20.314 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-04T07:40:20.364 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T07:40:20.364 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T07:40:20.364 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T07:40:20.364 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T07:40:20.365 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 41907sec preferred_lft 41907sec 2024-03-04T07:40:20.366 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link proto kernel_ll 2024-03-04T07:40:20.366 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-04T07:40:20.366 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T07:40:20.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:20.366 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T07:40:20.366 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T07:40:20.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-04T07:40:20.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-04T07:40:20.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-04T07:40:20.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T07:40:20.367 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T07:40:20.695 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:20 smithi116 ceph-mon[32150]: pgmap v388: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:20.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:20.783 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T07:40:20.854 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-04T07:40:20.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:20.858 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-04T07:40:20.909 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-04T07:40:20.909 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T07:40:20.909 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T07:40:20.909 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T07:40:20.909 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T07:40:20.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:20.910 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T07:40:20.910 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T07:40:20.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T07:40:20.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T07:40:20.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T07:40:20.910 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T07:40:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:20 smithi152 ceph-mon[37154]: pgmap v388: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:21.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:20 smithi133 ceph-mon[36505]: pgmap v388: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:21.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:21.612 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-04T07:40:21.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:21.669 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-04T07:40:21.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:21.719 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T07:40:21.719 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T07:40:21.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T07:40:21.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-04T07:40:21.720 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T07:40:21.857 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-04T07:40:21.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-04T07:40:21.858 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T07:40:22.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:22.144 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2024-03-04T07:40:22.144 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-04T07:40:22.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-04T07:40:22.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T07:40:22.144 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-04T07:40:22.311 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T07:40:22.312 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:22.364 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T07:40:22.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T07:40:22.365 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:22.410 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-04T07:40:22.410 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-04T07:40:22.410 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/9d10f24b-566a-421f-bbda-ac3ac0eb7a11 2024-03-04T07:40:22.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:22.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/9d10f24b-566a-421f-bbda-ac3ac0eb7a11 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-04T07:40:22.592 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-04T07:40:22.592 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-04T07:40:22.821 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:22 smithi116 ceph-mon[32150]: pgmap v389: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:22 smithi152 ceph-mon[37154]: pgmap v389: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:23.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:22 smithi133 ceph-mon[36505]: pgmap v389: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-03-04T07:40:23.222 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-04T07:40:23.222 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-04T07:40:23.222 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.116:3300,172.21.15.133:3300,172.21.15.152:3300" 2024-03-04T07:40:23.222 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: trying mount with new device syntax: 0@af2ccd48-d9f8-11ee-95c6-87774f69a715.cephfs=/volumes/qa/sv_1/9d10f24b-566a-421f-bbda-ac3ac0eb7a11 2024-03-04T07:40:23.222 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.116:3300/172.21.15.133:3300/172.21.15.152:3300" will pass to kernel 2024-03-04T07:40:23.231 INFO:teuthology.orchestra.run.smithi116.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T07:40:23.231 INFO:teuthology.orchestra.run.smithi116.stdout: You just mounted a file system that supports labels which does not 2024-03-04T07:40:23.231 INFO:teuthology.orchestra.run.smithi116.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T07:40:23.231 INFO:teuthology.orchestra.run.smithi116.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T07:40:23.231 INFO:teuthology.orchestra.run.smithi116.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T07:40:23.237 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T07:40:23.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:23.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:23.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:23.331 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T07:40:23.389 INFO:teuthology.orchestra.run.smithi116.stdout:client25121 2024-03-04T07:40:23.389 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:40:23.390 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T07:40:23.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:23.442 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T07:40:23.542 INFO:teuthology.orchestra.run.smithi116.stdout:client25121 2024-03-04T07:40:23.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:23.543 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T07:40:23.595 INFO:teuthology.orchestra.run.smithi116.stdout:client25121 2024-03-04T07:40:23.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:23.596 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715.client25121/status') 2024-03-04T07:40:23.681 INFO:teuthology.orchestra.run.smithi116.stdout:instance: client.25121 (3)172.21.15.116:0/136102915 2024-03-04T07:40:23.682 INFO:teuthology.orchestra.run.smithi116.stdout:blocklisted: false 2024-03-04T07:40:23.682 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records in 2024-03-04T07:40:23.682 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records out 2024-03-04T07:40:23.682 INFO:teuthology.orchestra.run.smithi116.stderr:71 bytes copied, 7.1036e-05 s, 999 kB/s 2024-03-04T07:40:23.687 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-04T07:40:23.698 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-04T07:40:23.721 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:40:23.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 -- 172.21.15.116:0/918058480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74077b70 msgr2=0x7f0b74078410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:23.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/918058480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74077b70 0x7f0b74078410 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009900 tx=0x7f0b6c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:40:23.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 -- 172.21.15.116:0/918058480 shutdown_connections 2024-03-04T07:40:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/918058480 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74077b70 0x7f0b74078410 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/918058480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b740771d0 0x7f0b74077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/918058480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b74075f90 0x7f0b740763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 -- 172.21.15.116:0/918058480 >> 172.21.15.116:0/918058480 conn(0x7f0b740fed80 msgr2=0x7f0b741011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 -- 172.21.15.116:0/918058480 shutdown_connections 2024-03-04T07:40:23.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.940+0000 7f0b7ba00640 1 -- 172.21.15.116:0/918058480 wait complete. 2024-03-04T07:40:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.941+0000 7f0b7ba00640 1 Processor -- start 2024-03-04T07:40:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.941+0000 7f0b7ba00640 1 -- start start 2024-03-04T07:40:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b7ba00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74075f90 0x7f0b7419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b7ba00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b740771d0 0x7f0b7419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b7ba00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b74077b70 0x7f0b741a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:23.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b7ba00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b741138e0 con 0x7f0b74077b70 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b7ba00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b74113760 con 0x7f0b74075f90 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b7ba00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b74113a60 con 0x7f0b740771d0 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74075f90 0x7f0b7419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74075f90 0x7f0b7419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51172/0 (socket says 172.21.15.116:51172) 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 -- 172.21.15.116:0/1709946434 learned_addr learned my addr 172.21.15.116:0/1709946434 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79a00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b74077b70 0x7f0b741a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b73e00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b740771d0 0x7f0b7419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 -- 172.21.15.116:0/1709946434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b740771d0 msgr2=0x7f0b7419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b740771d0 0x7f0b7419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 -- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b74077b70 msgr2=0x7f0b741a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b74077b70 0x7f0b741a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.942+0000 7f0b79000640 1 -- 172.21.15.116:0/1709946434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b741a5ac0 con 0x7f0b74075f90 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.943+0000 7f0b79000640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74075f90 0x7f0b7419dea0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e9f0 tx=0x7f0b6400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.943+0000 7f0b71600640 1 -- 172.21.15.116:0/1709946434 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400cd80 con 0x7f0b74075f90 2024-03-04T07:40:23.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.943+0000 7f0b71600640 1 -- 172.21.15.116:0/1709946434 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b64005500 con 0x7f0b74075f90 2024-03-04T07:40:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.943+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b741a5db0 con 0x7f0b74075f90 2024-03-04T07:40:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.943+0000 7f0b71600640 1 -- 172.21.15.116:0/1709946434 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64010690 con 0x7f0b74075f90 2024-03-04T07:40:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.943+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b741a6220 con 0x7f0b74075f90 2024-03-04T07:40:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.944+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b74110280 con 0x7f0b74075f90 2024-03-04T07:40:23.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.946+0000 7f0b71600640 1 -- 172.21.15.116:0/1709946434 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f0b640046c0 con 0x7f0b74075f90 2024-03-04T07:40:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.947+0000 7f0b71600640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b540a1e00 0x7f0b540a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.947+0000 7f0b71600640 1 -- 172.21.15.116:0/1709946434 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0b640ce150 con 0x7f0b74075f90 2024-03-04T07:40:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.947+0000 7f0b73e00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b540a1e00 0x7f0b540a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:23.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.948+0000 7f0b73e00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b540a1e00 0x7f0b540a42c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c0099a0 tx=0x7f0b5c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:23.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:23.951+0000 7f0b71600640 1 -- 172.21.15.116:0/1709946434 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0b6409b540 con 0x7f0b74075f90 2024-03-04T07:40:24.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.149+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b74062270 con 0x7f0b74075f90 2024-03-04T07:40:24.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.151+0000 7f0b71600640 1 -- 172.21.15.116:0/1709946434 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f0b6409ae40 con 0x7f0b74075f90 2024-03-04T07:40:24.152 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:24.152 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:40:24.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.153+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b540a1e00 msgr2=0x7f0b540a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.153+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b540a1e00 0x7f0b540a42c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c0099a0 tx=0x7f0b5c009500 comp rx=0 tx=0).stop 2024-03-04T07:40:24.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.153+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74075f90 msgr2=0x7f0b7419dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.153+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74075f90 0x7f0b7419dea0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e9f0 tx=0x7f0b6400edf0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 shutdown_connections 2024-03-04T07:40:24.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b74077b70 0x7f0b741a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b540a1e00 0x7f0b540a42c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b740771d0 0x7f0b7419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 --2- 172.21.15.116:0/1709946434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b74075f90 0x7f0b7419dea0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 >> 172.21.15.116:0/1709946434 conn(0x7f0b740fed80 msgr2=0x7f0b74100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:24.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 shutdown_connections 2024-03-04T07:40:24.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.154+0000 7f0b7ba00640 1 -- 172.21.15.116:0/1709946434 wait complete. 2024-03-04T07:40:24.173 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:40:24.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:40:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 -- 172.21.15.116:0/806796637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4101f30 msgr2=0x7f66a4104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 --2- 172.21.15.116:0/806796637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4101f30 0x7f66a4104320 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6694009900 tx=0x7f669402bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 -- 172.21.15.116:0/806796637 shutdown_connections 2024-03-04T07:40:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 --2- 172.21.15.116:0/806796637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4107190 0x7f66a41095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 --2- 172.21.15.116:0/806796637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66a4104860 0x7f66a4106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 --2- 172.21.15.116:0/806796637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4101f30 0x7f66a4104320 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 -- 172.21.15.116:0/806796637 >> 172.21.15.116:0/806796637 conn(0x7f66a40fb940 msgr2=0x7f66a40fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 -- 172.21.15.116:0/806796637 shutdown_connections 2024-03-04T07:40:24.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.382+0000 7f66a3e00640 1 -- 172.21.15.116:0/806796637 wait complete. 2024-03-04T07:40:24.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 Processor -- start 2024-03-04T07:40:24.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 -- start start 2024-03-04T07:40:24.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66a4101f30 0x7f66a41018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4104860 0x7f66a40ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4107190 0x7f66a4100420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a410d190 con 0x7f66a4104860 2024-03-04T07:40:24.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a410d010 con 0x7f66a4107190 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.383+0000 7f66a3e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a410d310 con 0x7f66a4101f30 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a2000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4104860 0x7f66a40ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4107190 0x7f66a4100420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a2000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4104860 0x7f66a40ffee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49924/0 (socket says 172.21.15.116:49924) 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4107190 0x7f66a4100420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51200/0 (socket says 172.21.15.116:51200) 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a2000640 1 -- 172.21.15.116:0/2342150785 learned_addr learned my addr 172.21.15.116:0/2342150785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a2a00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66a4101f30 0x7f66a41018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 -- 172.21.15.116:0/2342150785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66a4101f30 msgr2=0x7f66a41018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66a4101f30 0x7f66a41018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 -- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4104860 msgr2=0x7f66a40ffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4104860 0x7f66a40ffee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 -- 172.21.15.116:0/2342150785 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a4100cb0 con 0x7f66a4107190 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.384+0000 7f66a3400640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4107190 0x7f66a4100420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f669c009fd0 tx=0x7f669c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.385+0000 7f669b400640 1 -- 172.21.15.116:0/2342150785 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c00ce20 con 0x7f66a4107190 2024-03-04T07:40:24.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.385+0000 7f669b400640 1 -- 172.21.15.116:0/2342150785 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f669c005020 con 0x7f66a4107190 2024-03-04T07:40:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.385+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a41a1900 con 0x7f66a4107190 2024-03-04T07:40:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.385+0000 7f669b400640 1 -- 172.21.15.116:0/2342150785 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c010720 con 0x7f66a4107190 2024-03-04T07:40:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.385+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a41a1e50 con 0x7f66a4107190 2024-03-04T07:40:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.386+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a40fd430 con 0x7f66a4107190 2024-03-04T07:40:24.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.388+0000 7f669b400640 1 -- 172.21.15.116:0/2342150785 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f669c005540 con 0x7f66a4107190 2024-03-04T07:40:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.389+0000 7f669b400640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f66880a1e00 0x7f66880a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.389+0000 7f669b400640 1 -- 172.21.15.116:0/2342150785 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f669c0ce3d0 con 0x7f66a4107190 2024-03-04T07:40:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.389+0000 7f66a2a00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f66880a1e00 0x7f66880a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.390+0000 7f66a2a00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f66880a1e00 0x7f66880a42c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f669402c240 tx=0x7f669402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:24.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.391+0000 7f669b400640 1 -- 172.21.15.116:0/2342150785 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f669c09b800 con 0x7f66a4107190 2024-03-04T07:40:24.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.589+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66a40621d0 con 0x7f66a4107190 2024-03-04T07:40:24.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.589+0000 7f669b400640 1 -- 172.21.15.116:0/2342150785 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f669c09b100 con 0x7f66a4107190 2024-03-04T07:40:24.590 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.592+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f66880a1e00 msgr2=0x7f66880a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.592+0000 7f66a3e00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f66880a1e00 0x7f66880a42c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f669402c240 tx=0x7f669402df90 comp rx=0 tx=0).stop 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.592+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4107190 msgr2=0x7f66a4100420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.592+0000 7f66a3e00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4107190 0x7f66a4100420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f669c009fd0 tx=0x7f669c00ef10 comp rx=0 tx=0).stop 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 shutdown_connections 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f66a4107190 0x7f66a4100420 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66a4104860 0x7f66a40ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f66880a1e00 0x7f66880a42c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 --2- 172.21.15.116:0/2342150785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66a4101f30 0x7f66a41018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 >> 172.21.15.116:0/2342150785 conn(0x7f66a40fb940 msgr2=0x7f66a4102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 shutdown_connections 2024-03-04T07:40:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.593+0000 7f66a3e00640 1 -- 172.21.15.116:0/2342150785 wait complete. 2024-03-04T07:40:24.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:40:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 -- 172.21.15.116:0/603082324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff448103810 msgr2=0x7ff448103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 --2- 172.21.15.116:0/603082324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff448103810 0x7ff448103c30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff438009ca0 tx=0x7ff43802c100 comp rx=0 tx=0).stop 2024-03-04T07:40:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 -- 172.21.15.116:0/603082324 shutdown_connections 2024-03-04T07:40:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 --2- 172.21.15.116:0/603082324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4481053c0 0x7ff44810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 --2- 172.21.15.116:0/603082324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff448104a00 0x7ff448104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 --2- 172.21.15.116:0/603082324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff448103810 0x7ff448103c30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 -- 172.21.15.116:0/603082324 >> 172.21.15.116:0/603082324 conn(0x7ff4480fed80 msgr2=0x7ff4481011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 -- 172.21.15.116:0/603082324 shutdown_connections 2024-03-04T07:40:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.817+0000 7ff44e600640 1 -- 172.21.15.116:0/603082324 wait complete. 2024-03-04T07:40:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 Processor -- start 2024-03-04T07:40:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 -- start start 2024-03-04T07:40:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff448103810 0x7ff448199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff448104a00 0x7ff44819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4481053c0 0x7ff4481a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44810f490 con 0x7ff448103810 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44810f310 con 0x7ff4481053c0 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff44e600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44810f610 con 0x7ff448104a00 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.818+0000 7ff447e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4481053c0 0x7ff4481a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4481053c0 0x7ff4481a10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51216/0 (socket says 172.21.15.116:51216) 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 -- 172.21.15.116:0/386753646 learned_addr learned my addr 172.21.15.116:0/386753646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447400640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff448103810 0x7ff448199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff446a00640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff448104a00 0x7ff44819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 -- 172.21.15.116:0/386753646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff448104a00 msgr2=0x7ff44819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff448104a00 0x7ff44819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 -- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff448103810 msgr2=0x7ff448199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff448103810 0x7ff448199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 -- 172.21.15.116:0/386753646 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4481a1750 con 0x7ff4481053c0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff447e00640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4481053c0 0x7ff4481a10e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff44000e970 tx=0x7ff44000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff43fe00640 1 -- 172.21.15.116:0/386753646 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000ccc0 con 0x7ff4481053c0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff43fe00640 1 -- 172.21.15.116:0/386753646 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff44000ce60 con 0x7ff4481053c0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4481a1a40 con 0x7ff4481053c0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.819+0000 7ff43fe00640 1 -- 172.21.15.116:0/386753646 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440010490 con 0x7ff4481053c0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.820+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481a1fe0 con 0x7ff4481053c0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.821+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44810ab80 con 0x7ff4481053c0 2024-03-04T07:40:24.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.822+0000 7ff43fe00640 1 -- 172.21.15.116:0/386753646 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff440005020 con 0x7ff4481053c0 2024-03-04T07:40:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.823+0000 7ff43fe00640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff42c0a1e00 0x7ff42c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.824+0000 7ff43fe00640 1 -- 172.21.15.116:0/386753646 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff4400ce170 con 0x7ff4481053c0 2024-03-04T07:40:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.824+0000 7ff447400640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff42c0a1e00 0x7ff42c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.825+0000 7ff447400640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff42c0a1e00 0x7ff42c0a42c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff43802c640 tx=0x7ff438009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:24.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:24.827+0000 7ff43fe00640 1 -- 172.21.15.116:0/386753646 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff44009b560 con 0x7ff4481053c0 2024-03-04T07:40:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:24 smithi116 ceph-mon[32150]: pgmap v390: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:24 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1709946434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:24 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2342150785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.020+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4480621d0 con 0x7ff4481053c0 2024-03-04T07:40:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.020+0000 7ff43fe00640 1 -- 172.21.15.116:0/386753646 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff44009ae60 con 0x7ff4481053c0 2024-03-04T07:40:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:24 smithi152 ceph-mon[37154]: pgmap v390: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:24 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1709946434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:24 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2342150785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:25.022 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff42c0a1e00 msgr2=0x7ff42c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff42c0a1e00 0x7ff42c0a42c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff43802c640 tx=0x7ff438009360 comp rx=0 tx=0).stop 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4481053c0 msgr2=0x7ff4481a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4481053c0 0x7ff4481a10e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff44000e970 tx=0x7ff44000ed70 comp rx=0 tx=0).stop 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 shutdown_connections 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff4481053c0 0x7ff4481a10e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff42c0a1e00 0x7ff42c0a42c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff448104a00 0x7ff44819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 --2- 172.21.15.116:0/386753646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff448103810 0x7ff448199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 >> 172.21.15.116:0/386753646 conn(0x7ff4480fed80 msgr2=0x7ff448107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 shutdown_connections 2024-03-04T07:40:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.023+0000 7ff44e600640 1 -- 172.21.15.116:0/386753646 wait complete. 2024-03-04T07:40:25.042 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:40:25.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:24 smithi133 ceph-mon[36505]: pgmap v390: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-04T07:40:25.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:24 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1709946434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:24 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2342150785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 -- 172.21.15.116:0/4124883629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa614103810 msgr2=0x7fa614103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 --2- 172.21.15.116:0/4124883629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa614103810 0x7fa614103c30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa604009860 tx=0x7fa60402bb70 comp rx=0 tx=0).stop 2024-03-04T07:40:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 -- 172.21.15.116:0/4124883629 shutdown_connections 2024-03-04T07:40:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 --2- 172.21.15.116:0/4124883629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6141053c0 0x7fa61410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 --2- 172.21.15.116:0/4124883629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614104a00 0x7fa614104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 --2- 172.21.15.116:0/4124883629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa614103810 0x7fa614103c30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 -- 172.21.15.116:0/4124883629 >> 172.21.15.116:0/4124883629 conn(0x7fa6140fed80 msgr2=0x7fa6141011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:25.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 -- 172.21.15.116:0/4124883629 shutdown_connections 2024-03-04T07:40:25.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.249+0000 7fa61b000640 1 -- 172.21.15.116:0/4124883629 wait complete. 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.250+0000 7fa61b000640 1 Processor -- start 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.250+0000 7fa61b000640 1 -- start start 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa61b000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614103810 0x7fa614199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa61b000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa614104a00 0x7fa61419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa61b000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6141053c0 0x7fa6141a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa61b000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61410f460 con 0x7fa614104a00 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa61b000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61410f2e0 con 0x7fa614103810 2024-03-04T07:40:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa61b000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61410f5e0 con 0x7fa6141053c0 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614103810 0x7fa614199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614103810 0x7fa614199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51222/0 (socket says 172.21.15.116:51222) 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 -- 172.21.15.116:0/3189317338 learned_addr learned my addr 172.21.15.116:0/3189317338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613400640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa614104a00 0x7fa61419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa619000640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6141053c0 0x7fa6141a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 -- 172.21.15.116:0/3189317338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6141053c0 msgr2=0x7fa6141a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6141053c0 0x7fa6141a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 -- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa614104a00 msgr2=0x7fa61419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa614104a00 0x7fa61419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.251+0000 7fa613e00640 1 -- 172.21.15.116:0/3189317338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6141a1750 con 0x7fa614103810 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.252+0000 7fa613e00640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614103810 0x7fa614199b30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6040349e0 tx=0x7fa604032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.252+0000 7fa610c00640 1 -- 172.21.15.116:0/3189317338 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa604041020 con 0x7fa614103810 2024-03-04T07:40:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.252+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6141a19e0 con 0x7fa614103810 2024-03-04T07:40:25.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.252+0000 7fa610c00640 1 -- 172.21.15.116:0/3189317338 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa60402c9f0 con 0x7fa614103810 2024-03-04T07:40:25.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.252+0000 7fa610c00640 1 -- 172.21.15.116:0/3189317338 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa604039040 con 0x7fa614103810 2024-03-04T07:40:25.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.252+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6141a1f20 con 0x7fa614103810 2024-03-04T07:40:25.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.253+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa614068810 con 0x7fa614103810 2024-03-04T07:40:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.255+0000 7fa610c00640 1 -- 172.21.15.116:0/3189317338 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa604044020 con 0x7fa614103810 2024-03-04T07:40:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.256+0000 7fa610c00640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa5f80a1e00 0x7fa5f80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.256+0000 7fa610c00640 1 -- 172.21.15.116:0/3189317338 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa6040ef570 con 0x7fa614103810 2024-03-04T07:40:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.257+0000 7fa613400640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa5f80a1e00 0x7fa5f80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.257+0000 7fa613400640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa5f80a1e00 0x7fa5f80a42c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa61419b040 tx=0x7fa600009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.261+0000 7fa610c00640 1 -- 172.21.15.116:0/3189317338 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa60404b020 con 0x7fa614103810 2024-03-04T07:40:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.457+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa614062290 con 0x7fa614103810 2024-03-04T07:40:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.458+0000 7fa610c00640 1 -- 172.21.15.116:0/3189317338 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa6040bc960 con 0x7fa614103810 2024-03-04T07:40:25.458 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:40:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.460+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa5f80a1e00 msgr2=0x7fa5f80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.460+0000 7fa61b000640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa5f80a1e00 0x7fa5f80a42c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa61419b040 tx=0x7fa600009260 comp rx=0 tx=0).stop 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.460+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614103810 msgr2=0x7fa614199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.460+0000 7fa61b000640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614103810 0x7fa614199b30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6040349e0 tx=0x7fa604032430 comp rx=0 tx=0).stop 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 shutdown_connections 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa5f80a1e00 0x7fa5f80a42c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6141053c0 0x7fa6141a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa614104a00 0x7fa61419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 --2- 172.21.15.116:0/3189317338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa614103810 0x7fa614199b30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 >> 172.21.15.116:0/3189317338 conn(0x7fa6140fed80 msgr2=0x7fa614107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 shutdown_connections 2024-03-04T07:40:25.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.461+0000 7fa61b000640 1 -- 172.21.15.116:0/3189317338 wait complete. 2024-03-04T07:40:25.481 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2750865748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc0107c10 msgr2=0x7f5dc0108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2750865748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc0107c10 0x7f5dc0108030 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5db0009900 tx=0x7f5db002bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2750865748 shutdown_connections 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2750865748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc01097c0 0x7f5dc0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2750865748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5dc0108e00 0x7f5dc0109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2750865748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc0107c10 0x7f5dc0108030 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2750865748 >> 172.21.15.116:0/2750865748 conn(0x7f5dc0076be0 msgr2=0x7f5dc0079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:25.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2750865748 shutdown_connections 2024-03-04T07:40:25.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.692+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2750865748 wait complete. 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 Processor -- start 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 -- start start 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5dc0107c10 0x7f5dc019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108e00 0x7f5dc019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc01097c0 0x7f5dc01a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc01138d0 con 0x7f5dc0108e00 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0113750 con 0x7f5dc01097c0 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.693+0000 7f5dc6e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0113a50 con 0x7f5dc0107c10 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5db7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108e00 0x7f5dc019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc01097c0 0x7f5dc01a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dbfe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5dc0107c10 0x7f5dc019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc01097c0 0x7f5dc01a54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51242/0 (socket says 172.21.15.116:51242) 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 -- 172.21.15.116:0/2281082705 learned_addr learned my addr 172.21.15.116:0/2281082705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 -- 172.21.15.116:0/2281082705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5dc0107c10 msgr2=0x7f5dc019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5dc0107c10 0x7f5dc019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 -- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108e00 msgr2=0x7f5dc019e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108e00 0x7f5dc019e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.694+0000 7f5dc4e00640 1 -- 172.21.15.116:0/2281082705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc01a5b50 con 0x7f5dc01097c0 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.695+0000 7f5dc4e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc01097c0 0x7f5dc01a54e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009fd0 tx=0x7f5db800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.695+0000 7f5dbd600640 1 -- 172.21.15.116:0/2281082705 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db8017070 con 0x7f5dc01097c0 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.695+0000 7f5dbd600640 1 -- 172.21.15.116:0/2281082705 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5db8003e50 con 0x7f5dc01097c0 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.695+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc01a5e40 con 0x7f5dc01097c0 2024-03-04T07:40:25.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.695+0000 7f5dbd600640 1 -- 172.21.15.116:0/2281082705 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db80106c0 con 0x7f5dc01097c0 2024-03-04T07:40:25.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.695+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc01a62b0 con 0x7f5dc01097c0 2024-03-04T07:40:25.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.696+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc010efb0 con 0x7f5dc01097c0 2024-03-04T07:40:25.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.698+0000 7f5dbd600640 1 -- 172.21.15.116:0/2281082705 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5db8003970 con 0x7f5dc01097c0 2024-03-04T07:40:25.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.699+0000 7f5dbd600640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5da40a1d80 0x7f5da40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:25.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.699+0000 7f5dbd600640 1 -- 172.21.15.116:0/2281082705 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5db80cdac0 con 0x7f5dc01097c0 2024-03-04T07:40:25.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.699+0000 7f5dbfe00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5da40a1d80 0x7f5da40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:25.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.700+0000 7f5dbfe00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5da40a1d80 0x7f5da40a4240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5db002c170 tx=0x7f5db002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.703+0000 7f5dbd600640 1 -- 172.21.15.116:0/2281082705 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5db809a860 con 0x7f5dc01097c0 2024-03-04T07:40:25.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:25 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/386753646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:25.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:25 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3189317338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:25.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.889+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dc00621d0 con 0x7f5dc01097c0 2024-03-04T07:40:25.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.892+0000 7f5dbd600640 1 -- 172.21.15.116:0/2281082705 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f5db809e900 con 0x7f5dc01097c0 2024-03-04T07:40:25.893 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:25.893 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:40:25.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5da40a1d80 msgr2=0x7f5da40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5da40a1d80 0x7f5da40a4240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5db002c170 tx=0x7f5db002df90 comp rx=0 tx=0).stop 2024-03-04T07:40:25.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc01097c0 msgr2=0x7f5dc01a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:25.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc01097c0 0x7f5dc01a54e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009fd0 tx=0x7f5db800b260 comp rx=0 tx=0).stop 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 shutdown_connections 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5dc01097c0 0x7f5dc01a54e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108e00 0x7f5dc019e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5da40a1d80 0x7f5da40a4240 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 --2- 172.21.15.116:0/2281082705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5dc0107c10 0x7f5dc019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 >> 172.21.15.116:0/2281082705 conn(0x7f5dc0076be0 msgr2=0x7f5dc010c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.894+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 shutdown_connections 2024-03-04T07:40:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:25.895+0000 7f5dc6e00640 1 -- 172.21.15.116:0/2281082705 wait complete. 2024-03-04T07:40:25.913 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:40:25.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:40:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:25 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/386753646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:26.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:25 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3189317338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:26.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:25 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/386753646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:26.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:25 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3189317338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 -- 172.21.15.116:0/2260080615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1049e0 msgr2=0x7f29cc104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 --2- 172.21.15.116:0/2260080615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1049e0 0x7f29cc104e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009900 tx=0x7f29b802bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 -- 172.21.15.116:0/2260080615 shutdown_connections 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 --2- 172.21.15.116:0/2260080615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1053a0 0x7f29cc10bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 --2- 172.21.15.116:0/2260080615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1049e0 0x7f29cc104e60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 --2- 172.21.15.116:0/2260080615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc1037f0 0x7f29cc103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 -- 172.21.15.116:0/2260080615 >> 172.21.15.116:0/2260080615 conn(0x7f29cc0fed80 msgr2=0x7f29cc1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 -- 172.21.15.116:0/2260080615 shutdown_connections 2024-03-04T07:40:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.129+0000 7f29d1a00640 1 -- 172.21.15.116:0/2260080615 wait complete. 2024-03-04T07:40:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 Processor -- start 2024-03-04T07:40:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 -- start start 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1037f0 0x7f29cc199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1049e0 0x7f29cc19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc1053a0 0x7f29cc1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc10f4c0 con 0x7f29cc1037f0 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc10f340 con 0x7f29cc1049e0 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.130+0000 7f29d1a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc10f640 con 0x7f29cc1053a0 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1049e0 0x7f29cc19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29caa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1037f0 0x7f29cc199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1049e0 0x7f29cc19a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56796/0 (socket says 172.21.15.116:56796) 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 -- 172.21.15.116:0/3806657875 learned_addr learned my addr 172.21.15.116:0/3806657875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29cb400640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc1053a0 0x7f29cc1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 -- 172.21.15.116:0/3806657875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc1053a0 msgr2=0x7f29cc1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc1053a0 0x7f29cc1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 -- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1037f0 msgr2=0x7f29cc199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1037f0 0x7f29cc199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 -- 172.21.15.116:0/3806657875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc1a1750 con 0x7f29cc1049e0 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.131+0000 7f29ca000640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1049e0 0x7f29cc19a070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.132+0000 7f29c3400640 1 -- 172.21.15.116:0/3806657875 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8043020 con 0x7f29cc1049e0 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.132+0000 7f29c3400640 1 -- 172.21.15.116:0/3806657875 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29b80054e0 con 0x7f29cc1049e0 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.132+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc1a19e0 con 0x7f29cc1049e0 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.132+0000 7f29c3400640 1 -- 172.21.15.116:0/3806657875 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8003c40 con 0x7f29cc1049e0 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.132+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc1a1f30 con 0x7f29cc1049e0 2024-03-04T07:40:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.133+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc10ab10 con 0x7f29cc1049e0 2024-03-04T07:40:26.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.135+0000 7f29c3400640 1 -- 172.21.15.116:0/3806657875 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f29b8046020 con 0x7f29cc1049e0 2024-03-04T07:40:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.136+0000 7f29c3400640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29b00a1d80 0x7f29b00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.137+0000 7f29c3400640 1 -- 172.21.15.116:0/3806657875 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f29b80efe20 con 0x7f29cc1049e0 2024-03-04T07:40:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.137+0000 7f29caa00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29b00a1d80 0x7f29b00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.137+0000 7f29caa00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29b00a1d80 0x7f29b00a4240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f29cc104840 tx=0x7f29bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:26.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.145+0000 7f29c3400640 1 -- 172.21.15.116:0/3806657875 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f29b804d020 con 0x7f29cc1049e0 2024-03-04T07:40:26.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.342+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29cc0621d0 con 0x7f29cc1049e0 2024-03-04T07:40:26.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.342+0000 7f29c3400640 1 -- 172.21.15.116:0/3806657875 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29b80bd210 con 0x7f29cc1049e0 2024-03-04T07:40:26.344 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:40:26.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.345+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29b00a1d80 msgr2=0x7f29b00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.345+0000 7f29d1a00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29b00a1d80 0x7f29b00a4240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f29cc104840 tx=0x7f29bc005f90 comp rx=0 tx=0).stop 2024-03-04T07:40:26.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.345+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1049e0 msgr2=0x7f29cc19a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.345+0000 7f29d1a00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1049e0 0x7f29cc19a070 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b802c650 comp rx=0 tx=0).stop 2024-03-04T07:40:26.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.345+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 shutdown_connections 2024-03-04T07:40:26.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.345+0000 7f29d1a00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29b00a1d80 0x7f29b00a4240 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.346+0000 7f29d1a00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29cc1053a0 0x7f29cc1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.346+0000 7f29d1a00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f29cc1049e0 0x7f29cc19a070 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.346+0000 7f29d1a00640 1 --2- 172.21.15.116:0/3806657875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc1037f0 0x7f29cc199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.346+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 >> 172.21.15.116:0/3806657875 conn(0x7f29cc0fed80 msgr2=0x7f29cc107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:26.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.346+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 shutdown_connections 2024-03-04T07:40:26.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.346+0000 7f29d1a00640 1 -- 172.21.15.116:0/3806657875 wait complete. 2024-03-04T07:40:26.364 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:40:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 -- 172.21.15.116:0/1029769610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1027d0 msgr2=0x7fab3c102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 --2- 172.21.15.116:0/1029769610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1027d0 0x7fab3c102c50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fab28009860 tx=0x7fab2802bb70 comp rx=0 tx=0).stop 2024-03-04T07:40:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 -- 172.21.15.116:0/1029769610 shutdown_connections 2024-03-04T07:40:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 --2- 172.21.15.116:0/1029769610 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c103190 0x7fab3c109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 --2- 172.21.15.116:0/1029769610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1027d0 0x7fab3c102c50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 --2- 172.21.15.116:0/1029769610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1015e0 0x7fab3c101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 -- 172.21.15.116:0/1029769610 >> 172.21.15.116:0/1029769610 conn(0x7fab3c0fcb70 msgr2=0x7fab3c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 -- 172.21.15.116:0/1029769610 shutdown_connections 2024-03-04T07:40:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.586+0000 7fab43800640 1 -- 172.21.15.116:0/1029769610 wait complete. 2024-03-04T07:40:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.587+0000 7fab43800640 1 Processor -- start 2024-03-04T07:40:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.587+0000 7fab43800640 1 -- start start 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab43800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1015e0 0x7fab3c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab43800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c1027d0 0x7fab3c0739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab43800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c103190 0x7fab3c073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab43800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c10d240 con 0x7fab3c103190 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab43800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c10d0c0 con 0x7fab3c1027d0 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab43800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c10d3c0 con 0x7fab3c1015e0 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab33e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c1027d0 0x7fab3c0739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab33e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c1027d0 0x7fab3c0739c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56824/0 (socket says 172.21.15.116:56824) 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab33e00640 1 -- 172.21.15.116:0/322671901 learned_addr learned my addr 172.21.15.116:0/322671901 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab40e00640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1015e0 0x7fab3c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.588+0000 7fab41800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c103190 0x7fab3c073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab33e00640 1 -- 172.21.15.116:0/322671901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1015e0 msgr2=0x7fab3c075380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab33e00640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1015e0 0x7fab3c075380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab33e00640 1 -- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c103190 msgr2=0x7fab3c073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab33e00640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c103190 0x7fab3c073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab33e00640 1 -- 172.21.15.116:0/322671901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c074790 con 0x7fab3c1027d0 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab41800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c103190 0x7fab3c073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab33e00640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c1027d0 0x7fab3c0739c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fab28005940 tx=0x7fab28032470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab3a000640 1 -- 172.21.15.116:0/322671901 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28041020 con 0x7fab3c1027d0 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab3a000640 1 -- 172.21.15.116:0/322671901 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab2802c9f0 con 0x7fab3c1027d0 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c1a19d0 con 0x7fab3c1027d0 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab3a000640 1 -- 172.21.15.116:0/322671901 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28039040 con 0x7fab3c1027d0 2024-03-04T07:40:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.589+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c1a1f20 con 0x7fab3c1027d0 2024-03-04T07:40:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.590+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c108950 con 0x7fab3c1027d0 2024-03-04T07:40:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.593+0000 7fab3a000640 1 -- 172.21.15.116:0/322671901 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fab28044020 con 0x7fab3c1027d0 2024-03-04T07:40:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.594+0000 7fab3a000640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab200a1d80 0x7fab200a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.594+0000 7fab3a000640 1 -- 172.21.15.116:0/322671901 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fab280ef810 con 0x7fab3c1027d0 2024-03-04T07:40:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.595+0000 7fab40e00640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab200a1d80 0x7fab200a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.595+0000 7fab40e00640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab200a1d80 0x7fab200a4240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab3c102630 tx=0x7fab2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:26.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.603+0000 7fab3a000640 1 -- 172.21.15.116:0/322671901 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fab2804b020 con 0x7fab3c1027d0 2024-03-04T07:40:26.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.801+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fab3c0621d0 con 0x7fab3c1027d0 2024-03-04T07:40:26.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.802+0000 7fab3a000640 1 -- 172.21.15.116:0/322671901 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fab280bcc00 con 0x7fab3c1027d0 2024-03-04T07:40:26.803 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:40:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab200a1d80 msgr2=0x7fab200a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab200a1d80 0x7fab200a4240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab3c102630 tx=0x7fab2c009260 comp rx=0 tx=0).stop 2024-03-04T07:40:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c1027d0 msgr2=0x7fab3c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c1027d0 0x7fab3c0739c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fab28005940 tx=0x7fab28032470 comp rx=0 tx=0).stop 2024-03-04T07:40:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 shutdown_connections 2024-03-04T07:40:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c103190 0x7fab3c073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab3c1027d0 0x7fab3c0739c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab200a1d80 0x7fab200a4240 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 --2- 172.21.15.116:0/322671901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab3c1015e0 0x7fab3c075380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 >> 172.21.15.116:0/322671901 conn(0x7fab3c0fcb70 msgr2=0x7fab3c105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 shutdown_connections 2024-03-04T07:40:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:26.805+0000 7fab43800640 1 -- 172.21.15.116:0/322671901 wait complete. 2024-03-04T07:40:26.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:40:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:26 smithi116 ceph-mon[32150]: pgmap v391: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-04T07:40:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:26 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2281082705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:26 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3806657875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:26 smithi152 ceph-mon[37154]: pgmap v391: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-04T07:40:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:26 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2281082705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:26 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3806657875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.047+0000 7f921e600640 1 -- 172.21.15.116:0/427445872 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181036f0 msgr2=0x7f9218103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.047+0000 7f921e600640 1 --2- 172.21.15.116:0/427445872 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181036f0 0x7f9218103b10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9208009900 tx=0x7f920802bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.047+0000 7f921e600640 1 -- 172.21.15.116:0/427445872 shutdown_connections 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.047+0000 7f921e600640 1 --2- 172.21.15.116:0/427445872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92181052a0 0x7f921810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.047+0000 7f921e600640 1 --2- 172.21.15.116:0/427445872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92181048e0 0x7f9218104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.047+0000 7f921e600640 1 --2- 172.21.15.116:0/427445872 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181036f0 0x7f9218103b10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.047+0000 7f921e600640 1 -- 172.21.15.116:0/427445872 >> 172.21.15.116:0/427445872 conn(0x7f92180fec80 msgr2=0x7f92181010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.048+0000 7f921e600640 1 -- 172.21.15.116:0/427445872 shutdown_connections 2024-03-04T07:40:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.048+0000 7f921e600640 1 -- 172.21.15.116:0/427445872 wait complete. 2024-03-04T07:40:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.048+0000 7f921e600640 1 Processor -- start 2024-03-04T07:40:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921e600640 1 -- start start 2024-03-04T07:40:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921e600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92181036f0 0x7f9218199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921e600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181048e0 0x7f9218199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921e600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92181052a0 0x7f92181a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921e600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921810f3b0 con 0x7f92181036f0 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921e600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921810f230 con 0x7f92181048e0 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921e600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921810f530 con 0x7f92181052a0 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f920fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181048e0 0x7f9218199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92181036f0 0x7f9218199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f920fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181048e0 0x7f9218199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56850/0 (socket says 172.21.15.116:56850) 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f920fe00640 1 -- 172.21.15.116:0/2314754608 learned_addr learned my addr 172.21.15.116:0/2314754608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.049+0000 7f921dc00640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92181052a0 0x7f92181a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f920fe00640 1 -- 172.21.15.116:0/2314754608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92181052a0 msgr2=0x7f92181a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f920fe00640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92181052a0 0x7f92181a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f920fe00640 1 -- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92181036f0 msgr2=0x7f9218199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f920fe00640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92181036f0 0x7f9218199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f920fe00640 1 -- 172.21.15.116:0/2314754608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92181a1640 con 0x7f92181048e0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f920fe00640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181048e0 0x7f9218199f60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f920000e9f0 tx=0x7f920000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f9216000640 1 -- 172.21.15.116:0/2314754608 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920000cd80 con 0x7f92181048e0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f9216000640 1 -- 172.21.15.116:0/2314754608 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9200005500 con 0x7f92181048e0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92181a1930 con 0x7f92181048e0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f9216000640 1 -- 172.21.15.116:0/2314754608 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200010690 con 0x7f92181048e0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.050+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92181a1e30 con 0x7f92181048e0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.051+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921810aa10 con 0x7f92181048e0 2024-03-04T07:40:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.053+0000 7f9216000640 1 -- 172.21.15.116:0/2314754608 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f92000046c0 con 0x7f92181048e0 2024-03-04T07:40:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.054+0000 7f9216000640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91fc0a1d80 0x7f91fc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.054+0000 7f9216000640 1 -- 172.21.15.116:0/2314754608 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f92000ce1e0 con 0x7f92181048e0 2024-03-04T07:40:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.055+0000 7f921d200640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91fc0a1d80 0x7f91fc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.055+0000 7f921d200640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91fc0a1d80 0x7f91fc0a4240 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f920802c170 tx=0x7f920802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.058+0000 7f9216000640 1 -- 172.21.15.116:0/2314754608 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f920009b5d0 con 0x7f92181048e0 2024-03-04T07:40:27.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:26 smithi133 ceph-mon[36505]: pgmap v391: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-03-04T07:40:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:26 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2281082705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:26 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3806657875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:27.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.263+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92180621d0 con 0x7f92181048e0 2024-03-04T07:40:27.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.264+0000 7f9216000640 1 -- 172.21.15.116:0/2314754608 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f920009aed0 con 0x7f92181048e0 2024-03-04T07:40:27.265 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91fc0a1d80 msgr2=0x7f91fc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91fc0a1d80 0x7f91fc0a4240 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f920802c170 tx=0x7f920802df90 comp rx=0 tx=0).stop 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181048e0 msgr2=0x7f9218199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181048e0 0x7f9218199f60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f920000e9f0 tx=0x7f920000edf0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 shutdown_connections 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f91fc0a1d80 0x7f91fc0a4240 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92181052a0 0x7f92181a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f92181048e0 0x7f9218199f60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 --2- 172.21.15.116:0/2314754608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92181036f0 0x7f9218199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 >> 172.21.15.116:0/2314754608 conn(0x7f92180fec80 msgr2=0x7f9218107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 shutdown_connections 2024-03-04T07:40:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.267+0000 7f921e600640 1 -- 172.21.15.116:0/2314754608 wait complete. 2024-03-04T07:40:27.287 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-04T07:40:27.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 -- 172.21.15.116:0/664476121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4107c10 msgr2=0x7f74c4108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 --2- 172.21.15.116:0/664476121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4107c10 0x7f74c4108030 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009900 tx=0x7f74b402bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 -- 172.21.15.116:0/664476121 shutdown_connections 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 --2- 172.21.15.116:0/664476121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74c41097c0 0x7f74c4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 --2- 172.21.15.116:0/664476121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74c4108e00 0x7f74c4109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 --2- 172.21.15.116:0/664476121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4107c10 0x7f74c4108030 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 -- 172.21.15.116:0/664476121 >> 172.21.15.116:0/664476121 conn(0x7f74c4076be0 msgr2=0x7f74c4079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 -- 172.21.15.116:0/664476121 shutdown_connections 2024-03-04T07:40:27.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.496+0000 7f74ca800640 1 -- 172.21.15.116:0/664476121 wait complete. 2024-03-04T07:40:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 Processor -- start 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 -- start start 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74c4107c10 0x7f74c419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4108e00 0x7f74c419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74c41097c0 0x7f74c41a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c41138d0 con 0x7f74c4107c10 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4113750 con 0x7f74c4108e00 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74ca800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4113a50 con 0x7f74c41097c0 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74baa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4108e00 0x7f74c419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74c3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74c4107c10 0x7f74c419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74baa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4108e00 0x7f74c419e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56870/0 (socket says 172.21.15.116:56870) 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.497+0000 7f74baa00640 1 -- 172.21.15.116:0/1548516700 learned_addr learned my addr 172.21.15.116:0/1548516700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74c3e00640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74c41097c0 0x7f74c41a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74baa00640 1 -- 172.21.15.116:0/1548516700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74c41097c0 msgr2=0x7f74c41a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74baa00640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74c41097c0 0x7f74c41a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74baa00640 1 -- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74c4107c10 msgr2=0x7f74c419df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74baa00640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74c4107c10 0x7f74c419df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74baa00640 1 -- 172.21.15.116:0/1548516700 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c41a5b10 con 0x7f74c4108e00 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74c3e00640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74c41097c0 0x7f74c41a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74baa00640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4108e00 0x7f74c419e480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e9f0 tx=0x7f74ac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74c0c00640 1 -- 172.21.15.116:0/1548516700 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac00cd80 con 0x7f74c4108e00 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74c0c00640 1 -- 172.21.15.116:0/1548516700 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74ac005500 con 0x7f74c4108e00 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74c0c00640 1 -- 172.21.15.116:0/1548516700 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac010690 con 0x7f74c4108e00 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c41a5e00 con 0x7f74c4108e00 2024-03-04T07:40:27.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.498+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c41a6270 con 0x7f74c4108e00 2024-03-04T07:40:27.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.499+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c410ef30 con 0x7f74c4108e00 2024-03-04T07:40:27.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.501+0000 7f74c0c00640 1 -- 172.21.15.116:0/1548516700 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f74ac0046c0 con 0x7f74c4108e00 2024-03-04T07:40:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.502+0000 7f74c0c00640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f74a80a1d80 0x7f74a80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.503+0000 7f74c0c00640 1 -- 172.21.15.116:0/1548516700 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f74ac0ce260 con 0x7f74c4108e00 2024-03-04T07:40:27.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.503+0000 7f74c3400640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f74a80a1d80 0x7f74a80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.503+0000 7f74c3400640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f74a80a1d80 0x7f74a80a4240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f74b402c170 tx=0x7f74b402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.506+0000 7f74c0c00640 1 -- 172.21.15.116:0/1548516700 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f74ac09b650 con 0x7f74c4108e00 2024-03-04T07:40:27.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.746+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c402cff0 con 0x7f74c4108e00 2024-03-04T07:40:27.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.747+0000 7f74c0c00640 1 -- 172.21.15.116:0/1548516700 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f74ac09af50 con 0x7f74c4108e00 2024-03-04T07:40:27.748 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:27.749 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:40:27.749 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:40:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f74a80a1d80 msgr2=0x7f74a80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f74a80a1d80 0x7f74a80a4240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f74b402c170 tx=0x7f74b402df90 comp rx=0 tx=0).stop 2024-03-04T07:40:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4108e00 msgr2=0x7f74c419e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4108e00 0x7f74c419e480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e9f0 tx=0x7f74ac00edf0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 shutdown_connections 2024-03-04T07:40:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f74a80a1d80 0x7f74a80a4240 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f74c41097c0 0x7f74c41a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74c4108e00 0x7f74c419e480 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 --2- 172.21.15.116:0/1548516700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74c4107c10 0x7f74c419df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 >> 172.21.15.116:0/1548516700 conn(0x7f74c4076be0 msgr2=0x7f74c410c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 shutdown_connections 2024-03-04T07:40:27.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.750+0000 7f74ca800640 1 -- 172.21.15.116:0/1548516700 wait complete. 2024-03-04T07:40:27.770 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-04T07:40:27.771 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-04T07:40:27.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T07:40:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 -- 172.21.15.116:0/497205711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120107c10 msgr2=0x7fd120108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 --2- 172.21.15.116:0/497205711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120107c10 0x7fd120108030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd110009ca0 tx=0x7fd11002c0f0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 -- 172.21.15.116:0/497205711 shutdown_connections 2024-03-04T07:40:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 --2- 172.21.15.116:0/497205711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1201097c0 0x7fd120110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 --2- 172.21.15.116:0/497205711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd120108e00 0x7fd120109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 --2- 172.21.15.116:0/497205711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120107c10 0x7fd120108030 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 -- 172.21.15.116:0/497205711 >> 172.21.15.116:0/497205711 conn(0x7fd120076be0 msgr2=0x7fd120079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 -- 172.21.15.116:0/497205711 shutdown_connections 2024-03-04T07:40:27.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.970+0000 7fd126400640 1 -- 172.21.15.116:0/497205711 wait complete. 2024-03-04T07:40:27.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.971+0000 7fd126400640 1 Processor -- start 2024-03-04T07:40:27.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.971+0000 7fd126400640 1 -- start start 2024-03-04T07:40:27.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd126400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd120107c10 0x7fd12019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd126400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120108e00 0x7fd12019e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd126400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1201097c0 0x7fd1201a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd126400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd120113860 con 0x7fd1201097c0 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd126400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201136e0 con 0x7fd120108e00 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd126400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201139e0 con 0x7fd120107c10 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120108e00 0x7fd12019e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120108e00 0x7fd12019e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56890/0 (socket says 172.21.15.116:56890) 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 -- 172.21.15.116:0/3606352970 learned_addr learned my addr 172.21.15.116:0/3606352970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11fe00640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1201097c0 0x7fd1201a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11f400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd120107c10 0x7fd12019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 -- 172.21.15.116:0/3606352970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd120107c10 msgr2=0x7fd12019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd120107c10 0x7fd12019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 -- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1201097c0 msgr2=0x7fd1201a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1201097c0 0x7fd1201a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11ea00640 1 -- 172.21.15.116:0/3606352970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1201a5b60 con 0x7fd120108e00 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.972+0000 7fd11fe00640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1201097c0 0x7fd1201a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.973+0000 7fd11ea00640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120108e00 0x7fd12019e4d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e9f0 tx=0x7fd10800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.973+0000 7fd117e00640 1 -- 172.21.15.116:0/3606352970 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10800cd80 con 0x7fd120108e00 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.973+0000 7fd117e00640 1 -- 172.21.15.116:0/3606352970 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd108010910 con 0x7fd120108e00 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.973+0000 7fd117e00640 1 -- 172.21.15.116:0/3606352970 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd108018990 con 0x7fd120108e00 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.973+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1201a5e50 con 0x7fd120108e00 2024-03-04T07:40:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.973+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1201a62c0 con 0x7fd120108e00 2024-03-04T07:40:27.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.974+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12010ef80 con 0x7fd120108e00 2024-03-04T07:40:27.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.976+0000 7fd117e00640 1 -- 172.21.15.116:0/3606352970 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd108010ac0 con 0x7fd120108e00 2024-03-04T07:40:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.977+0000 7fd117e00640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1040a1d80 0x7fd1040a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.977+0000 7fd117e00640 1 -- 172.21.15.116:0/3606352970 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd1080ce170 con 0x7fd120108e00 2024-03-04T07:40:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.977+0000 7fd11f400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1040a1d80 0x7fd1040a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:27.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.978+0000 7fd11f400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1040a1d80 0x7fd1040a4240 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd11002c700 tx=0x7fd110032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:27.985+0000 7fd117e00640 1 -- 172.21.15.116:0/3606352970 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd10809b560 con 0x7fd120108e00 2024-03-04T07:40:28.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:27 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/322671901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:28.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:27 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2314754608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:28.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:27 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/322671901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:28.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:27 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2314754608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:28.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:27 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/322671901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:28.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:27 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2314754608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:28.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.181+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd120061a10 con 0x7fd1040a1d80 2024-03-04T07:40:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.189+0000 7fd117e00640 1 -- 172.21.15.116:0/3606352970 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259026 (secure 0 0 0) 0x7fd120061a10 con 0x7fd1040a1d80 2024-03-04T07:40:28.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:28.196 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T07:40:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1040a1d80 msgr2=0x7fd1040a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1040a1d80 0x7fd1040a4240 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd11002c700 tx=0x7fd110032590 comp rx=0 tx=0).stop 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120108e00 msgr2=0x7fd12019e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120108e00 0x7fd12019e4d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e9f0 tx=0x7fd10800eeb0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 shutdown_connections 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1201097c0 0x7fd1201a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd120108e00 0x7fd12019e4d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1040a1d80 0x7fd1040a4240 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 --2- 172.21.15.116:0/3606352970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd120107c10 0x7fd12019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 >> 172.21.15.116:0/3606352970 conn(0x7fd120076be0 msgr2=0x7fd12010c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.195+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 shutdown_connections 2024-03-04T07:40:28.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.196+0000 7fd126400640 1 -- 172.21.15.116:0/3606352970 wait complete. 2024-03-04T07:40:28.215 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-03-04T07:40:27.534951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653005,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":440,"num_read_kb":328,"num_write":267,"num_write_kb":3029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":710,"ondisk_log_size":710,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360564,"kb_used_data":11412,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581260,"statfs":{"total":1151940427776,"available":1151571210240,"internally_reserved":0,"allocated":11685888,"data_stored":7427417,"data_compressed":182424,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":310,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004497"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509850+0000","last_change":"2024-03-04T07:37:55.137540+0000","last_active":"2024-03-04T07:39:27.509850+0000","last_peered":"2024-03-04T07:39:27.509850+0000","last_clean":"2024-03-04T07:39:27.509850+0000","last_became_active":"2024-03-04T07:37:55.137183+0000","last_became_peered":"2024-03-04T07:37:55.137183+0000","last_unstale":"2024-03-04T07:39:27.509850+0000","last_undegraded":"2024-03-04T07:39:27.509850+0000","last_fullsized":"2024-03-04T07:39:27.509850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:15.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503533+0000","last_change":"2024-03-04T07:37:55.136049+0000","last_active":"2024-03-04T07:39:27.503533+0000","last_peered":"2024-03-04T07:39:27.503533+0000","last_clean":"2024-03-04T07:39:27.503533+0000","last_became_active":"2024-03-04T07:37:55.132153+0000","last_became_peered":"2024-03-04T07:37:55.132153+0000","last_unstale":"2024-03-04T07:39:27.503533+0000","last_undegraded":"2024-03-04T07:39:27.503533+0000","last_fullsized":"2024-03-04T07:39:27.503533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:52.426892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.508509+0000","last_change":"2024-03-04T07:37:55.133571+0000","last_active":"2024-03-04T07:39:27.508509+0000","last_peered":"2024-03-04T07:39:27.508509+0000","last_clean":"2024-03-04T07:39:27.508509+0000","last_became_active":"2024-03-04T07:37:55.133214+0000","last_became_peered":"2024-03-04T07:37:55.133214+0000","last_unstale":"2024-03-04T07:39:27.508509+0000","last_undegraded":"2024-03-04T07:39:27.508509+0000","last_fullsized":"2024-03-04T07:39:27.508509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:17.924826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.480393+0000","last_change":"2024-03-04T07:37:55.118146+0000","last_active":"2024-03-04T07:39:27.480393+0000","last_peered":"2024-03-04T07:39:27.480393+0000","last_clean":"2024-03-04T07:39:27.480393+0000","last_became_active":"2024-03-04T07:37:55.117687+0000","last_became_peered":"2024-03-04T07:37:55.117687+0000","last_unstale":"2024-03-04T07:39:27.480393+0000","last_undegraded":"2024-03-04T07:39:27.480393+0000","last_fullsized":"2024-03-04T07:39:27.480393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:34:35.343400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.499265+0000","last_change":"2024-03-04T07:37:55.135943+0000","last_active":"2024-03-04T07:39:27.499265+0000","last_peered":"2024-03-04T07:39:27.499265+0000","last_clean":"2024-03-04T07:39:27.499265+0000","last_became_active":"2024-03-04T07:37:55.135549+0000","last_became_peered":"2024-03-04T07:37:55.135549+0000","last_unstale":"2024-03-04T07:39:27.499265+0000","last_undegraded":"2024-03-04T07:39:27.499265+0000","last_fullsized":"2024-03-04T07:39:27.499265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:35:53.953771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487475+0000","last_change":"2024-03-04T07:37:55.116949+0000","last_active":"2024-03-04T07:39:27.487475+0000","last_peered":"2024-03-04T07:39:27.487475+0000","last_clean":"2024-03-04T07:39:27.487475+0000","last_became_active":"2024-03-04T07:37:55.116620+0000","last_became_peered":"2024-03-04T07:37:55.116620+0000","last_unstale":"2024-03-04T07:39:27.487475+0000","last_undegraded":"2024-03-04T07:39:27.487475+0000","last_fullsized":"2024-03-04T07:39:27.487475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:39.655217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489760+0000","last_change":"2024-03-04T07:37:55.100975+0000","last_active":"2024-03-04T07:39:27.489760+0000","last_peered":"2024-03-04T07:39:27.489760+0000","last_clean":"2024-03-04T07:39:27.489760+0000","last_became_active":"2024-03-04T07:37:55.100636+0000","last_became_peered":"2024-03-04T07:37:55.100636+0000","last_unstale":"2024-03-04T07:39:27.489760+0000","last_undegraded":"2024-03-04T07:39:27.489760+0000","last_fullsized":"2024-03-04T07:39:27.489760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:29.593763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523017+0000","last_change":"2024-03-04T07:37:55.131895+0000","last_active":"2024-03-04T07:39:27.523017+0000","last_peered":"2024-03-04T07:39:27.523017+0000","last_clean":"2024-03-04T07:39:27.523017+0000","last_became_active":"2024-03-04T07:37:55.129682+0000","last_became_peered":"2024-03-04T07:37:55.129682+0000","last_unstale":"2024-03-04T07:39:27.523017+0000","last_undegraded":"2024-03-04T07:39:27.523017+0000","last_fullsized":"2024-03-04T07:39:27.523017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:54.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489551+0000","last_change":"2024-03-04T07:37:55.114475+0000","last_active":"2024-03-04T07:39:27.489551+0000","last_peered":"2024-03-04T07:39:27.489551+0000","last_clean":"2024-03-04T07:39:27.489551+0000","last_became_active":"2024-03-04T07:37:55.114068+0000","last_became_peered":"2024-03-04T07:37:55.114068+0000","last_unstale":"2024-03-04T07:39:27.489551+0000","last_undegraded":"2024-03-04T07:39:27.489551+0000","last_fullsized":"2024-03-04T07:39:27.489551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:21.466143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.477511+0000","last_change":"2024-03-04T07:37:55.115660+0000","last_active":"2024-03-04T07:39:27.477511+0000","last_peered":"2024-03-04T07:39:27.477511+0000","last_clean":"2024-03-04T07:39:27.477511+0000","last_became_active":"2024-03-04T07:37:55.115324+0000","last_became_peered":"2024-03-04T07:37:55.115324+0000","last_unstale":"2024-03-04T07:39:27.477511+0000","last_undegraded":"2024-03-04T07:39:27.477511+0000","last_fullsized":"2024-03-04T07:39:27.477511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:42.119728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489282+0000","last_change":"2024-03-04T07:37:55.119834+0000","last_active":"2024-03-04T07:39:27.489282+0000","last_peered":"2024-03-04T07:39:27.489282+0000","last_clean":"2024-03-04T07:39:27.489282+0000","last_became_active":"2024-03-04T07:37:55.119467+0000","last_became_peered":"2024-03-04T07:37:55.119467+0000","last_unstale":"2024-03-04T07:39:27.489282+0000","last_undegraded":"2024-03-04T07:39:27.489282+0000","last_fullsized":"2024-03-04T07:39:27.489282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:49:19.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.479644+0000","last_change":"2024-03-04T07:37:55.118154+0000","last_active":"2024-03-04T07:39:27.479644+0000","last_peered":"2024-03-04T07:39:27.479644+0000","last_clean":"2024-03-04T07:39:27.479644+0000","last_became_active":"2024-03-04T07:37:55.117809+0000","last_became_peered":"2024-03-04T07:37:55.117809+0000","last_unstale":"2024-03-04T07:39:27.479644+0000","last_undegraded":"2024-03-04T07:39:27.479644+0000","last_fullsized":"2024-03-04T07:39:27.479644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:50.195946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503528+0000","last_change":"2024-03-04T07:37:55.123325+0000","last_active":"2024-03-04T07:39:27.503528+0000","last_peered":"2024-03-04T07:39:27.503528+0000","last_clean":"2024-03-04T07:39:27.503528+0000","last_became_active":"2024-03-04T07:37:55.122848+0000","last_became_peered":"2024-03-04T07:37:55.122848+0000","last_unstale":"2024-03-04T07:39:27.503528+0000","last_undegraded":"2024-03-04T07:39:27.503528+0000","last_fullsized":"2024-03-04T07:39:27.503528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:18.842011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503051+0000","last_change":"2024-03-04T07:37:55.132447+0000","last_active":"2024-03-04T07:39:27.503051+0000","last_peered":"2024-03-04T07:39:27.503051+0000","last_clean":"2024-03-04T07:39:27.503051+0000","last_became_active":"2024-03-04T07:37:55.131512+0000","last_became_peered":"2024-03-04T07:37:55.131512+0000","last_unstale":"2024-03-04T07:39:27.503051+0000","last_undegraded":"2024-03-04T07:39:27.503051+0000","last_fullsized":"2024-03-04T07:39:27.503051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:23.960029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504478+0000","last_change":"2024-03-04T07:37:55.131715+0000","last_active":"2024-03-04T07:39:27.504478+0000","last_peered":"2024-03-04T07:39:27.504478+0000","last_clean":"2024-03-04T07:39:27.504478+0000","last_became_active":"2024-03-04T07:37:55.131326+0000","last_became_peered":"2024-03-04T07:37:55.131326+0000","last_unstale":"2024-03-04T07:39:27.504478+0000","last_undegraded":"2024-03-04T07:39:27.504478+0000","last_fullsized":"2024-03-04T07:39:27.504478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:56.231578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502604+0000","last_change":"2024-03-04T07:37:55.137395+0000","last_active":"2024-03-04T07:39:27.502604+0000","last_peered":"2024-03-04T07:39:27.502604+0000","last_clean":"2024-03-04T07:39:27.502604+0000","last_became_active":"2024-03-04T07:37:55.137054+0000","last_became_peered":"2024-03-04T07:37:55.137054+0000","last_unstale":"2024-03-04T07:39:27.502604+0000","last_undegraded":"2024-03-04T07:39:27.502604+0000","last_fullsized":"2024-03-04T07:39:27.502604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:19.222481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509061+0000","last_change":"2024-03-04T07:37:55.137414+0000","last_active":"2024-03-04T07:39:27.509061+0000","last_peered":"2024-03-04T07:39:27.509061+0000","last_clean":"2024-03-04T07:39:27.509061+0000","last_became_active":"2024-03-04T07:37:55.137157+0000","last_became_peered":"2024-03-04T07:37:55.137157+0000","last_unstale":"2024-03-04T07:39:27.509061+0000","last_undegraded":"2024-03-04T07:39:27.509061+0000","last_fullsized":"2024-03-04T07:39:27.509061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:36.593491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502012+0000","last_change":"2024-03-04T07:37:55.121960+0000","last_active":"2024-03-04T07:39:27.502012+0000","last_peered":"2024-03-04T07:39:27.502012+0000","last_clean":"2024-03-04T07:39:27.502012+0000","last_became_active":"2024-03-04T07:37:55.115543+0000","last_became_peered":"2024-03-04T07:37:55.115543+0000","last_unstale":"2024-03-04T07:39:27.502012+0000","last_undegraded":"2024-03-04T07:39:27.502012+0000","last_fullsized":"2024-03-04T07:39:27.502012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:36.320475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.506619+0000","last_change":"2024-03-04T07:37:55.128908+0000","last_active":"2024-03-04T07:39:27.506619+0000","last_peered":"2024-03-04T07:39:27.506619+0000","last_clean":"2024-03-04T07:39:27.506619+0000","last_became_active":"2024-03-04T07:37:55.128500+0000","last_became_peered":"2024-03-04T07:37:55.128500+0000","last_unstale":"2024-03-04T07:39:27.506619+0000","last_undegraded":"2024-03-04T07:39:27.506619+0000","last_fullsized":"2024-03-04T07:39:27.506619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:10.598424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805031+0000","last_change":"2024-03-04T07:37:54.121610+0000","last_active":"2024-03-04T07:38:58.805031+0000","last_peered":"2024-03-04T07:38:58.805031+0000","last_clean":"2024-03-04T07:38:58.805031+0000","last_became_active":"2024-03-04T07:37:54.121150+0000","last_became_peered":"2024-03-04T07:37:54.121150+0000","last_unstale":"2024-03-04T07:38:58.805031+0000","last_undegraded":"2024-03-04T07:38:58.805031+0000","last_fullsized":"2024-03-04T07:38:58.805031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:05:03.965626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.518629+0000","last_change":"2024-03-04T07:37:55.138904+0000","last_active":"2024-03-04T07:39:27.518629+0000","last_peered":"2024-03-04T07:39:27.518629+0000","last_clean":"2024-03-04T07:39:27.518629+0000","last_became_active":"2024-03-04T07:37:55.138376+0000","last_became_peered":"2024-03-04T07:37:55.138376+0000","last_unstale":"2024-03-04T07:39:27.518629+0000","last_undegraded":"2024-03-04T07:39:27.518629+0000","last_fullsized":"2024-03-04T07:39:27.518629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:28.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807860+0000","last_change":"2024-03-04T07:37:54.139218+0000","last_active":"2024-03-04T07:38:58.807860+0000","last_peered":"2024-03-04T07:38:58.807860+0000","last_clean":"2024-03-04T07:38:58.807860+0000","last_became_active":"2024-03-04T07:37:54.138893+0000","last_became_peered":"2024-03-04T07:37:54.138893+0000","last_unstale":"2024-03-04T07:38:58.807860+0000","last_undegraded":"2024-03-04T07:38:58.807860+0000","last_fullsized":"2024-03-04T07:38:58.807860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:01.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511166+0000","last_change":"2024-03-04T07:37:55.135953+0000","last_active":"2024-03-04T07:39:27.511166+0000","last_peered":"2024-03-04T07:39:27.511166+0000","last_clean":"2024-03-04T07:39:27.511166+0000","last_became_active":"2024-03-04T07:37:55.132880+0000","last_became_peered":"2024-03-04T07:37:55.132880+0000","last_unstale":"2024-03-04T07:39:27.511166+0000","last_undegraded":"2024-03-04T07:39:27.511166+0000","last_fullsized":"2024-03-04T07:39:27.511166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:03.848428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808631+0000","last_change":"2024-03-04T07:37:54.122493+0000","last_active":"2024-03-04T07:38:58.808631+0000","last_peered":"2024-03-04T07:38:58.808631+0000","last_clean":"2024-03-04T07:38:58.808631+0000","last_became_active":"2024-03-04T07:37:54.122058+0000","last_became_peered":"2024-03-04T07:37:54.122058+0000","last_unstale":"2024-03-04T07:38:58.808631+0000","last_undegraded":"2024-03-04T07:38:58.808631+0000","last_fullsized":"2024-03-04T07:38:58.808631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:02.516175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.491455+0000","last_change":"2024-03-04T07:37:55.136155+0000","last_active":"2024-03-04T07:39:27.491455+0000","last_peered":"2024-03-04T07:39:27.491455+0000","last_clean":"2024-03-04T07:39:27.491455+0000","last_became_active":"2024-03-04T07:37:55.135796+0000","last_became_peered":"2024-03-04T07:37:55.135796+0000","last_unstale":"2024-03-04T07:39:27.491455+0000","last_undegraded":"2024-03-04T07:39:27.491455+0000","last_fullsized":"2024-03-04T07:39:27.491455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:37.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808949+0000","last_change":"2024-03-04T07:37:54.118780+0000","last_active":"2024-03-04T07:38:58.808949+0000","last_peered":"2024-03-04T07:38:58.808949+0000","last_clean":"2024-03-04T07:38:58.808949+0000","last_became_active":"2024-03-04T07:37:54.118448+0000","last_became_peered":"2024-03-04T07:37:54.118448+0000","last_unstale":"2024-03-04T07:38:58.808949+0000","last_undegraded":"2024-03-04T07:38:58.808949+0000","last_fullsized":"2024-03-04T07:38:58.808949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:25.547046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501807+0000","last_change":"2024-03-04T07:38:59.021729+0000","last_active":"2024-03-04T07:39:27.501807+0000","last_peered":"2024-03-04T07:39:27.501807+0000","last_clean":"2024-03-04T07:39:27.501807+0000","last_became_active":"2024-03-04T07:38:59.021127+0000","last_became_peered":"2024-03-04T07:38:59.021127+0000","last_unstale":"2024-03-04T07:39:27.501807+0000","last_undegraded":"2024-03-04T07:39:27.501807+0000","last_fullsized":"2024-03-04T07:39:27.501807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:08:07.825530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807893+0000","last_change":"2024-03-04T07:37:54.122241+0000","last_active":"2024-03-04T07:38:58.807893+0000","last_peered":"2024-03-04T07:38:58.807893+0000","last_clean":"2024-03-04T07:38:58.807893+0000","last_became_active":"2024-03-04T07:37:54.120990+0000","last_became_peered":"2024-03-04T07:37:54.120990+0000","last_unstale":"2024-03-04T07:38:58.807893+0000","last_undegraded":"2024-03-04T07:38:58.807893+0000","last_fullsized":"2024-03-04T07:38:58.807893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:03.233754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501726+0000","last_change":"2024-03-04T07:37:55.121692+0000","last_active":"2024-03-04T07:39:27.501726+0000","last_peered":"2024-03-04T07:39:27.501726+0000","last_clean":"2024-03-04T07:39:27.501726+0000","last_became_active":"2024-03-04T07:37:55.121345+0000","last_became_peered":"2024-03-04T07:37:55.121345+0000","last_unstale":"2024-03-04T07:39:27.501726+0000","last_undegraded":"2024-03-04T07:39:27.501726+0000","last_fullsized":"2024-03-04T07:39:27.501726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:52:42.045265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807447+0000","last_change":"2024-03-04T07:37:54.133489+0000","last_active":"2024-03-04T07:38:58.807447+0000","last_peered":"2024-03-04T07:38:58.807447+0000","last_clean":"2024-03-04T07:38:58.807447+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807447+0000","last_undegraded":"2024-03-04T07:38:58.807447+0000","last_fullsized":"2024-03-04T07:38:58.807447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:49:03.600907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.510134+0000","last_change":"2024-03-04T07:37:55.128278+0000","last_active":"2024-03-04T07:39:27.510134+0000","last_peered":"2024-03-04T07:39:27.510134+0000","last_clean":"2024-03-04T07:39:27.510134+0000","last_became_active":"2024-03-04T07:37:55.127913+0000","last_became_peered":"2024-03-04T07:37:55.127913+0000","last_unstale":"2024-03-04T07:39:27.510134+0000","last_undegraded":"2024-03-04T07:39:27.510134+0000","last_fullsized":"2024-03-04T07:39:27.510134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:54:18.623855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803866+0000","last_change":"2024-03-04T07:37:54.132195+0000","last_active":"2024-03-04T07:38:58.803866+0000","last_peered":"2024-03-04T07:38:58.803866+0000","last_clean":"2024-03-04T07:38:58.803866+0000","last_became_active":"2024-03-04T07:37:54.131830+0000","last_became_peered":"2024-03-04T07:37:54.131830+0000","last_unstale":"2024-03-04T07:38:58.803866+0000","last_undegraded":"2024-03-04T07:38:58.803866+0000","last_fullsized":"2024-03-04T07:38:58.803866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.744342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522878+0000","last_change":"2024-03-04T07:37:55.138281+0000","last_active":"2024-03-04T07:39:27.522878+0000","last_peered":"2024-03-04T07:39:27.522878+0000","last_clean":"2024-03-04T07:39:27.522878+0000","last_became_active":"2024-03-04T07:37:55.137919+0000","last_became_peered":"2024-03-04T07:37:55.137919+0000","last_unstale":"2024-03-04T07:39:27.522878+0000","last_undegraded":"2024-03-04T07:39:27.522878+0000","last_fullsized":"2024-03-04T07:39:27.522878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:53:06.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807612+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:38:58.807612+0000","last_peered":"2024-03-04T07:38:58.807612+0000","last_clean":"2024-03-04T07:38:58.807612+0000","last_became_active":"2024-03-04T07:37:54.121503+0000","last_became_peered":"2024-03-04T07:37:54.121503+0000","last_unstale":"2024-03-04T07:38:58.807612+0000","last_undegraded":"2024-03-04T07:38:58.807612+0000","last_fullsized":"2024-03-04T07:38:58.807612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:03:07.531246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519901+0000","last_change":"2024-03-04T07:37:55.126924+0000","last_active":"2024-03-04T07:39:27.519901+0000","last_peered":"2024-03-04T07:39:27.519901+0000","last_clean":"2024-03-04T07:39:27.519901+0000","last_became_active":"2024-03-04T07:37:55.126266+0000","last_became_peered":"2024-03-04T07:37:55.126266+0000","last_unstale":"2024-03-04T07:39:27.519901+0000","last_undegraded":"2024-03-04T07:39:27.519901+0000","last_fullsized":"2024-03-04T07:39:27.519901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:01.603430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807917+0000","last_change":"2024-03-04T07:37:54.119864+0000","last_active":"2024-03-04T07:38:58.807917+0000","last_peered":"2024-03-04T07:38:58.807917+0000","last_clean":"2024-03-04T07:38:58.807917+0000","last_became_active":"2024-03-04T07:37:54.119515+0000","last_became_peered":"2024-03-04T07:37:54.119515+0000","last_unstale":"2024-03-04T07:38:58.807917+0000","last_undegraded":"2024-03-04T07:38:58.807917+0000","last_fullsized":"2024-03-04T07:38:58.807917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:10.422458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.490884+0000","last_change":"2024-03-04T07:37:55.120135+0000","last_active":"2024-03-04T07:39:27.490884+0000","last_peered":"2024-03-04T07:39:27.490884+0000","last_clean":"2024-03-04T07:39:27.490884+0000","last_became_active":"2024-03-04T07:37:55.119670+0000","last_became_peered":"2024-03-04T07:37:55.119670+0000","last_unstale":"2024-03-04T07:39:27.490884+0000","last_undegraded":"2024-03-04T07:39:27.490884+0000","last_fullsized":"2024-03-04T07:39:27.490884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:09.688447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807249+0000","last_change":"2024-03-04T07:37:54.120222+0000","last_active":"2024-03-04T07:38:58.807249+0000","last_peered":"2024-03-04T07:38:58.807249+0000","last_clean":"2024-03-04T07:38:58.807249+0000","last_became_active":"2024-03-04T07:37:54.119802+0000","last_became_peered":"2024-03-04T07:37:54.119802+0000","last_unstale":"2024-03-04T07:38:58.807249+0000","last_undegraded":"2024-03-04T07:38:58.807249+0000","last_fullsized":"2024-03-04T07:38:58.807249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:12:40.811870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509588+0000","last_change":"2024-03-04T07:37:55.136371+0000","last_active":"2024-03-04T07:39:27.509588+0000","last_peered":"2024-03-04T07:39:27.509588+0000","last_clean":"2024-03-04T07:39:27.509588+0000","last_became_active":"2024-03-04T07:37:55.132587+0000","last_became_peered":"2024-03-04T07:37:55.132587+0000","last_unstale":"2024-03-04T07:39:27.509588+0000","last_undegraded":"2024-03-04T07:39:27.509588+0000","last_fullsized":"2024-03-04T07:39:27.509588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:24.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.809061+0000","last_change":"2024-03-04T07:37:54.123041+0000","last_active":"2024-03-04T07:38:58.809061+0000","last_peered":"2024-03-04T07:38:58.809061+0000","last_clean":"2024-03-04T07:38:58.809061+0000","last_became_active":"2024-03-04T07:37:54.122723+0000","last_became_peered":"2024-03-04T07:37:54.122723+0000","last_unstale":"2024-03-04T07:38:58.809061+0000","last_undegraded":"2024-03-04T07:38:58.809061+0000","last_fullsized":"2024-03-04T07:38:58.809061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:06:30.845376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488470+0000","last_change":"2024-03-04T07:37:55.115387+0000","last_active":"2024-03-04T07:39:27.488470+0000","last_peered":"2024-03-04T07:39:27.488470+0000","last_clean":"2024-03-04T07:39:27.488470+0000","last_became_active":"2024-03-04T07:37:55.114947+0000","last_became_peered":"2024-03-04T07:37:55.114947+0000","last_unstale":"2024-03-04T07:39:27.488470+0000","last_undegraded":"2024-03-04T07:39:27.488470+0000","last_fullsized":"2024-03-04T07:39:27.488470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:09:11.840989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":361,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:40:27.069018+0000","last_change":"2024-03-04T07:37:54.117767+0000","last_active":"2024-03-04T07:40:27.069018+0000","last_peered":"2024-03-04T07:40:27.069018+0000","last_clean":"2024-03-04T07:40:27.069018+0000","last_became_active":"2024-03-04T07:37:54.117371+0000","last_became_peered":"2024-03-04T07:37:54.117371+0000","last_unstale":"2024-03-04T07:40:27.069018+0000","last_undegraded":"2024-03-04T07:40:27.069018+0000","last_fullsized":"2024-03-04T07:40:27.069018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:08:32.164636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61605,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":310,"num_read_kb":215,"num_write":31,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488421+0000","last_change":"2024-03-04T07:37:55.112707+0000","last_active":"2024-03-04T07:39:27.488421+0000","last_peered":"2024-03-04T07:39:27.488421+0000","last_clean":"2024-03-04T07:39:27.488421+0000","last_became_active":"2024-03-04T07:37:55.112327+0000","last_became_peered":"2024-03-04T07:37:55.112327+0000","last_unstale":"2024-03-04T07:39:27.488421+0000","last_undegraded":"2024-03-04T07:39:27.488421+0000","last_fullsized":"2024-03-04T07:39:27.488421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:03.901687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811773+0000","last_change":"2024-03-04T07:37:54.125286+0000","last_active":"2024-03-04T07:38:58.811773+0000","last_peered":"2024-03-04T07:38:58.811773+0000","last_clean":"2024-03-04T07:38:58.811773+0000","last_became_active":"2024-03-04T07:37:54.124966+0000","last_became_peered":"2024-03-04T07:37:54.124966+0000","last_unstale":"2024-03-04T07:38:58.811773+0000","last_undegraded":"2024-03-04T07:38:58.811773+0000","last_fullsized":"2024-03-04T07:38:58.811773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:19.880681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.524159+0000","last_change":"2024-03-04T07:37:55.137952+0000","last_active":"2024-03-04T07:39:27.524159+0000","last_peered":"2024-03-04T07:39:27.524159+0000","last_clean":"2024-03-04T07:39:27.524159+0000","last_became_active":"2024-03-04T07:37:55.137583+0000","last_became_peered":"2024-03-04T07:37:55.137583+0000","last_unstale":"2024-03-04T07:39:27.524159+0000","last_undegraded":"2024-03-04T07:39:27.524159+0000","last_fullsized":"2024-03-04T07:39:27.524159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:30.145453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807594+0000","last_change":"2024-03-04T07:37:54.134733+0000","last_active":"2024-03-04T07:38:58.807594+0000","last_peered":"2024-03-04T07:38:58.807594+0000","last_clean":"2024-03-04T07:38:58.807594+0000","last_became_active":"2024-03-04T07:37:54.134400+0000","last_became_peered":"2024-03-04T07:37:54.134400+0000","last_unstale":"2024-03-04T07:38:58.807594+0000","last_undegraded":"2024-03-04T07:38:58.807594+0000","last_fullsized":"2024-03-04T07:38:58.807594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:24.192479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511483+0000","last_change":"2024-03-04T07:37:55.124991+0000","last_active":"2024-03-04T07:39:27.511483+0000","last_peered":"2024-03-04T07:39:27.511483+0000","last_clean":"2024-03-04T07:39:27.511483+0000","last_became_active":"2024-03-04T07:37:55.124621+0000","last_became_peered":"2024-03-04T07:37:55.124621+0000","last_unstale":"2024-03-04T07:39:27.511483+0000","last_undegraded":"2024-03-04T07:39:27.511483+0000","last_fullsized":"2024-03-04T07:39:27.511483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:05.252145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021001+0000","last_change":"2024-03-04T07:37:54.118743+0000","last_active":"2024-03-04T07:38:59.021001+0000","last_peered":"2024-03-04T07:38:59.021001+0000","last_clean":"2024-03-04T07:38:59.021001+0000","last_became_active":"2024-03-04T07:37:54.118386+0000","last_became_peered":"2024-03-04T07:37:54.118386+0000","last_unstale":"2024-03-04T07:38:59.021001+0000","last_undegraded":"2024-03-04T07:38:59.021001+0000","last_fullsized":"2024-03-04T07:38:59.021001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:27.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.473563+0000","last_change":"2024-03-04T07:37:55.115431+0000","last_active":"2024-03-04T07:39:27.473563+0000","last_peered":"2024-03-04T07:39:27.473563+0000","last_clean":"2024-03-04T07:39:27.473563+0000","last_became_active":"2024-03-04T07:37:55.115064+0000","last_became_peered":"2024-03-04T07:37:55.115064+0000","last_unstale":"2024-03-04T07:39:27.473563+0000","last_undegraded":"2024-03-04T07:39:27.473563+0000","last_fullsized":"2024-03-04T07:39:27.473563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:46:57.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.029383+0000","last_change":"2024-03-04T07:38:59.028943+0000","last_active":"2024-03-04T07:38:59.029383+0000","last_peered":"2024-03-04T07:38:59.029383+0000","last_clean":"2024-03-04T07:38:59.029383+0000","last_became_active":"2024-03-04T07:38:59.022232+0000","last_became_peered":"2024-03-04T07:38:59.022232+0000","last_unstale":"2024-03-04T07:38:59.029383+0000","last_undegraded":"2024-03-04T07:38:59.029383+0000","last_fullsized":"2024-03-04T07:38:59.029383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:49.281601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.485427+0000","last_change":"2024-03-04T07:37:55.114882+0000","last_active":"2024-03-04T07:39:27.485427+0000","last_peered":"2024-03-04T07:39:27.485427+0000","last_clean":"2024-03-04T07:39:27.485427+0000","last_became_active":"2024-03-04T07:37:55.113139+0000","last_became_peered":"2024-03-04T07:37:55.113139+0000","last_unstale":"2024-03-04T07:39:27.485427+0000","last_undegraded":"2024-03-04T07:39:27.485427+0000","last_fullsized":"2024-03-04T07:39:27.485427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:28.133710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808775+0000","last_change":"2024-03-04T07:37:54.119350+0000","last_active":"2024-03-04T07:38:58.808775+0000","last_peered":"2024-03-04T07:38:58.808775+0000","last_clean":"2024-03-04T07:38:58.808775+0000","last_became_active":"2024-03-04T07:37:54.119017+0000","last_became_peered":"2024-03-04T07:37:54.119017+0000","last_unstale":"2024-03-04T07:38:58.808775+0000","last_undegraded":"2024-03-04T07:38:58.808775+0000","last_fullsized":"2024-03-04T07:38:58.808775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:00.503583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.490355+0000","last_change":"2024-03-04T07:38:58.812598+0000","last_active":"2024-03-04T07:39:27.490355+0000","last_peered":"2024-03-04T07:39:27.490355+0000","last_clean":"2024-03-04T07:39:27.490355+0000","last_became_active":"2024-03-04T07:38:58.812187+0000","last_became_peered":"2024-03-04T07:38:58.812187+0000","last_unstale":"2024-03-04T07:39:27.490355+0000","last_undegraded":"2024-03-04T07:39:27.490355+0000","last_fullsized":"2024-03-04T07:39:27.490355+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:43.753007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804447+0000","last_change":"2024-03-04T07:37:54.125548+0000","last_active":"2024-03-04T07:38:58.804447+0000","last_peered":"2024-03-04T07:38:58.804447+0000","last_clean":"2024-03-04T07:38:58.804447+0000","last_became_active":"2024-03-04T07:37:54.125127+0000","last_became_peered":"2024-03-04T07:37:54.125127+0000","last_unstale":"2024-03-04T07:38:58.804447+0000","last_undegraded":"2024-03-04T07:38:58.804447+0000","last_fullsized":"2024-03-04T07:38:58.804447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:26.256247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.506689+0000","last_change":"2024-03-04T07:37:55.128542+0000","last_active":"2024-03-04T07:39:27.506689+0000","last_peered":"2024-03-04T07:39:27.506689+0000","last_clean":"2024-03-04T07:39:27.506689+0000","last_became_active":"2024-03-04T07:37:55.128239+0000","last_became_peered":"2024-03-04T07:37:55.128239+0000","last_unstale":"2024-03-04T07:39:27.506689+0000","last_undegraded":"2024-03-04T07:39:27.506689+0000","last_fullsized":"2024-03-04T07:39:27.506689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:17.078095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807386+0000","last_change":"2024-03-04T07:37:54.133672+0000","last_active":"2024-03-04T07:38:58.807386+0000","last_peered":"2024-03-04T07:38:58.807386+0000","last_clean":"2024-03-04T07:38:58.807386+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807386+0000","last_undegraded":"2024-03-04T07:38:58.807386+0000","last_fullsized":"2024-03-04T07:38:58.807386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:52.345158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501983+0000","last_change":"2024-03-04T07:38:58.815766+0000","last_active":"2024-03-04T07:39:27.501983+0000","last_peered":"2024-03-04T07:39:27.501983+0000","last_clean":"2024-03-04T07:39:27.501983+0000","last_became_active":"2024-03-04T07:38:58.815387+0000","last_became_peered":"2024-03-04T07:38:58.815387+0000","last_unstale":"2024-03-04T07:39:27.501983+0000","last_undegraded":"2024-03-04T07:39:27.501983+0000","last_fullsized":"2024-03-04T07:39:27.501983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:21.582879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488110+0000","last_change":"2024-03-04T07:38:58.814581+0000","last_active":"2024-03-04T07:39:27.488110+0000","last_peered":"2024-03-04T07:39:27.488110+0000","last_clean":"2024-03-04T07:39:27.488110+0000","last_became_active":"2024-03-04T07:38:58.814186+0000","last_became_peered":"2024-03-04T07:38:58.814186+0000","last_unstale":"2024-03-04T07:39:27.488110+0000","last_undegraded":"2024-03-04T07:39:27.488110+0000","last_fullsized":"2024-03-04T07:39:27.488110+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:03:33.824292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803774+0000","last_change":"2024-03-04T07:37:54.139335+0000","last_active":"2024-03-04T07:38:58.803774+0000","last_peered":"2024-03-04T07:38:58.803774+0000","last_clean":"2024-03-04T07:38:58.803774+0000","last_became_active":"2024-03-04T07:37:54.138969+0000","last_became_peered":"2024-03-04T07:37:54.138969+0000","last_unstale":"2024-03-04T07:38:58.803774+0000","last_undegraded":"2024-03-04T07:38:58.803774+0000","last_fullsized":"2024-03-04T07:38:58.803774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:41:15.204738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523815+0000","last_change":"2024-03-04T07:38:58.810554+0000","last_active":"2024-03-04T07:39:27.523815+0000","last_peered":"2024-03-04T07:39:27.523815+0000","last_clean":"2024-03-04T07:39:27.523815+0000","last_became_active":"2024-03-04T07:38:58.809978+0000","last_became_peered":"2024-03-04T07:38:58.809978+0000","last_unstale":"2024-03-04T07:39:27.523815+0000","last_undegraded":"2024-03-04T07:39:27.523815+0000","last_fullsized":"2024-03-04T07:39:27.523815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:33.417630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021011+0000","last_change":"2024-03-04T07:37:54.119906+0000","last_active":"2024-03-04T07:38:59.021011+0000","last_peered":"2024-03-04T07:38:59.021011+0000","last_clean":"2024-03-04T07:38:59.021011+0000","last_became_active":"2024-03-04T07:37:54.119592+0000","last_became_peered":"2024-03-04T07:37:54.119592+0000","last_unstale":"2024-03-04T07:38:59.021011+0000","last_undegraded":"2024-03-04T07:38:59.021011+0000","last_fullsized":"2024-03-04T07:38:59.021011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:29.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":343,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807128+0000","last_change":"2024-03-04T07:36:35.278528+0000","last_active":"2024-03-04T07:38:58.807128+0000","last_peered":"2024-03-04T07:38:58.807128+0000","last_clean":"2024-03-04T07:38:58.807128+0000","last_became_active":"2024-03-04T07:36:35.277953+0000","last_became_peered":"2024-03-04T07:36:35.277953+0000","last_unstale":"2024-03-04T07:38:58.807128+0000","last_undegraded":"2024-03-04T07:38:58.807128+0000","last_fullsized":"2024-03-04T07:38:58.807128+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_clean_scrub_stamp":"2024-03-04T07:32:42.283525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:58:47.223527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.496346+0000","last_change":"2024-03-04T07:37:55.124204+0000","last_active":"2024-03-04T07:39:27.496346+0000","last_peered":"2024-03-04T07:39:27.496346+0000","last_clean":"2024-03-04T07:39:27.496346+0000","last_became_active":"2024-03-04T07:37:55.123863+0000","last_became_peered":"2024-03-04T07:37:55.123863+0000","last_unstale":"2024-03-04T07:39:27.496346+0000","last_undegraded":"2024-03-04T07:39:27.496346+0000","last_fullsized":"2024-03-04T07:39:27.496346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:33.830367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807256+0000","last_change":"2024-03-04T07:37:54.121538+0000","last_active":"2024-03-04T07:38:58.807256+0000","last_peered":"2024-03-04T07:38:58.807256+0000","last_clean":"2024-03-04T07:38:58.807256+0000","last_became_active":"2024-03-04T07:37:54.120939+0000","last_became_peered":"2024-03-04T07:37:54.120939+0000","last_unstale":"2024-03-04T07:38:58.807256+0000","last_undegraded":"2024-03-04T07:38:58.807256+0000","last_fullsized":"2024-03-04T07:38:58.807256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:29.597247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808527+0000","last_change":"2024-03-04T07:37:54.128180+0000","last_active":"2024-03-04T07:38:58.808527+0000","last_peered":"2024-03-04T07:38:58.808527+0000","last_clean":"2024-03-04T07:38:58.808527+0000","last_became_active":"2024-03-04T07:37:54.127800+0000","last_became_peered":"2024-03-04T07:37:54.127800+0000","last_unstale":"2024-03-04T07:38:58.808527+0000","last_undegraded":"2024-03-04T07:38:58.808527+0000","last_fullsized":"2024-03-04T07:38:58.808527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:54.046873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504475+0000","last_change":"2024-03-04T07:37:55.133482+0000","last_active":"2024-03-04T07:39:27.504475+0000","last_peered":"2024-03-04T07:39:27.504475+0000","last_clean":"2024-03-04T07:39:27.504475+0000","last_became_active":"2024-03-04T07:37:55.133137+0000","last_became_peered":"2024-03-04T07:37:55.133137+0000","last_unstale":"2024-03-04T07:39:27.504475+0000","last_undegraded":"2024-03-04T07:39:27.504475+0000","last_fullsized":"2024-03-04T07:39:27.504475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:53.509564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:40:04.733943+0000","last_change":"2024-03-04T07:37:54.134294+0000","last_active":"2024-03-04T07:40:04.733943+0000","last_peered":"2024-03-04T07:40:04.733943+0000","last_clean":"2024-03-04T07:40:04.733943+0000","last_became_active":"2024-03-04T07:37:54.133677+0000","last_became_peered":"2024-03-04T07:37:54.133677+0000","last_unstale":"2024-03-04T07:40:04.733943+0000","last_undegraded":"2024-03-04T07:40:04.733943+0000","last_fullsized":"2024-03-04T07:40:04.733943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:57:54.082419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522888+0000","last_change":"2024-03-04T07:37:55.138305+0000","last_active":"2024-03-04T07:39:27.522888+0000","last_peered":"2024-03-04T07:39:27.522888+0000","last_clean":"2024-03-04T07:39:27.522888+0000","last_became_active":"2024-03-04T07:37:55.129937+0000","last_became_peered":"2024-03-04T07:37:55.129937+0000","last_unstale":"2024-03-04T07:39:27.522888+0000","last_undegraded":"2024-03-04T07:39:27.522888+0000","last_fullsized":"2024-03-04T07:39:27.522888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:45.136539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021306+0000","last_change":"2024-03-04T07:37:54.133817+0000","last_active":"2024-03-04T07:38:59.021306+0000","last_peered":"2024-03-04T07:38:59.021306+0000","last_clean":"2024-03-04T07:38:59.021306+0000","last_became_active":"2024-03-04T07:37:54.133471+0000","last_became_peered":"2024-03-04T07:37:54.133471+0000","last_unstale":"2024-03-04T07:38:59.021306+0000","last_undegraded":"2024-03-04T07:38:59.021306+0000","last_fullsized":"2024-03-04T07:38:59.021306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:00:51.483859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.517445+0000","last_change":"2024-03-04T07:37:55.144639+0000","last_active":"2024-03-04T07:39:27.517445+0000","last_peered":"2024-03-04T07:39:27.517445+0000","last_clean":"2024-03-04T07:39:27.517445+0000","last_became_active":"2024-03-04T07:37:55.144194+0000","last_became_peered":"2024-03-04T07:37:55.144194+0000","last_unstale":"2024-03-04T07:39:27.517445+0000","last_undegraded":"2024-03-04T07:39:27.517445+0000","last_fullsized":"2024-03-04T07:39:27.517445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:42.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.816116+0000","last_change":"2024-03-04T07:38:58.815642+0000","last_active":"2024-03-04T07:38:58.816116+0000","last_peered":"2024-03-04T07:38:58.816116+0000","last_clean":"2024-03-04T07:38:58.816116+0000","last_became_active":"2024-03-04T07:38:58.815361+0000","last_became_peered":"2024-03-04T07:38:58.815361+0000","last_unstale":"2024-03-04T07:38:58.816116+0000","last_undegraded":"2024-03-04T07:38:58.816116+0000","last_fullsized":"2024-03-04T07:38:58.816116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:48:15.751929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487964+0000","last_change":"2024-03-04T07:37:55.115145+0000","last_active":"2024-03-04T07:39:27.487964+0000","last_peered":"2024-03-04T07:39:27.487964+0000","last_clean":"2024-03-04T07:39:27.487964+0000","last_became_active":"2024-03-04T07:37:55.114791+0000","last_became_peered":"2024-03-04T07:37:55.114791+0000","last_unstale":"2024-03-04T07:39:27.487964+0000","last_undegraded":"2024-03-04T07:39:27.487964+0000","last_fullsized":"2024-03-04T07:39:27.487964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:19.685977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.806430+0000","last_change":"2024-03-04T07:37:54.141630+0000","last_active":"2024-03-04T07:38:58.806430+0000","last_peered":"2024-03-04T07:38:58.806430+0000","last_clean":"2024-03-04T07:38:58.806430+0000","last_became_active":"2024-03-04T07:37:54.139820+0000","last_became_peered":"2024-03-04T07:37:54.139820+0000","last_unstale":"2024-03-04T07:38:58.806430+0000","last_undegraded":"2024-03-04T07:38:58.806430+0000","last_fullsized":"2024-03-04T07:38:58.806430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:22.419437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503153+0000","last_change":"2024-03-04T07:37:55.133270+0000","last_active":"2024-03-04T07:39:27.503153+0000","last_peered":"2024-03-04T07:39:27.503153+0000","last_clean":"2024-03-04T07:39:27.503153+0000","last_became_active":"2024-03-04T07:37:55.132909+0000","last_became_peered":"2024-03-04T07:37:55.132909+0000","last_unstale":"2024-03-04T07:39:27.503153+0000","last_undegraded":"2024-03-04T07:39:27.503153+0000","last_fullsized":"2024-03-04T07:39:27.503153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:21.026141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812572+0000","last_change":"2024-03-04T07:37:54.135988+0000","last_active":"2024-03-04T07:38:58.812572+0000","last_peered":"2024-03-04T07:38:58.812572+0000","last_clean":"2024-03-04T07:38:58.812572+0000","last_became_active":"2024-03-04T07:37:54.135627+0000","last_became_peered":"2024-03-04T07:37:54.135627+0000","last_unstale":"2024-03-04T07:38:58.812572+0000","last_undegraded":"2024-03-04T07:38:58.812572+0000","last_fullsized":"2024-03-04T07:38:58.812572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:06.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522153+0000","last_change":"2024-03-04T07:37:55.144551+0000","last_active":"2024-03-04T07:39:27.522153+0000","last_peered":"2024-03-04T07:39:27.522153+0000","last_clean":"2024-03-04T07:39:27.522153+0000","last_became_active":"2024-03-04T07:37:55.143882+0000","last_became_peered":"2024-03-04T07:37:55.143882+0000","last_unstale":"2024-03-04T07:39:27.522153+0000","last_undegraded":"2024-03-04T07:39:27.522153+0000","last_fullsized":"2024-03-04T07:39:27.522153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:07.254941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.817917+0000","last_change":"2024-03-04T07:38:58.817440+0000","last_active":"2024-03-04T07:38:58.817917+0000","last_peered":"2024-03-04T07:38:58.817917+0000","last_clean":"2024-03-04T07:38:58.817917+0000","last_became_active":"2024-03-04T07:38:58.813620+0000","last_became_peered":"2024-03-04T07:38:58.813620+0000","last_unstale":"2024-03-04T07:38:58.817917+0000","last_undegraded":"2024-03-04T07:38:58.817917+0000","last_fullsized":"2024-03-04T07:38:58.817917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:10:41.554608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.505218+0000","last_change":"2024-03-04T07:37:55.133870+0000","last_active":"2024-03-04T07:39:27.505218+0000","last_peered":"2024-03-04T07:39:27.505218+0000","last_clean":"2024-03-04T07:39:27.505218+0000","last_became_active":"2024-03-04T07:37:55.133542+0000","last_became_peered":"2024-03-04T07:37:55.133542+0000","last_unstale":"2024-03-04T07:39:27.505218+0000","last_undegraded":"2024-03-04T07:39:27.505218+0000","last_fullsized":"2024-03-04T07:39:27.505218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:17:44.373426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811749+0000","last_change":"2024-03-04T07:37:54.126326+0000","last_active":"2024-03-04T07:38:58.811749+0000","last_peered":"2024-03-04T07:38:58.811749+0000","last_clean":"2024-03-04T07:38:58.811749+0000","last_became_active":"2024-03-04T07:37:54.125989+0000","last_became_peered":"2024-03-04T07:37:54.125989+0000","last_unstale":"2024-03-04T07:38:58.811749+0000","last_undegraded":"2024-03-04T07:38:58.811749+0000","last_fullsized":"2024-03-04T07:38:58.811749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:41.108528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488757+0000","last_change":"2024-03-04T07:37:55.120052+0000","last_active":"2024-03-04T07:39:27.488757+0000","last_peered":"2024-03-04T07:39:27.488757+0000","last_clean":"2024-03-04T07:39:27.488757+0000","last_became_active":"2024-03-04T07:37:55.119708+0000","last_became_peered":"2024-03-04T07:37:55.119708+0000","last_unstale":"2024-03-04T07:39:27.488757+0000","last_undegraded":"2024-03-04T07:39:27.488757+0000","last_fullsized":"2024-03-04T07:39:27.488757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:25:17.679416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808512+0000","last_change":"2024-03-04T07:37:54.134018+0000","last_active":"2024-03-04T07:38:58.808512+0000","last_peered":"2024-03-04T07:38:58.808512+0000","last_clean":"2024-03-04T07:38:58.808512+0000","last_became_active":"2024-03-04T07:37:54.133673+0000","last_became_peered":"2024-03-04T07:37:54.133673+0000","last_unstale":"2024-03-04T07:38:58.808512+0000","last_undegraded":"2024-03-04T07:38:58.808512+0000","last_fullsized":"2024-03-04T07:38:58.808512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:49.438338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489514+0000","last_change":"2024-03-04T07:38:58.817071+0000","last_active":"2024-03-04T07:39:27.489514+0000","last_peered":"2024-03-04T07:39:27.489514+0000","last_clean":"2024-03-04T07:39:27.489514+0000","last_became_active":"2024-03-04T07:38:58.812139+0000","last_became_peered":"2024-03-04T07:38:58.812139+0000","last_unstale":"2024-03-04T07:39:27.489514+0000","last_undegraded":"2024-03-04T07:39:27.489514+0000","last_fullsized":"2024-03-04T07:39:27.489514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:37.557255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805106+0000","last_change":"2024-03-04T07:37:54.140806+0000","last_active":"2024-03-04T07:38:58.805106+0000","last_peered":"2024-03-04T07:38:58.805106+0000","last_clean":"2024-03-04T07:38:58.805106+0000","last_became_active":"2024-03-04T07:37:54.140419+0000","last_became_peered":"2024-03-04T07:37:54.140419+0000","last_unstale":"2024-03-04T07:38:58.805106+0000","last_undegraded":"2024-03-04T07:38:58.805106+0000","last_fullsized":"2024-03-04T07:38:58.805106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:13:59.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489032+0000","last_change":"2024-03-04T07:37:55.113658+0000","last_active":"2024-03-04T07:39:27.489032+0000","last_peered":"2024-03-04T07:39:27.489032+0000","last_clean":"2024-03-04T07:39:27.489032+0000","last_became_active":"2024-03-04T07:37:55.113300+0000","last_became_peered":"2024-03-04T07:37:55.113300+0000","last_unstale":"2024-03-04T07:39:27.489032+0000","last_undegraded":"2024-03-04T07:39:27.489032+0000","last_fullsized":"2024-03-04T07:39:27.489032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:43:56.907387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487646+0000","last_change":"2024-03-04T07:37:55.117769+0000","last_active":"2024-03-04T07:39:27.487646+0000","last_peered":"2024-03-04T07:39:27.487646+0000","last_clean":"2024-03-04T07:39:27.487646+0000","last_became_active":"2024-03-04T07:37:55.117408+0000","last_became_peered":"2024-03-04T07:37:55.117408+0000","last_unstale":"2024-03-04T07:39:27.487646+0000","last_undegraded":"2024-03-04T07:39:27.487646+0000","last_fullsized":"2024-03-04T07:39:27.487646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:22.143108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.483985+0000","last_change":"2024-03-04T07:37:55.115581+0000","last_active":"2024-03-04T07:39:27.483985+0000","last_peered":"2024-03-04T07:39:27.483985+0000","last_clean":"2024-03-04T07:39:27.483985+0000","last_became_active":"2024-03-04T07:37:55.113387+0000","last_became_peered":"2024-03-04T07:37:55.113387+0000","last_unstale":"2024-03-04T07:39:27.483985+0000","last_undegraded":"2024-03-04T07:39:27.483985+0000","last_fullsized":"2024-03-04T07:39:27.483985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:59.273650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.492691+0000","last_change":"2024-03-04T07:37:55.135175+0000","last_active":"2024-03-04T07:39:27.492691+0000","last_peered":"2024-03-04T07:39:27.492691+0000","last_clean":"2024-03-04T07:39:27.492691+0000","last_became_active":"2024-03-04T07:37:55.134787+0000","last_became_peered":"2024-03-04T07:37:55.134787+0000","last_unstale":"2024-03-04T07:39:27.492691+0000","last_undegraded":"2024-03-04T07:39:27.492691+0000","last_fullsized":"2024-03-04T07:39:27.492691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:04.227886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511558+0000","last_change":"2024-03-04T07:37:55.138146+0000","last_active":"2024-03-04T07:39:27.511558+0000","last_peered":"2024-03-04T07:39:27.511558+0000","last_clean":"2024-03-04T07:39:27.511558+0000","last_became_active":"2024-03-04T07:37:55.137806+0000","last_became_peered":"2024-03-04T07:37:55.137806+0000","last_unstale":"2024-03-04T07:39:27.511558+0000","last_undegraded":"2024-03-04T07:39:27.511558+0000","last_fullsized":"2024-03-04T07:39:27.511558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:10.596224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.507386+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.507386+0000","last_peered":"2024-03-04T07:39:27.507386+0000","last_clean":"2024-03-04T07:39:27.507386+0000","last_became_active":"2024-03-04T07:37:55.140670+0000","last_became_peered":"2024-03-04T07:37:55.140670+0000","last_unstale":"2024-03-04T07:39:27.507386+0000","last_undegraded":"2024-03-04T07:39:27.507386+0000","last_fullsized":"2024-03-04T07:39:27.507386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:29:54.439417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487788+0000","last_change":"2024-03-04T07:37:55.115015+0000","last_active":"2024-03-04T07:39:27.487788+0000","last_peered":"2024-03-04T07:39:27.487788+0000","last_clean":"2024-03-04T07:39:27.487788+0000","last_became_active":"2024-03-04T07:37:55.114668+0000","last_became_peered":"2024-03-04T07:37:55.114668+0000","last_unstale":"2024-03-04T07:39:27.487788+0000","last_undegraded":"2024-03-04T07:39:27.487788+0000","last_fullsized":"2024-03-04T07:39:27.487788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:29:03.309234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.482820+0000","last_change":"2024-03-04T07:37:55.119514+0000","last_active":"2024-03-04T07:39:27.482820+0000","last_peered":"2024-03-04T07:39:27.482820+0000","last_clean":"2024-03-04T07:39:27.482820+0000","last_became_active":"2024-03-04T07:37:55.119166+0000","last_became_peered":"2024-03-04T07:37:55.119166+0000","last_unstale":"2024-03-04T07:39:27.482820+0000","last_undegraded":"2024-03-04T07:39:27.482820+0000","last_fullsized":"2024-03-04T07:39:27.482820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:33.468311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.520592+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.520592+0000","last_peered":"2024-03-04T07:39:27.520592+0000","last_clean":"2024-03-04T07:39:27.520592+0000","last_became_active":"2024-03-04T07:37:55.140723+0000","last_became_peered":"2024-03-04T07:37:55.140723+0000","last_unstale":"2024-03-04T07:39:27.520592+0000","last_undegraded":"2024-03-04T07:39:27.520592+0000","last_fullsized":"2024-03-04T07:39:27.520592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:29.751190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.510302+0000","last_change":"2024-03-04T07:37:55.127627+0000","last_active":"2024-03-04T07:39:27.510302+0000","last_peered":"2024-03-04T07:39:27.510302+0000","last_clean":"2024-03-04T07:39:27.510302+0000","last_became_active":"2024-03-04T07:37:55.127217+0000","last_became_peered":"2024-03-04T07:37:55.127217+0000","last_unstale":"2024-03-04T07:39:27.510302+0000","last_undegraded":"2024-03-04T07:39:27.510302+0000","last_fullsized":"2024-03-04T07:39:27.510302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:47.826970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519782+0000","last_change":"2024-03-04T07:37:55.129445+0000","last_active":"2024-03-04T07:39:27.519782+0000","last_peered":"2024-03-04T07:39:27.519782+0000","last_clean":"2024-03-04T07:39:27.519782+0000","last_became_active":"2024-03-04T07:37:55.129013+0000","last_became_peered":"2024-03-04T07:37:55.129013+0000","last_unstale":"2024-03-04T07:39:27.519782+0000","last_undegraded":"2024-03-04T07:39:27.519782+0000","last_fullsized":"2024-03-04T07:39:27.519782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:15.578436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523619+0000","last_change":"2024-03-04T07:37:55.126397+0000","last_active":"2024-03-04T07:39:27.523619+0000","last_peered":"2024-03-04T07:39:27.523619+0000","last_clean":"2024-03-04T07:39:27.523619+0000","last_became_active":"2024-03-04T07:37:55.126092+0000","last_became_peered":"2024-03-04T07:37:55.126092+0000","last_unstale":"2024-03-04T07:39:27.523619+0000","last_undegraded":"2024-03-04T07:39:27.523619+0000","last_fullsized":"2024-03-04T07:39:27.523619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:47.968250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511142+0000","last_change":"2024-03-04T07:37:55.140265+0000","last_active":"2024-03-04T07:39:27.511142+0000","last_peered":"2024-03-04T07:39:27.511142+0000","last_clean":"2024-03-04T07:39:27.511142+0000","last_became_active":"2024-03-04T07:37:55.139768+0000","last_became_peered":"2024-03-04T07:37:55.139768+0000","last_unstale":"2024-03-04T07:39:27.511142+0000","last_undegraded":"2024-03-04T07:39:27.511142+0000","last_fullsized":"2024-03-04T07:39:27.511142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:52.924188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488738+0000","last_change":"2024-03-04T07:37:55.118929+0000","last_active":"2024-03-04T07:39:27.488738+0000","last_peered":"2024-03-04T07:39:27.488738+0000","last_clean":"2024-03-04T07:39:27.488738+0000","last_became_active":"2024-03-04T07:37:55.118549+0000","last_became_peered":"2024-03-04T07:37:55.118549+0000","last_unstale":"2024-03-04T07:39:27.488738+0000","last_undegraded":"2024-03-04T07:39:27.488738+0000","last_fullsized":"2024-03-04T07:39:27.488738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:15.599080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244954,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62403,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":314,"num_read_kb":219,"num_write":84,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":187209,"data_compressed":5550,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":455726,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1108010,"data_compressed":43363,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456425,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456330,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332162,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1046793,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455843,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":517021,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455816,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":517365,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182489,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":455947,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920853,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455748,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":1046393,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61605,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61639,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61799,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T07:40:28.219 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T07:40:28.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 -- 172.21.15.116:0/2261809764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 msgr2=0x7f148010bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 --2- 172.21.15.116:0/2261809764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f148010bc50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1478009be0 tx=0x7f147802bef0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 -- 172.21.15.116:0/2261809764 shutdown_connections 2024-03-04T07:40:28.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 --2- 172.21.15.116:0/2261809764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f148010bc50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 --2- 172.21.15.116:0/2261809764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1480104a00 0x7f1480104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 --2- 172.21.15.116:0/2261809764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1480103810 0x7f1480103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 -- 172.21.15.116:0/2261809764 >> 172.21.15.116:0/2261809764 conn(0x7f14800fedc0 msgr2=0x7f14801011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:28.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 -- 172.21.15.116:0/2261809764 shutdown_connections 2024-03-04T07:40:28.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.449+0000 7f1488400640 1 -- 172.21.15.116:0/2261809764 wait complete. 2024-03-04T07:40:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.450+0000 7f1488400640 1 Processor -- start 2024-03-04T07:40:28.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.450+0000 7f1488400640 1 -- start start 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1488400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1480103810 0x7f1480199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1488400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1480104a00 0x7f148019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1488400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f14801a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1488400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148010f4e0 con 0x7f1480104a00 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1488400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148010f360 con 0x7f14801053c0 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1488400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148010f660 con 0x7f1480103810 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1485a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1480103810 0x7f1480199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1486400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f14801a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1485000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1480104a00 0x7f148019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1485a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1480103810 0x7f1480199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:35306/0 (socket says 172.21.15.116:35306) 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1485a00640 1 -- 172.21.15.116:0/2538013884 learned_addr learned my addr 172.21.15.116:0/2538013884 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1486400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f14801a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56918/0 (socket says 172.21.15.116:56918) 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1486400640 1 -- 172.21.15.116:0/2538013884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1480103810 msgr2=0x7f1480199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1486400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1480103810 0x7f1480199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1486400640 1 -- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1480104a00 msgr2=0x7f148019a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1486400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1480104a00 0x7f148019a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.451+0000 7f1486400640 1 -- 172.21.15.116:0/2538013884 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14801a1720 con 0x7f14801053c0 2024-03-04T07:40:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.452+0000 7f1486400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f14801a10b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f147802c430 tx=0x7f147802cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:28.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.452+0000 7f147e000640 1 -- 172.21.15.116:0/2538013884 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478043020 con 0x7f14801053c0 2024-03-04T07:40:28.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.452+0000 7f147e000640 1 -- 172.21.15.116:0/2538013884 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1478003e50 con 0x7f14801053c0 2024-03-04T07:40:28.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.452+0000 7f147e000640 1 -- 172.21.15.116:0/2538013884 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478034430 con 0x7f14801053c0 2024-03-04T07:40:28.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.452+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14801a19b0 con 0x7f14801053c0 2024-03-04T07:40:28.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.452+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14801a1eb0 con 0x7f14801053c0 2024-03-04T07:40:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.453+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148010aad0 con 0x7f14801053c0 2024-03-04T07:40:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.455+0000 7f147e000640 1 -- 172.21.15.116:0/2538013884 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1478046020 con 0x7f14801053c0 2024-03-04T07:40:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.456+0000 7f147e000640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14680a1e00 0x7f14680a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.457+0000 7f147e000640 1 -- 172.21.15.116:0/2538013884 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f14780f02e0 con 0x7f14801053c0 2024-03-04T07:40:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.457+0000 7f1485a00640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14680a1e00 0x7f14680a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.457+0000 7f1485a00640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14680a1e00 0x7f14680a42c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f14740058c0 tx=0x7f1474005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:28.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.460+0000 7f147e000640 1 -- 172.21.15.116:0/2538013884 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f147804d020 con 0x7f14801053c0 2024-03-04T07:40:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.657+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1480061a10 con 0x7f14680a1e00 2024-03-04T07:40:28.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.682+0000 7f147e000640 1 -- 172.21.15.116:0/2538013884 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259026 (secure 0 0 0) 0x7f1480061a10 con 0x7f14680a1e00 2024-03-04T07:40:28.687 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:28.688 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T07:40:28.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14680a1e00 msgr2=0x7f14680a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14680a1e00 0x7f14680a42c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f14740058c0 tx=0x7f1474005f90 comp rx=0 tx=0).stop 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 msgr2=0x7f14801a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f14801a10b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f147802c430 tx=0x7f147802cb90 comp rx=0 tx=0).stop 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 shutdown_connections 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f14801053c0 0x7f14801a10b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1480104a00 0x7f148019a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14680a1e00 0x7f14680a42c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 --2- 172.21.15.116:0/2538013884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1480103810 0x7f1480199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 >> 172.21.15.116:0/2538013884 conn(0x7f14800fedc0 msgr2=0x7f1480107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.688+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 shutdown_connections 2024-03-04T07:40:28.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.689+0000 7f1488400640 1 -- 172.21.15.116:0/2538013884 wait complete. 2024-03-04T07:40:28.707 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-03-04T07:40:27.534951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653005,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":440,"num_read_kb":328,"num_write":267,"num_write_kb":3029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":710,"ondisk_log_size":710,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360564,"kb_used_data":11412,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581260,"statfs":{"total":1151940427776,"available":1151571210240,"internally_reserved":0,"allocated":11685888,"data_stored":7427417,"data_compressed":182424,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":310,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004497"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509850+0000","last_change":"2024-03-04T07:37:55.137540+0000","last_active":"2024-03-04T07:39:27.509850+0000","last_peered":"2024-03-04T07:39:27.509850+0000","last_clean":"2024-03-04T07:39:27.509850+0000","last_became_active":"2024-03-04T07:37:55.137183+0000","last_became_peered":"2024-03-04T07:37:55.137183+0000","last_unstale":"2024-03-04T07:39:27.509850+0000","last_undegraded":"2024-03-04T07:39:27.509850+0000","last_fullsized":"2024-03-04T07:39:27.509850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:15.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503533+0000","last_change":"2024-03-04T07:37:55.136049+0000","last_active":"2024-03-04T07:39:27.503533+0000","last_peered":"2024-03-04T07:39:27.503533+0000","last_clean":"2024-03-04T07:39:27.503533+0000","last_became_active":"2024-03-04T07:37:55.132153+0000","last_became_peered":"2024-03-04T07:37:55.132153+0000","last_unstale":"2024-03-04T07:39:27.503533+0000","last_undegraded":"2024-03-04T07:39:27.503533+0000","last_fullsized":"2024-03-04T07:39:27.503533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:52.426892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.508509+0000","last_change":"2024-03-04T07:37:55.133571+0000","last_active":"2024-03-04T07:39:27.508509+0000","last_peered":"2024-03-04T07:39:27.508509+0000","last_clean":"2024-03-04T07:39:27.508509+0000","last_became_active":"2024-03-04T07:37:55.133214+0000","last_became_peered":"2024-03-04T07:37:55.133214+0000","last_unstale":"2024-03-04T07:39:27.508509+0000","last_undegraded":"2024-03-04T07:39:27.508509+0000","last_fullsized":"2024-03-04T07:39:27.508509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:17.924826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.480393+0000","last_change":"2024-03-04T07:37:55.118146+0000","last_active":"2024-03-04T07:39:27.480393+0000","last_peered":"2024-03-04T07:39:27.480393+0000","last_clean":"2024-03-04T07:39:27.480393+0000","last_became_active":"2024-03-04T07:37:55.117687+0000","last_became_peered":"2024-03-04T07:37:55.117687+0000","last_unstale":"2024-03-04T07:39:27.480393+0000","last_undegraded":"2024-03-04T07:39:27.480393+0000","last_fullsized":"2024-03-04T07:39:27.480393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:34:35.343400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.499265+0000","last_change":"2024-03-04T07:37:55.135943+0000","last_active":"2024-03-04T07:39:27.499265+0000","last_peered":"2024-03-04T07:39:27.499265+0000","last_clean":"2024-03-04T07:39:27.499265+0000","last_became_active":"2024-03-04T07:37:55.135549+0000","last_became_peered":"2024-03-04T07:37:55.135549+0000","last_unstale":"2024-03-04T07:39:27.499265+0000","last_undegraded":"2024-03-04T07:39:27.499265+0000","last_fullsized":"2024-03-04T07:39:27.499265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:35:53.953771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487475+0000","last_change":"2024-03-04T07:37:55.116949+0000","last_active":"2024-03-04T07:39:27.487475+0000","last_peered":"2024-03-04T07:39:27.487475+0000","last_clean":"2024-03-04T07:39:27.487475+0000","last_became_active":"2024-03-04T07:37:55.116620+0000","last_became_peered":"2024-03-04T07:37:55.116620+0000","last_unstale":"2024-03-04T07:39:27.487475+0000","last_undegraded":"2024-03-04T07:39:27.487475+0000","last_fullsized":"2024-03-04T07:39:27.487475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:39.655217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489760+0000","last_change":"2024-03-04T07:37:55.100975+0000","last_active":"2024-03-04T07:39:27.489760+0000","last_peered":"2024-03-04T07:39:27.489760+0000","last_clean":"2024-03-04T07:39:27.489760+0000","last_became_active":"2024-03-04T07:37:55.100636+0000","last_became_peered":"2024-03-04T07:37:55.100636+0000","last_unstale":"2024-03-04T07:39:27.489760+0000","last_undegraded":"2024-03-04T07:39:27.489760+0000","last_fullsized":"2024-03-04T07:39:27.489760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:29.593763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523017+0000","last_change":"2024-03-04T07:37:55.131895+0000","last_active":"2024-03-04T07:39:27.523017+0000","last_peered":"2024-03-04T07:39:27.523017+0000","last_clean":"2024-03-04T07:39:27.523017+0000","last_became_active":"2024-03-04T07:37:55.129682+0000","last_became_peered":"2024-03-04T07:37:55.129682+0000","last_unstale":"2024-03-04T07:39:27.523017+0000","last_undegraded":"2024-03-04T07:39:27.523017+0000","last_fullsized":"2024-03-04T07:39:27.523017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:54.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489551+0000","last_change":"2024-03-04T07:37:55.114475+0000","last_active":"2024-03-04T07:39:27.489551+0000","last_peered":"2024-03-04T07:39:27.489551+0000","last_clean":"2024-03-04T07:39:27.489551+0000","last_became_active":"2024-03-04T07:37:55.114068+0000","last_became_peered":"2024-03-04T07:37:55.114068+0000","last_unstale":"2024-03-04T07:39:27.489551+0000","last_undegraded":"2024-03-04T07:39:27.489551+0000","last_fullsized":"2024-03-04T07:39:27.489551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:21.466143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.477511+0000","last_change":"2024-03-04T07:37:55.115660+0000","last_active":"2024-03-04T07:39:27.477511+0000","last_peered":"2024-03-04T07:39:27.477511+0000","last_clean":"2024-03-04T07:39:27.477511+0000","last_became_active":"2024-03-04T07:37:55.115324+0000","last_became_peered":"2024-03-04T07:37:55.115324+0000","last_unstale":"2024-03-04T07:39:27.477511+0000","last_undegraded":"2024-03-04T07:39:27.477511+0000","last_fullsized":"2024-03-04T07:39:27.477511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:42.119728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489282+0000","last_change":"2024-03-04T07:37:55.119834+0000","last_active":"2024-03-04T07:39:27.489282+0000","last_peered":"2024-03-04T07:39:27.489282+0000","last_clean":"2024-03-04T07:39:27.489282+0000","last_became_active":"2024-03-04T07:37:55.119467+0000","last_became_peered":"2024-03-04T07:37:55.119467+0000","last_unstale":"2024-03-04T07:39:27.489282+0000","last_undegraded":"2024-03-04T07:39:27.489282+0000","last_fullsized":"2024-03-04T07:39:27.489282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:49:19.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.479644+0000","last_change":"2024-03-04T07:37:55.118154+0000","last_active":"2024-03-04T07:39:27.479644+0000","last_peered":"2024-03-04T07:39:27.479644+0000","last_clean":"2024-03-04T07:39:27.479644+0000","last_became_active":"2024-03-04T07:37:55.117809+0000","last_became_peered":"2024-03-04T07:37:55.117809+0000","last_unstale":"2024-03-04T07:39:27.479644+0000","last_undegraded":"2024-03-04T07:39:27.479644+0000","last_fullsized":"2024-03-04T07:39:27.479644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:50.195946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503528+0000","last_change":"2024-03-04T07:37:55.123325+0000","last_active":"2024-03-04T07:39:27.503528+0000","last_peered":"2024-03-04T07:39:27.503528+0000","last_clean":"2024-03-04T07:39:27.503528+0000","last_became_active":"2024-03-04T07:37:55.122848+0000","last_became_peered":"2024-03-04T07:37:55.122848+0000","last_unstale":"2024-03-04T07:39:27.503528+0000","last_undegraded":"2024-03-04T07:39:27.503528+0000","last_fullsized":"2024-03-04T07:39:27.503528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:18.842011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503051+0000","last_change":"2024-03-04T07:37:55.132447+0000","last_active":"2024-03-04T07:39:27.503051+0000","last_peered":"2024-03-04T07:39:27.503051+0000","last_clean":"2024-03-04T07:39:27.503051+0000","last_became_active":"2024-03-04T07:37:55.131512+0000","last_became_peered":"2024-03-04T07:37:55.131512+0000","last_unstale":"2024-03-04T07:39:27.503051+0000","last_undegraded":"2024-03-04T07:39:27.503051+0000","last_fullsized":"2024-03-04T07:39:27.503051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:23.960029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504478+0000","last_change":"2024-03-04T07:37:55.131715+0000","last_active":"2024-03-04T07:39:27.504478+0000","last_peered":"2024-03-04T07:39:27.504478+0000","last_clean":"2024-03-04T07:39:27.504478+0000","last_became_active":"2024-03-04T07:37:55.131326+0000","last_became_peered":"2024-03-04T07:37:55.131326+0000","last_unstale":"2024-03-04T07:39:27.504478+0000","last_undegraded":"2024-03-04T07:39:27.504478+0000","last_fullsized":"2024-03-04T07:39:27.504478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:56.231578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502604+0000","last_change":"2024-03-04T07:37:55.137395+0000","last_active":"2024-03-04T07:39:27.502604+0000","last_peered":"2024-03-04T07:39:27.502604+0000","last_clean":"2024-03-04T07:39:27.502604+0000","last_became_active":"2024-03-04T07:37:55.137054+0000","last_became_peered":"2024-03-04T07:37:55.137054+0000","last_unstale":"2024-03-04T07:39:27.502604+0000","last_undegraded":"2024-03-04T07:39:27.502604+0000","last_fullsized":"2024-03-04T07:39:27.502604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:19.222481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509061+0000","last_change":"2024-03-04T07:37:55.137414+0000","last_active":"2024-03-04T07:39:27.509061+0000","last_peered":"2024-03-04T07:39:27.509061+0000","last_clean":"2024-03-04T07:39:27.509061+0000","last_became_active":"2024-03-04T07:37:55.137157+0000","last_became_peered":"2024-03-04T07:37:55.137157+0000","last_unstale":"2024-03-04T07:39:27.509061+0000","last_undegraded":"2024-03-04T07:39:27.509061+0000","last_fullsized":"2024-03-04T07:39:27.509061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:36.593491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502012+0000","last_change":"2024-03-04T07:37:55.121960+0000","last_active":"2024-03-04T07:39:27.502012+0000","last_peered":"2024-03-04T07:39:27.502012+0000","last_clean":"2024-03-04T07:39:27.502012+0000","last_became_active":"2024-03-04T07:37:55.115543+0000","last_became_peered":"2024-03-04T07:37:55.115543+0000","last_unstale":"2024-03-04T07:39:27.502012+0000","last_undegraded":"2024-03-04T07:39:27.502012+0000","last_fullsized":"2024-03-04T07:39:27.502012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:36.320475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.506619+0000","last_change":"2024-03-04T07:37:55.128908+0000","last_active":"2024-03-04T07:39:27.506619+0000","last_peered":"2024-03-04T07:39:27.506619+0000","last_clean":"2024-03-04T07:39:27.506619+0000","last_became_active":"2024-03-04T07:37:55.128500+0000","last_became_peered":"2024-03-04T07:37:55.128500+0000","last_unstale":"2024-03-04T07:39:27.506619+0000","last_undegraded":"2024-03-04T07:39:27.506619+0000","last_fullsized":"2024-03-04T07:39:27.506619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:10.598424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805031+0000","last_change":"2024-03-04T07:37:54.121610+0000","last_active":"2024-03-04T07:38:58.805031+0000","last_peered":"2024-03-04T07:38:58.805031+0000","last_clean":"2024-03-04T07:38:58.805031+0000","last_became_active":"2024-03-04T07:37:54.121150+0000","last_became_peered":"2024-03-04T07:37:54.121150+0000","last_unstale":"2024-03-04T07:38:58.805031+0000","last_undegraded":"2024-03-04T07:38:58.805031+0000","last_fullsized":"2024-03-04T07:38:58.805031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:05:03.965626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.518629+0000","last_change":"2024-03-04T07:37:55.138904+0000","last_active":"2024-03-04T07:39:27.518629+0000","last_peered":"2024-03-04T07:39:27.518629+0000","last_clean":"2024-03-04T07:39:27.518629+0000","last_became_active":"2024-03-04T07:37:55.138376+0000","last_became_peered":"2024-03-04T07:37:55.138376+0000","last_unstale":"2024-03-04T07:39:27.518629+0000","last_undegraded":"2024-03-04T07:39:27.518629+0000","last_fullsized":"2024-03-04T07:39:27.518629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:28.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807860+0000","last_change":"2024-03-04T07:37:54.139218+0000","last_active":"2024-03-04T07:38:58.807860+0000","last_peered":"2024-03-04T07:38:58.807860+0000","last_clean":"2024-03-04T07:38:58.807860+0000","last_became_active":"2024-03-04T07:37:54.138893+0000","last_became_peered":"2024-03-04T07:37:54.138893+0000","last_unstale":"2024-03-04T07:38:58.807860+0000","last_undegraded":"2024-03-04T07:38:58.807860+0000","last_fullsized":"2024-03-04T07:38:58.807860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:01.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511166+0000","last_change":"2024-03-04T07:37:55.135953+0000","last_active":"2024-03-04T07:39:27.511166+0000","last_peered":"2024-03-04T07:39:27.511166+0000","last_clean":"2024-03-04T07:39:27.511166+0000","last_became_active":"2024-03-04T07:37:55.132880+0000","last_became_peered":"2024-03-04T07:37:55.132880+0000","last_unstale":"2024-03-04T07:39:27.511166+0000","last_undegraded":"2024-03-04T07:39:27.511166+0000","last_fullsized":"2024-03-04T07:39:27.511166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:03.848428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808631+0000","last_change":"2024-03-04T07:37:54.122493+0000","last_active":"2024-03-04T07:38:58.808631+0000","last_peered":"2024-03-04T07:38:58.808631+0000","last_clean":"2024-03-04T07:38:58.808631+0000","last_became_active":"2024-03-04T07:37:54.122058+0000","last_became_peered":"2024-03-04T07:37:54.122058+0000","last_unstale":"2024-03-04T07:38:58.808631+0000","last_undegraded":"2024-03-04T07:38:58.808631+0000","last_fullsized":"2024-03-04T07:38:58.808631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:02.516175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.491455+0000","last_change":"2024-03-04T07:37:55.136155+0000","last_active":"2024-03-04T07:39:27.491455+0000","last_peered":"2024-03-04T07:39:27.491455+0000","last_clean":"2024-03-04T07:39:27.491455+0000","last_became_active":"2024-03-04T07:37:55.135796+0000","last_became_peered":"2024-03-04T07:37:55.135796+0000","last_unstale":"2024-03-04T07:39:27.491455+0000","last_undegraded":"2024-03-04T07:39:27.491455+0000","last_fullsized":"2024-03-04T07:39:27.491455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:37.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808949+0000","last_change":"2024-03-04T07:37:54.118780+0000","last_active":"2024-03-04T07:38:58.808949+0000","last_peered":"2024-03-04T07:38:58.808949+0000","last_clean":"2024-03-04T07:38:58.808949+0000","last_became_active":"2024-03-04T07:37:54.118448+0000","last_became_peered":"2024-03-04T07:37:54.118448+0000","last_unstale":"2024-03-04T07:38:58.808949+0000","last_undegraded":"2024-03-04T07:38:58.808949+0000","last_fullsized":"2024-03-04T07:38:58.808949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:25.547046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501807+0000","last_change":"2024-03-04T07:38:59.021729+0000","last_active":"2024-03-04T07:39:27.501807+0000","last_peered":"2024-03-04T07:39:27.501807+0000","last_clean":"2024-03-04T07:39:27.501807+0000","last_became_active":"2024-03-04T07:38:59.021127+0000","last_became_peered":"2024-03-04T07:38:59.021127+0000","last_unstale":"2024-03-04T07:39:27.501807+0000","last_undegraded":"2024-03-04T07:39:27.501807+0000","last_fullsized":"2024-03-04T07:39:27.501807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:08:07.825530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807893+0000","last_change":"2024-03-04T07:37:54.122241+0000","last_active":"2024-03-04T07:38:58.807893+0000","last_peered":"2024-03-04T07:38:58.807893+0000","last_clean":"2024-03-04T07:38:58.807893+0000","last_became_active":"2024-03-04T07:37:54.120990+0000","last_became_peered":"2024-03-04T07:37:54.120990+0000","last_unstale":"2024-03-04T07:38:58.807893+0000","last_undegraded":"2024-03-04T07:38:58.807893+0000","last_fullsized":"2024-03-04T07:38:58.807893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:03.233754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501726+0000","last_change":"2024-03-04T07:37:55.121692+0000","last_active":"2024-03-04T07:39:27.501726+0000","last_peered":"2024-03-04T07:39:27.501726+0000","last_clean":"2024-03-04T07:39:27.501726+0000","last_became_active":"2024-03-04T07:37:55.121345+0000","last_became_peered":"2024-03-04T07:37:55.121345+0000","last_unstale":"2024-03-04T07:39:27.501726+0000","last_undegraded":"2024-03-04T07:39:27.501726+0000","last_fullsized":"2024-03-04T07:39:27.501726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:52:42.045265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807447+0000","last_change":"2024-03-04T07:37:54.133489+0000","last_active":"2024-03-04T07:38:58.807447+0000","last_peered":"2024-03-04T07:38:58.807447+0000","last_clean":"2024-03-04T07:38:58.807447+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807447+0000","last_undegraded":"2024-03-04T07:38:58.807447+0000","last_fullsized":"2024-03-04T07:38:58.807447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:49:03.600907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.510134+0000","last_change":"2024-03-04T07:37:55.128278+0000","last_active":"2024-03-04T07:39:27.510134+0000","last_peered":"2024-03-04T07:39:27.510134+0000","last_clean":"2024-03-04T07:39:27.510134+0000","last_became_active":"2024-03-04T07:37:55.127913+0000","last_became_peered":"2024-03-04T07:37:55.127913+0000","last_unstale":"2024-03-04T07:39:27.510134+0000","last_undegraded":"2024-03-04T07:39:27.510134+0000","last_fullsized":"2024-03-04T07:39:27.510134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:54:18.623855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803866+0000","last_change":"2024-03-04T07:37:54.132195+0000","last_active":"2024-03-04T07:38:58.803866+0000","last_peered":"2024-03-04T07:38:58.803866+0000","last_clean":"2024-03-04T07:38:58.803866+0000","last_became_active":"2024-03-04T07:37:54.131830+0000","last_became_peered":"2024-03-04T07:37:54.131830+0000","last_unstale":"2024-03-04T07:38:58.803866+0000","last_undegraded":"2024-03-04T07:38:58.803866+0000","last_fullsized":"2024-03-04T07:38:58.803866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.744342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522878+0000","last_change":"2024-03-04T07:37:55.138281+0000","last_active":"2024-03-04T07:39:27.522878+0000","last_peered":"2024-03-04T07:39:27.522878+0000","last_clean":"2024-03-04T07:39:27.522878+0000","last_became_active":"2024-03-04T07:37:55.137919+0000","last_became_peered":"2024-03-04T07:37:55.137919+0000","last_unstale":"2024-03-04T07:39:27.522878+0000","last_undegraded":"2024-03-04T07:39:27.522878+0000","last_fullsized":"2024-03-04T07:39:27.522878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:53:06.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807612+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:38:58.807612+0000","last_peered":"2024-03-04T07:38:58.807612+0000","last_clean":"2024-03-04T07:38:58.807612+0000","last_became_active":"2024-03-04T07:37:54.121503+0000","last_became_peered":"2024-03-04T07:37:54.121503+0000","last_unstale":"2024-03-04T07:38:58.807612+0000","last_undegraded":"2024-03-04T07:38:58.807612+0000","last_fullsized":"2024-03-04T07:38:58.807612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:03:07.531246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519901+0000","last_change":"2024-03-04T07:37:55.126924+0000","last_active":"2024-03-04T07:39:27.519901+0000","last_peered":"2024-03-04T07:39:27.519901+0000","last_clean":"2024-03-04T07:39:27.519901+0000","last_became_active":"2024-03-04T07:37:55.126266+0000","last_became_peered":"2024-03-04T07:37:55.126266+0000","last_unstale":"2024-03-04T07:39:27.519901+0000","last_undegraded":"2024-03-04T07:39:27.519901+0000","last_fullsized":"2024-03-04T07:39:27.519901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:01.603430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807917+0000","last_change":"2024-03-04T07:37:54.119864+0000","last_active":"2024-03-04T07:38:58.807917+0000","last_peered":"2024-03-04T07:38:58.807917+0000","last_clean":"2024-03-04T07:38:58.807917+0000","last_became_active":"2024-03-04T07:37:54.119515+0000","last_became_peered":"2024-03-04T07:37:54.119515+0000","last_unstale":"2024-03-04T07:38:58.807917+0000","last_undegraded":"2024-03-04T07:38:58.807917+0000","last_fullsized":"2024-03-04T07:38:58.807917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:10.422458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.490884+0000","last_change":"2024-03-04T07:37:55.120135+0000","last_active":"2024-03-04T07:39:27.490884+0000","last_peered":"2024-03-04T07:39:27.490884+0000","last_clean":"2024-03-04T07:39:27.490884+0000","last_became_active":"2024-03-04T07:37:55.119670+0000","last_became_peered":"2024-03-04T07:37:55.119670+0000","last_unstale":"2024-03-04T07:39:27.490884+0000","last_undegraded":"2024-03-04T07:39:27.490884+0000","last_fullsized":"2024-03-04T07:39:27.490884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:09.688447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807249+0000","last_change":"2024-03-04T07:37:54.120222+0000","last_active":"2024-03-04T07:38:58.807249+0000","last_peered":"2024-03-04T07:38:58.807249+0000","last_clean":"2024-03-04T07:38:58.807249+0000","last_became_active":"2024-03-04T07:37:54.119802+0000","last_became_peered":"2024-03-04T07:37:54.119802+0000","last_unstale":"2024-03-04T07:38:58.807249+0000","last_undegraded":"2024-03-04T07:38:58.807249+0000","last_fullsized":"2024-03-04T07:38:58.807249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:12:40.811870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509588+0000","last_change":"2024-03-04T07:37:55.136371+0000","last_active":"2024-03-04T07:39:27.509588+0000","last_peered":"2024-03-04T07:39:27.509588+0000","last_clean":"2024-03-04T07:39:27.509588+0000","last_became_active":"2024-03-04T07:37:55.132587+0000","last_became_peered":"2024-03-04T07:37:55.132587+0000","last_unstale":"2024-03-04T07:39:27.509588+0000","last_undegraded":"2024-03-04T07:39:27.509588+0000","last_fullsized":"2024-03-04T07:39:27.509588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:24.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.809061+0000","last_change":"2024-03-04T07:37:54.123041+0000","last_active":"2024-03-04T07:38:58.809061+0000","last_peered":"2024-03-04T07:38:58.809061+0000","last_clean":"2024-03-04T07:38:58.809061+0000","last_became_active":"2024-03-04T07:37:54.122723+0000","last_became_peered":"2024-03-04T07:37:54.122723+0000","last_unstale":"2024-03-04T07:38:58.809061+0000","last_undegraded":"2024-03-04T07:38:58.809061+0000","last_fullsized":"2024-03-04T07:38:58.809061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:06:30.845376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488470+0000","last_change":"2024-03-04T07:37:55.115387+0000","last_active":"2024-03-04T07:39:27.488470+0000","last_peered":"2024-03-04T07:39:27.488470+0000","last_clean":"2024-03-04T07:39:27.488470+0000","last_became_active":"2024-03-04T07:37:55.114947+0000","last_became_peered":"2024-03-04T07:37:55.114947+0000","last_unstale":"2024-03-04T07:39:27.488470+0000","last_undegraded":"2024-03-04T07:39:27.488470+0000","last_fullsized":"2024-03-04T07:39:27.488470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:09:11.840989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":361,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:40:27.069018+0000","last_change":"2024-03-04T07:37:54.117767+0000","last_active":"2024-03-04T07:40:27.069018+0000","last_peered":"2024-03-04T07:40:27.069018+0000","last_clean":"2024-03-04T07:40:27.069018+0000","last_became_active":"2024-03-04T07:37:54.117371+0000","last_became_peered":"2024-03-04T07:37:54.117371+0000","last_unstale":"2024-03-04T07:40:27.069018+0000","last_undegraded":"2024-03-04T07:40:27.069018+0000","last_fullsized":"2024-03-04T07:40:27.069018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:08:32.164636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61605,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":310,"num_read_kb":215,"num_write":31,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488421+0000","last_change":"2024-03-04T07:37:55.112707+0000","last_active":"2024-03-04T07:39:27.488421+0000","last_peered":"2024-03-04T07:39:27.488421+0000","last_clean":"2024-03-04T07:39:27.488421+0000","last_became_active":"2024-03-04T07:37:55.112327+0000","last_became_peered":"2024-03-04T07:37:55.112327+0000","last_unstale":"2024-03-04T07:39:27.488421+0000","last_undegraded":"2024-03-04T07:39:27.488421+0000","last_fullsized":"2024-03-04T07:39:27.488421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:03.901687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811773+0000","last_change":"2024-03-04T07:37:54.125286+0000","last_active":"2024-03-04T07:38:58.811773+0000","last_peered":"2024-03-04T07:38:58.811773+0000","last_clean":"2024-03-04T07:38:58.811773+0000","last_became_active":"2024-03-04T07:37:54.124966+0000","last_became_peered":"2024-03-04T07:37:54.124966+0000","last_unstale":"2024-03-04T07:38:58.811773+0000","last_undegraded":"2024-03-04T07:38:58.811773+0000","last_fullsized":"2024-03-04T07:38:58.811773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:19.880681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.524159+0000","last_change":"2024-03-04T07:37:55.137952+0000","last_active":"2024-03-04T07:39:27.524159+0000","last_peered":"2024-03-04T07:39:27.524159+0000","last_clean":"2024-03-04T07:39:27.524159+0000","last_became_active":"2024-03-04T07:37:55.137583+0000","last_became_peered":"2024-03-04T07:37:55.137583+0000","last_unstale":"2024-03-04T07:39:27.524159+0000","last_undegraded":"2024-03-04T07:39:27.524159+0000","last_fullsized":"2024-03-04T07:39:27.524159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:30.145453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807594+0000","last_change":"2024-03-04T07:37:54.134733+0000","last_active":"2024-03-04T07:38:58.807594+0000","last_peered":"2024-03-04T07:38:58.807594+0000","last_clean":"2024-03-04T07:38:58.807594+0000","last_became_active":"2024-03-04T07:37:54.134400+0000","last_became_peered":"2024-03-04T07:37:54.134400+0000","last_unstale":"2024-03-04T07:38:58.807594+0000","last_undegraded":"2024-03-04T07:38:58.807594+0000","last_fullsized":"2024-03-04T07:38:58.807594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:24.192479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511483+0000","last_change":"2024-03-04T07:37:55.124991+0000","last_active":"2024-03-04T07:39:27.511483+0000","last_peered":"2024-03-04T07:39:27.511483+0000","last_clean":"2024-03-04T07:39:27.511483+0000","last_became_active":"2024-03-04T07:37:55.124621+0000","last_became_peered":"2024-03-04T07:37:55.124621+0000","last_unstale":"2024-03-04T07:39:27.511483+0000","last_undegraded":"2024-03-04T07:39:27.511483+0000","last_fullsized":"2024-03-04T07:39:27.511483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:05.252145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021001+0000","last_change":"2024-03-04T07:37:54.118743+0000","last_active":"2024-03-04T07:38:59.021001+0000","last_peered":"2024-03-04T07:38:59.021001+0000","last_clean":"2024-03-04T07:38:59.021001+0000","last_became_active":"2024-03-04T07:37:54.118386+0000","last_became_peered":"2024-03-04T07:37:54.118386+0000","last_unstale":"2024-03-04T07:38:59.021001+0000","last_undegraded":"2024-03-04T07:38:59.021001+0000","last_fullsized":"2024-03-04T07:38:59.021001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:27.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.473563+0000","last_change":"2024-03-04T07:37:55.115431+0000","last_active":"2024-03-04T07:39:27.473563+0000","last_peered":"2024-03-04T07:39:27.473563+0000","last_clean":"2024-03-04T07:39:27.473563+0000","last_became_active":"2024-03-04T07:37:55.115064+0000","last_became_peered":"2024-03-04T07:37:55.115064+0000","last_unstale":"2024-03-04T07:39:27.473563+0000","last_undegraded":"2024-03-04T07:39:27.473563+0000","last_fullsized":"2024-03-04T07:39:27.473563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:46:57.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.029383+0000","last_change":"2024-03-04T07:38:59.028943+0000","last_active":"2024-03-04T07:38:59.029383+0000","last_peered":"2024-03-04T07:38:59.029383+0000","last_clean":"2024-03-04T07:38:59.029383+0000","last_became_active":"2024-03-04T07:38:59.022232+0000","last_became_peered":"2024-03-04T07:38:59.022232+0000","last_unstale":"2024-03-04T07:38:59.029383+0000","last_undegraded":"2024-03-04T07:38:59.029383+0000","last_fullsized":"2024-03-04T07:38:59.029383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:49.281601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.485427+0000","last_change":"2024-03-04T07:37:55.114882+0000","last_active":"2024-03-04T07:39:27.485427+0000","last_peered":"2024-03-04T07:39:27.485427+0000","last_clean":"2024-03-04T07:39:27.485427+0000","last_became_active":"2024-03-04T07:37:55.113139+0000","last_became_peered":"2024-03-04T07:37:55.113139+0000","last_unstale":"2024-03-04T07:39:27.485427+0000","last_undegraded":"2024-03-04T07:39:27.485427+0000","last_fullsized":"2024-03-04T07:39:27.485427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:28.133710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808775+0000","last_change":"2024-03-04T07:37:54.119350+0000","last_active":"2024-03-04T07:38:58.808775+0000","last_peered":"2024-03-04T07:38:58.808775+0000","last_clean":"2024-03-04T07:38:58.808775+0000","last_became_active":"2024-03-04T07:37:54.119017+0000","last_became_peered":"2024-03-04T07:37:54.119017+0000","last_unstale":"2024-03-04T07:38:58.808775+0000","last_undegraded":"2024-03-04T07:38:58.808775+0000","last_fullsized":"2024-03-04T07:38:58.808775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:00.503583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.490355+0000","last_change":"2024-03-04T07:38:58.812598+0000","last_active":"2024-03-04T07:39:27.490355+0000","last_peered":"2024-03-04T07:39:27.490355+0000","last_clean":"2024-03-04T07:39:27.490355+0000","last_became_active":"2024-03-04T07:38:58.812187+0000","last_became_peered":"2024-03-04T07:38:58.812187+0000","last_unstale":"2024-03-04T07:39:27.490355+0000","last_undegraded":"2024-03-04T07:39:27.490355+0000","last_fullsized":"2024-03-04T07:39:27.490355+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:43.753007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804447+0000","last_change":"2024-03-04T07:37:54.125548+0000","last_active":"2024-03-04T07:38:58.804447+0000","last_peered":"2024-03-04T07:38:58.804447+0000","last_clean":"2024-03-04T07:38:58.804447+0000","last_became_active":"2024-03-04T07:37:54.125127+0000","last_became_peered":"2024-03-04T07:37:54.125127+0000","last_unstale":"2024-03-04T07:38:58.804447+0000","last_undegraded":"2024-03-04T07:38:58.804447+0000","last_fullsized":"2024-03-04T07:38:58.804447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:26.256247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.506689+0000","last_change":"2024-03-04T07:37:55.128542+0000","last_active":"2024-03-04T07:39:27.506689+0000","last_peered":"2024-03-04T07:39:27.506689+0000","last_clean":"2024-03-04T07:39:27.506689+0000","last_became_active":"2024-03-04T07:37:55.128239+0000","last_became_peered":"2024-03-04T07:37:55.128239+0000","last_unstale":"2024-03-04T07:39:27.506689+0000","last_undegraded":"2024-03-04T07:39:27.506689+0000","last_fullsized":"2024-03-04T07:39:27.506689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:17.078095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807386+0000","last_change":"2024-03-04T07:37:54.133672+0000","last_active":"2024-03-04T07:38:58.807386+0000","last_peered":"2024-03-04T07:38:58.807386+0000","last_clean":"2024-03-04T07:38:58.807386+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807386+0000","last_undegraded":"2024-03-04T07:38:58.807386+0000","last_fullsized":"2024-03-04T07:38:58.807386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:52.345158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501983+0000","last_change":"2024-03-04T07:38:58.815766+0000","last_active":"2024-03-04T07:39:27.501983+0000","last_peered":"2024-03-04T07:39:27.501983+0000","last_clean":"2024-03-04T07:39:27.501983+0000","last_became_active":"2024-03-04T07:38:58.815387+0000","last_became_peered":"2024-03-04T07:38:58.815387+0000","last_unstale":"2024-03-04T07:39:27.501983+0000","last_undegraded":"2024-03-04T07:39:27.501983+0000","last_fullsized":"2024-03-04T07:39:27.501983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:21.582879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488110+0000","last_change":"2024-03-04T07:38:58.814581+0000","last_active":"2024-03-04T07:39:27.488110+0000","last_peered":"2024-03-04T07:39:27.488110+0000","last_clean":"2024-03-04T07:39:27.488110+0000","last_became_active":"2024-03-04T07:38:58.814186+0000","last_became_peered":"2024-03-04T07:38:58.814186+0000","last_unstale":"2024-03-04T07:39:27.488110+0000","last_undegraded":"2024-03-04T07:39:27.488110+0000","last_fullsized":"2024-03-04T07:39:27.488110+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:03:33.824292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803774+0000","last_change":"2024-03-04T07:37:54.139335+0000","last_active":"2024-03-04T07:38:58.803774+0000","last_peered":"2024-03-04T07:38:58.803774+0000","last_clean":"2024-03-04T07:38:58.803774+0000","last_became_active":"2024-03-04T07:37:54.138969+0000","last_became_peered":"2024-03-04T07:37:54.138969+0000","last_unstale":"2024-03-04T07:38:58.803774+0000","last_undegraded":"2024-03-04T07:38:58.803774+0000","last_fullsized":"2024-03-04T07:38:58.803774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:41:15.204738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523815+0000","last_change":"2024-03-04T07:38:58.810554+0000","last_active":"2024-03-04T07:39:27.523815+0000","last_peered":"2024-03-04T07:39:27.523815+0000","last_clean":"2024-03-04T07:39:27.523815+0000","last_became_active":"2024-03-04T07:38:58.809978+0000","last_became_peered":"2024-03-04T07:38:58.809978+0000","last_unstale":"2024-03-04T07:39:27.523815+0000","last_undegraded":"2024-03-04T07:39:27.523815+0000","last_fullsized":"2024-03-04T07:39:27.523815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:33.417630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021011+0000","last_change":"2024-03-04T07:37:54.119906+0000","last_active":"2024-03-04T07:38:59.021011+0000","last_peered":"2024-03-04T07:38:59.021011+0000","last_clean":"2024-03-04T07:38:59.021011+0000","last_became_active":"2024-03-04T07:37:54.119592+0000","last_became_peered":"2024-03-04T07:37:54.119592+0000","last_unstale":"2024-03-04T07:38:59.021011+0000","last_undegraded":"2024-03-04T07:38:59.021011+0000","last_fullsized":"2024-03-04T07:38:59.021011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:29.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":343,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807128+0000","last_change":"2024-03-04T07:36:35.278528+0000","last_active":"2024-03-04T07:38:58.807128+0000","last_peered":"2024-03-04T07:38:58.807128+0000","last_clean":"2024-03-04T07:38:58.807128+0000","last_became_active":"2024-03-04T07:36:35.277953+0000","last_became_peered":"2024-03-04T07:36:35.277953+0000","last_unstale":"2024-03-04T07:38:58.807128+0000","last_undegraded":"2024-03-04T07:38:58.807128+0000","last_fullsized":"2024-03-04T07:38:58.807128+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_clean_scrub_stamp":"2024-03-04T07:32:42.283525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:58:47.223527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.496346+0000","last_change":"2024-03-04T07:37:55.124204+0000","last_active":"2024-03-04T07:39:27.496346+0000","last_peered":"2024-03-04T07:39:27.496346+0000","last_clean":"2024-03-04T07:39:27.496346+0000","last_became_active":"2024-03-04T07:37:55.123863+0000","last_became_peered":"2024-03-04T07:37:55.123863+0000","last_unstale":"2024-03-04T07:39:27.496346+0000","last_undegraded":"2024-03-04T07:39:27.496346+0000","last_fullsized":"2024-03-04T07:39:27.496346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:33.830367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807256+0000","last_change":"2024-03-04T07:37:54.121538+0000","last_active":"2024-03-04T07:38:58.807256+0000","last_peered":"2024-03-04T07:38:58.807256+0000","last_clean":"2024-03-04T07:38:58.807256+0000","last_became_active":"2024-03-04T07:37:54.120939+0000","last_became_peered":"2024-03-04T07:37:54.120939+0000","last_unstale":"2024-03-04T07:38:58.807256+0000","last_undegraded":"2024-03-04T07:38:58.807256+0000","last_fullsized":"2024-03-04T07:38:58.807256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:29.597247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808527+0000","last_change":"2024-03-04T07:37:54.128180+0000","last_active":"2024-03-04T07:38:58.808527+0000","last_peered":"2024-03-04T07:38:58.808527+0000","last_clean":"2024-03-04T07:38:58.808527+0000","last_became_active":"2024-03-04T07:37:54.127800+0000","last_became_peered":"2024-03-04T07:37:54.127800+0000","last_unstale":"2024-03-04T07:38:58.808527+0000","last_undegraded":"2024-03-04T07:38:58.808527+0000","last_fullsized":"2024-03-04T07:38:58.808527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:54.046873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504475+0000","last_change":"2024-03-04T07:37:55.133482+0000","last_active":"2024-03-04T07:39:27.504475+0000","last_peered":"2024-03-04T07:39:27.504475+0000","last_clean":"2024-03-04T07:39:27.504475+0000","last_became_active":"2024-03-04T07:37:55.133137+0000","last_became_peered":"2024-03-04T07:37:55.133137+0000","last_unstale":"2024-03-04T07:39:27.504475+0000","last_undegraded":"2024-03-04T07:39:27.504475+0000","last_fullsized":"2024-03-04T07:39:27.504475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:53.509564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:40:04.733943+0000","last_change":"2024-03-04T07:37:54.134294+0000","last_active":"2024-03-04T07:40:04.733943+0000","last_peered":"2024-03-04T07:40:04.733943+0000","last_clean":"2024-03-04T07:40:04.733943+0000","last_became_active":"2024-03-04T07:37:54.133677+0000","last_became_peered":"2024-03-04T07:37:54.133677+0000","last_unstale":"2024-03-04T07:40:04.733943+0000","last_undegraded":"2024-03-04T07:40:04.733943+0000","last_fullsized":"2024-03-04T07:40:04.733943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:57:54.082419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522888+0000","last_change":"2024-03-04T07:37:55.138305+0000","last_active":"2024-03-04T07:39:27.522888+0000","last_peered":"2024-03-04T07:39:27.522888+0000","last_clean":"2024-03-04T07:39:27.522888+0000","last_became_active":"2024-03-04T07:37:55.129937+0000","last_became_peered":"2024-03-04T07:37:55.129937+0000","last_unstale":"2024-03-04T07:39:27.522888+0000","last_undegraded":"2024-03-04T07:39:27.522888+0000","last_fullsized":"2024-03-04T07:39:27.522888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:45.136539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021306+0000","last_change":"2024-03-04T07:37:54.133817+0000","last_active":"2024-03-04T07:38:59.021306+0000","last_peered":"2024-03-04T07:38:59.021306+0000","last_clean":"2024-03-04T07:38:59.021306+0000","last_became_active":"2024-03-04T07:37:54.133471+0000","last_became_peered":"2024-03-04T07:37:54.133471+0000","last_unstale":"2024-03-04T07:38:59.021306+0000","last_undegraded":"2024-03-04T07:38:59.021306+0000","last_fullsized":"2024-03-04T07:38:59.021306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:00:51.483859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.517445+0000","last_change":"2024-03-04T07:37:55.144639+0000","last_active":"2024-03-04T07:39:27.517445+0000","last_peered":"2024-03-04T07:39:27.517445+0000","last_clean":"2024-03-04T07:39:27.517445+0000","last_became_active":"2024-03-04T07:37:55.144194+0000","last_became_peered":"2024-03-04T07:37:55.144194+0000","last_unstale":"2024-03-04T07:39:27.517445+0000","last_undegraded":"2024-03-04T07:39:27.517445+0000","last_fullsized":"2024-03-04T07:39:27.517445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:42.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.816116+0000","last_change":"2024-03-04T07:38:58.815642+0000","last_active":"2024-03-04T07:38:58.816116+0000","last_peered":"2024-03-04T07:38:58.816116+0000","last_clean":"2024-03-04T07:38:58.816116+0000","last_became_active":"2024-03-04T07:38:58.815361+0000","last_became_peered":"2024-03-04T07:38:58.815361+0000","last_unstale":"2024-03-04T07:38:58.816116+0000","last_undegraded":"2024-03-04T07:38:58.816116+0000","last_fullsized":"2024-03-04T07:38:58.816116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:48:15.751929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487964+0000","last_change":"2024-03-04T07:37:55.115145+0000","last_active":"2024-03-04T07:39:27.487964+0000","last_peered":"2024-03-04T07:39:27.487964+0000","last_clean":"2024-03-04T07:39:27.487964+0000","last_became_active":"2024-03-04T07:37:55.114791+0000","last_became_peered":"2024-03-04T07:37:55.114791+0000","last_unstale":"2024-03-04T07:39:27.487964+0000","last_undegraded":"2024-03-04T07:39:27.487964+0000","last_fullsized":"2024-03-04T07:39:27.487964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:19.685977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.806430+0000","last_change":"2024-03-04T07:37:54.141630+0000","last_active":"2024-03-04T07:38:58.806430+0000","last_peered":"2024-03-04T07:38:58.806430+0000","last_clean":"2024-03-04T07:38:58.806430+0000","last_became_active":"2024-03-04T07:37:54.139820+0000","last_became_peered":"2024-03-04T07:37:54.139820+0000","last_unstale":"2024-03-04T07:38:58.806430+0000","last_undegraded":"2024-03-04T07:38:58.806430+0000","last_fullsized":"2024-03-04T07:38:58.806430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:22.419437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503153+0000","last_change":"2024-03-04T07:37:55.133270+0000","last_active":"2024-03-04T07:39:27.503153+0000","last_peered":"2024-03-04T07:39:27.503153+0000","last_clean":"2024-03-04T07:39:27.503153+0000","last_became_active":"2024-03-04T07:37:55.132909+0000","last_became_peered":"2024-03-04T07:37:55.132909+0000","last_unstale":"2024-03-04T07:39:27.503153+0000","last_undegraded":"2024-03-04T07:39:27.503153+0000","last_fullsized":"2024-03-04T07:39:27.503153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:21.026141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812572+0000","last_change":"2024-03-04T07:37:54.135988+0000","last_active":"2024-03-04T07:38:58.812572+0000","last_peered":"2024-03-04T07:38:58.812572+0000","last_clean":"2024-03-04T07:38:58.812572+0000","last_became_active":"2024-03-04T07:37:54.135627+0000","last_became_peered":"2024-03-04T07:37:54.135627+0000","last_unstale":"2024-03-04T07:38:58.812572+0000","last_undegraded":"2024-03-04T07:38:58.812572+0000","last_fullsized":"2024-03-04T07:38:58.812572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:06.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522153+0000","last_change":"2024-03-04T07:37:55.144551+0000","last_active":"2024-03-04T07:39:27.522153+0000","last_peered":"2024-03-04T07:39:27.522153+0000","last_clean":"2024-03-04T07:39:27.522153+0000","last_became_active":"2024-03-04T07:37:55.143882+0000","last_became_peered":"2024-03-04T07:37:55.143882+0000","last_unstale":"2024-03-04T07:39:27.522153+0000","last_undegraded":"2024-03-04T07:39:27.522153+0000","last_fullsized":"2024-03-04T07:39:27.522153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:07.254941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.817917+0000","last_change":"2024-03-04T07:38:58.817440+0000","last_active":"2024-03-04T07:38:58.817917+0000","last_peered":"2024-03-04T07:38:58.817917+0000","last_clean":"2024-03-04T07:38:58.817917+0000","last_became_active":"2024-03-04T07:38:58.813620+0000","last_became_peered":"2024-03-04T07:38:58.813620+0000","last_unstale":"2024-03-04T07:38:58.817917+0000","last_undegraded":"2024-03-04T07:38:58.817917+0000","last_fullsized":"2024-03-04T07:38:58.817917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:10:41.554608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.505218+0000","last_change":"2024-03-04T07:37:55.133870+0000","last_active":"2024-03-04T07:39:27.505218+0000","last_peered":"2024-03-04T07:39:27.505218+0000","last_clean":"2024-03-04T07:39:27.505218+0000","last_became_active":"2024-03-04T07:37:55.133542+0000","last_became_peered":"2024-03-04T07:37:55.133542+0000","last_unstale":"2024-03-04T07:39:27.505218+0000","last_undegraded":"2024-03-04T07:39:27.505218+0000","last_fullsized":"2024-03-04T07:39:27.505218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:17:44.373426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811749+0000","last_change":"2024-03-04T07:37:54.126326+0000","last_active":"2024-03-04T07:38:58.811749+0000","last_peered":"2024-03-04T07:38:58.811749+0000","last_clean":"2024-03-04T07:38:58.811749+0000","last_became_active":"2024-03-04T07:37:54.125989+0000","last_became_peered":"2024-03-04T07:37:54.125989+0000","last_unstale":"2024-03-04T07:38:58.811749+0000","last_undegraded":"2024-03-04T07:38:58.811749+0000","last_fullsized":"2024-03-04T07:38:58.811749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:41.108528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488757+0000","last_change":"2024-03-04T07:37:55.120052+0000","last_active":"2024-03-04T07:39:27.488757+0000","last_peered":"2024-03-04T07:39:27.488757+0000","last_clean":"2024-03-04T07:39:27.488757+0000","last_became_active":"2024-03-04T07:37:55.119708+0000","last_became_peered":"2024-03-04T07:37:55.119708+0000","last_unstale":"2024-03-04T07:39:27.488757+0000","last_undegraded":"2024-03-04T07:39:27.488757+0000","last_fullsized":"2024-03-04T07:39:27.488757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:25:17.679416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808512+0000","last_change":"2024-03-04T07:37:54.134018+0000","last_active":"2024-03-04T07:38:58.808512+0000","last_peered":"2024-03-04T07:38:58.808512+0000","last_clean":"2024-03-04T07:38:58.808512+0000","last_became_active":"2024-03-04T07:37:54.133673+0000","last_became_peered":"2024-03-04T07:37:54.133673+0000","last_unstale":"2024-03-04T07:38:58.808512+0000","last_undegraded":"2024-03-04T07:38:58.808512+0000","last_fullsized":"2024-03-04T07:38:58.808512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:49.438338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489514+0000","last_change":"2024-03-04T07:38:58.817071+0000","last_active":"2024-03-04T07:39:27.489514+0000","last_peered":"2024-03-04T07:39:27.489514+0000","last_clean":"2024-03-04T07:39:27.489514+0000","last_became_active":"2024-03-04T07:38:58.812139+0000","last_became_peered":"2024-03-04T07:38:58.812139+0000","last_unstale":"2024-03-04T07:39:27.489514+0000","last_undegraded":"2024-03-04T07:39:27.489514+0000","last_fullsized":"2024-03-04T07:39:27.489514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:37.557255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805106+0000","last_change":"2024-03-04T07:37:54.140806+0000","last_active":"2024-03-04T07:38:58.805106+0000","last_peered":"2024-03-04T07:38:58.805106+0000","last_clean":"2024-03-04T07:38:58.805106+0000","last_became_active":"2024-03-04T07:37:54.140419+0000","last_became_peered":"2024-03-04T07:37:54.140419+0000","last_unstale":"2024-03-04T07:38:58.805106+0000","last_undegraded":"2024-03-04T07:38:58.805106+0000","last_fullsized":"2024-03-04T07:38:58.805106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:13:59.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489032+0000","last_change":"2024-03-04T07:37:55.113658+0000","last_active":"2024-03-04T07:39:27.489032+0000","last_peered":"2024-03-04T07:39:27.489032+0000","last_clean":"2024-03-04T07:39:27.489032+0000","last_became_active":"2024-03-04T07:37:55.113300+0000","last_became_peered":"2024-03-04T07:37:55.113300+0000","last_unstale":"2024-03-04T07:39:27.489032+0000","last_undegraded":"2024-03-04T07:39:27.489032+0000","last_fullsized":"2024-03-04T07:39:27.489032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:43:56.907387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487646+0000","last_change":"2024-03-04T07:37:55.117769+0000","last_active":"2024-03-04T07:39:27.487646+0000","last_peered":"2024-03-04T07:39:27.487646+0000","last_clean":"2024-03-04T07:39:27.487646+0000","last_became_active":"2024-03-04T07:37:55.117408+0000","last_became_peered":"2024-03-04T07:37:55.117408+0000","last_unstale":"2024-03-04T07:39:27.487646+0000","last_undegraded":"2024-03-04T07:39:27.487646+0000","last_fullsized":"2024-03-04T07:39:27.487646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:22.143108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.483985+0000","last_change":"2024-03-04T07:37:55.115581+0000","last_active":"2024-03-04T07:39:27.483985+0000","last_peered":"2024-03-04T07:39:27.483985+0000","last_clean":"2024-03-04T07:39:27.483985+0000","last_became_active":"2024-03-04T07:37:55.113387+0000","last_became_peered":"2024-03-04T07:37:55.113387+0000","last_unstale":"2024-03-04T07:39:27.483985+0000","last_undegraded":"2024-03-04T07:39:27.483985+0000","last_fullsized":"2024-03-04T07:39:27.483985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:59.273650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.492691+0000","last_change":"2024-03-04T07:37:55.135175+0000","last_active":"2024-03-04T07:39:27.492691+0000","last_peered":"2024-03-04T07:39:27.492691+0000","last_clean":"2024-03-04T07:39:27.492691+0000","last_became_active":"2024-03-04T07:37:55.134787+0000","last_became_peered":"2024-03-04T07:37:55.134787+0000","last_unstale":"2024-03-04T07:39:27.492691+0000","last_undegraded":"2024-03-04T07:39:27.492691+0000","last_fullsized":"2024-03-04T07:39:27.492691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:04.227886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511558+0000","last_change":"2024-03-04T07:37:55.138146+0000","last_active":"2024-03-04T07:39:27.511558+0000","last_peered":"2024-03-04T07:39:27.511558+0000","last_clean":"2024-03-04T07:39:27.511558+0000","last_became_active":"2024-03-04T07:37:55.137806+0000","last_became_peered":"2024-03-04T07:37:55.137806+0000","last_unstale":"2024-03-04T07:39:27.511558+0000","last_undegraded":"2024-03-04T07:39:27.511558+0000","last_fullsized":"2024-03-04T07:39:27.511558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:10.596224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.507386+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.507386+0000","last_peered":"2024-03-04T07:39:27.507386+0000","last_clean":"2024-03-04T07:39:27.507386+0000","last_became_active":"2024-03-04T07:37:55.140670+0000","last_became_peered":"2024-03-04T07:37:55.140670+0000","last_unstale":"2024-03-04T07:39:27.507386+0000","last_undegraded":"2024-03-04T07:39:27.507386+0000","last_fullsized":"2024-03-04T07:39:27.507386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:29:54.439417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487788+0000","last_change":"2024-03-04T07:37:55.115015+0000","last_active":"2024-03-04T07:39:27.487788+0000","last_peered":"2024-03-04T07:39:27.487788+0000","last_clean":"2024-03-04T07:39:27.487788+0000","last_became_active":"2024-03-04T07:37:55.114668+0000","last_became_peered":"2024-03-04T07:37:55.114668+0000","last_unstale":"2024-03-04T07:39:27.487788+0000","last_undegraded":"2024-03-04T07:39:27.487788+0000","last_fullsized":"2024-03-04T07:39:27.487788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:29:03.309234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.482820+0000","last_change":"2024-03-04T07:37:55.119514+0000","last_active":"2024-03-04T07:39:27.482820+0000","last_peered":"2024-03-04T07:39:27.482820+0000","last_clean":"2024-03-04T07:39:27.482820+0000","last_became_active":"2024-03-04T07:37:55.119166+0000","last_became_peered":"2024-03-04T07:37:55.119166+0000","last_unstale":"2024-03-04T07:39:27.482820+0000","last_undegraded":"2024-03-04T07:39:27.482820+0000","last_fullsized":"2024-03-04T07:39:27.482820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:33.468311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.520592+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.520592+0000","last_peered":"2024-03-04T07:39:27.520592+0000","last_clean":"2024-03-04T07:39:27.520592+0000","last_became_active":"2024-03-04T07:37:55.140723+0000","last_became_peered":"2024-03-04T07:37:55.140723+0000","last_unstale":"2024-03-04T07:39:27.520592+0000","last_undegraded":"2024-03-04T07:39:27.520592+0000","last_fullsized":"2024-03-04T07:39:27.520592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:29.751190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.510302+0000","last_change":"2024-03-04T07:37:55.127627+0000","last_active":"2024-03-04T07:39:27.510302+0000","last_peered":"2024-03-04T07:39:27.510302+0000","last_clean":"2024-03-04T07:39:27.510302+0000","last_became_active":"2024-03-04T07:37:55.127217+0000","last_became_peered":"2024-03-04T07:37:55.127217+0000","last_unstale":"2024-03-04T07:39:27.510302+0000","last_undegraded":"2024-03-04T07:39:27.510302+0000","last_fullsized":"2024-03-04T07:39:27.510302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:47.826970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519782+0000","last_change":"2024-03-04T07:37:55.129445+0000","last_active":"2024-03-04T07:39:27.519782+0000","last_peered":"2024-03-04T07:39:27.519782+0000","last_clean":"2024-03-04T07:39:27.519782+0000","last_became_active":"2024-03-04T07:37:55.129013+0000","last_became_peered":"2024-03-04T07:37:55.129013+0000","last_unstale":"2024-03-04T07:39:27.519782+0000","last_undegraded":"2024-03-04T07:39:27.519782+0000","last_fullsized":"2024-03-04T07:39:27.519782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:15.578436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523619+0000","last_change":"2024-03-04T07:37:55.126397+0000","last_active":"2024-03-04T07:39:27.523619+0000","last_peered":"2024-03-04T07:39:27.523619+0000","last_clean":"2024-03-04T07:39:27.523619+0000","last_became_active":"2024-03-04T07:37:55.126092+0000","last_became_peered":"2024-03-04T07:37:55.126092+0000","last_unstale":"2024-03-04T07:39:27.523619+0000","last_undegraded":"2024-03-04T07:39:27.523619+0000","last_fullsized":"2024-03-04T07:39:27.523619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:47.968250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511142+0000","last_change":"2024-03-04T07:37:55.140265+0000","last_active":"2024-03-04T07:39:27.511142+0000","last_peered":"2024-03-04T07:39:27.511142+0000","last_clean":"2024-03-04T07:39:27.511142+0000","last_became_active":"2024-03-04T07:37:55.139768+0000","last_became_peered":"2024-03-04T07:37:55.139768+0000","last_unstale":"2024-03-04T07:39:27.511142+0000","last_undegraded":"2024-03-04T07:39:27.511142+0000","last_fullsized":"2024-03-04T07:39:27.511142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:52.924188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488738+0000","last_change":"2024-03-04T07:37:55.118929+0000","last_active":"2024-03-04T07:39:27.488738+0000","last_peered":"2024-03-04T07:39:27.488738+0000","last_clean":"2024-03-04T07:39:27.488738+0000","last_became_active":"2024-03-04T07:37:55.118549+0000","last_became_peered":"2024-03-04T07:37:55.118549+0000","last_unstale":"2024-03-04T07:39:27.488738+0000","last_undegraded":"2024-03-04T07:39:27.488738+0000","last_fullsized":"2024-03-04T07:39:27.488738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:15.599080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244954,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62403,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":314,"num_read_kb":219,"num_write":84,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":187209,"data_compressed":5550,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":455726,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1108010,"data_compressed":43363,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456425,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456330,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332162,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1046793,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455843,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":517021,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455816,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":517365,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182489,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":455947,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920853,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455748,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":1046393,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61605,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61639,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61799,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T07:40:28.711 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-04T07:40:28.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T07:40:28.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 -- 172.21.15.116:0/1488008696 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e00ffe50 msgr2=0x7f27e0100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 --2- 172.21.15.116:0/1488008696 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e00ffe50 0x7f27e0100270 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009900 tx=0x7f27d402bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 -- 172.21.15.116:0/1488008696 shutdown_connections 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 --2- 172.21.15.116:0/1488008696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e0101150 0x7f27e01015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 --2- 172.21.15.116:0/1488008696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e01007b0 0x7f27e0100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 --2- 172.21.15.116:0/1488008696 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e00ffe50 0x7f27e0100270 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 -- 172.21.15.116:0/1488008696 >> 172.21.15.116:0/1488008696 conn(0x7f27e00fba50 msgr2=0x7f27e00fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 -- 172.21.15.116:0/1488008696 shutdown_connections 2024-03-04T07:40:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.945+0000 7f27e7e00640 1 -- 172.21.15.116:0/1488008696 wait complete. 2024-03-04T07:40:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.946+0000 7f27e7e00640 1 Processor -- start 2024-03-04T07:40:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.946+0000 7f27e7e00640 1 -- start start 2024-03-04T07:40:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e7e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e00ffe50 0x7f27e0199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e01007b0 0x7f27e019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e0101150 0x7f27e01a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e010f420 con 0x7f27e01007b0 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e7e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e010f2a0 con 0x7f27e0101150 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e7e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e010f5a0 con 0x7f27e00ffe50 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e00ffe50 0x7f27e0199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e01007b0 0x7f27e019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e0101150 0x7f27e01a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e00ffe50 0x7f27e0199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:35322/0 (socket says 172.21.15.116:35322) 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5400640 1 -- 172.21.15.116:0/745653709 learned_addr learned my addr 172.21.15.116:0/745653709 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5e00640 1 -- 172.21.15.116:0/745653709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e00ffe50 msgr2=0x7f27e0199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e00ffe50 0x7f27e0199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5e00640 1 -- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e01007b0 msgr2=0x7f27e019a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e01007b0 0x7f27e019a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.947+0000 7f27e5e00640 1 -- 172.21.15.116:0/745653709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e01a16e0 con 0x7f27e0101150 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.948+0000 7f27e5e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e0101150 0x7f27e01a1070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f27d800da10 tx=0x7f27d800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.948+0000 7f27de000640 1 -- 172.21.15.116:0/745653709 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d800bd70 con 0x7f27e0101150 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.948+0000 7f27de000640 1 -- 172.21.15.116:0/745653709 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27d8003e50 con 0x7f27e0101150 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.948+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e01a19d0 con 0x7f27e0101150 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.948+0000 7f27de000640 1 -- 172.21.15.116:0/745653709 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d80106a0 con 0x7f27e0101150 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.948+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e01a1ed0 con 0x7f27e0101150 2024-03-04T07:40:28.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.950+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e010ad90 con 0x7f27e0101150 2024-03-04T07:40:28.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.951+0000 7f27de000640 1 -- 172.21.15.116:0/745653709 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f27d8003970 con 0x7f27e0101150 2024-03-04T07:40:28.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.952+0000 7f27de000640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27c80a1d80 0x7f27c80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:28.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.953+0000 7f27de000640 1 -- 172.21.15.116:0/745653709 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f27d80ce1e0 con 0x7f27e0101150 2024-03-04T07:40:28.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.953+0000 7f27e5400640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27c80a1d80 0x7f27c80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:28.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.953+0000 7f27e5400640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27c80a1d80 0x7f27c80a4240 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f27d4000c00 tx=0x7f27d402c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:28.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:28.957+0000 7f27de000640 1 -- 172.21.15.116:0/745653709 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f27d809b5d0 con 0x7f27e0101150 2024-03-04T07:40:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:28 smithi116 ceph-mon[32150]: pgmap v392: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:28 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1548516700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:28 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:40:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:28 smithi152 ceph-mon[37154]: pgmap v392: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:28 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1548516700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:28 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:40:29.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:28 smithi133 ceph-mon[36505]: pgmap v392: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:28 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1548516700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:28 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:40:29.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.166+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 --> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27e0061a10 con 0x7f27c80a1d80 2024-03-04T07:40:29.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.177+0000 7f27de000640 1 -- 172.21.15.116:0/745653709 <== mgr.14152 v2:172.21.15.116:6800/142890918 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259026 (secure 0 0 0) 0x7f27e0061a10 con 0x7f27c80a1d80 2024-03-04T07:40:29.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:29.182 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-04T07:40:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27c80a1d80 msgr2=0x7f27c80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27c80a1d80 0x7f27c80a4240 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f27d4000c00 tx=0x7f27d402c880 comp rx=0 tx=0).stop 2024-03-04T07:40:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e0101150 msgr2=0x7f27e01a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e0101150 0x7f27e01a1070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f27d800da10 tx=0x7f27d800ded0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 shutdown_connections 2024-03-04T07:40:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27e0101150 0x7f27e01a1070 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e01007b0 0x7f27e019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f27c80a1d80 0x7f27c80a4240 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 --2- 172.21.15.116:0/745653709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e00ffe50 0x7f27e0199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 >> 172.21.15.116:0/745653709 conn(0x7f27e00fba50 msgr2=0x7f27e00fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 shutdown_connections 2024-03-04T07:40:29.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.181+0000 7f27e7e00640 1 -- 172.21.15.116:0/745653709 wait complete. 2024-03-04T07:40:29.200 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-03-04T07:40:27.534951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653005,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":440,"num_read_kb":328,"num_write":267,"num_write_kb":3029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":710,"ondisk_log_size":710,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360564,"kb_used_data":11412,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581260,"statfs":{"total":1151940427776,"available":1151571210240,"internally_reserved":0,"allocated":11685888,"data_stored":7427417,"data_compressed":182424,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":310,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004497"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509850+0000","last_change":"2024-03-04T07:37:55.137540+0000","last_active":"2024-03-04T07:39:27.509850+0000","last_peered":"2024-03-04T07:39:27.509850+0000","last_clean":"2024-03-04T07:39:27.509850+0000","last_became_active":"2024-03-04T07:37:55.137183+0000","last_became_peered":"2024-03-04T07:37:55.137183+0000","last_unstale":"2024-03-04T07:39:27.509850+0000","last_undegraded":"2024-03-04T07:39:27.509850+0000","last_fullsized":"2024-03-04T07:39:27.509850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:15.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503533+0000","last_change":"2024-03-04T07:37:55.136049+0000","last_active":"2024-03-04T07:39:27.503533+0000","last_peered":"2024-03-04T07:39:27.503533+0000","last_clean":"2024-03-04T07:39:27.503533+0000","last_became_active":"2024-03-04T07:37:55.132153+0000","last_became_peered":"2024-03-04T07:37:55.132153+0000","last_unstale":"2024-03-04T07:39:27.503533+0000","last_undegraded":"2024-03-04T07:39:27.503533+0000","last_fullsized":"2024-03-04T07:39:27.503533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:52.426892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.508509+0000","last_change":"2024-03-04T07:37:55.133571+0000","last_active":"2024-03-04T07:39:27.508509+0000","last_peered":"2024-03-04T07:39:27.508509+0000","last_clean":"2024-03-04T07:39:27.508509+0000","last_became_active":"2024-03-04T07:37:55.133214+0000","last_became_peered":"2024-03-04T07:37:55.133214+0000","last_unstale":"2024-03-04T07:39:27.508509+0000","last_undegraded":"2024-03-04T07:39:27.508509+0000","last_fullsized":"2024-03-04T07:39:27.508509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:17.924826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.480393+0000","last_change":"2024-03-04T07:37:55.118146+0000","last_active":"2024-03-04T07:39:27.480393+0000","last_peered":"2024-03-04T07:39:27.480393+0000","last_clean":"2024-03-04T07:39:27.480393+0000","last_became_active":"2024-03-04T07:37:55.117687+0000","last_became_peered":"2024-03-04T07:37:55.117687+0000","last_unstale":"2024-03-04T07:39:27.480393+0000","last_undegraded":"2024-03-04T07:39:27.480393+0000","last_fullsized":"2024-03-04T07:39:27.480393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:34:35.343400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.499265+0000","last_change":"2024-03-04T07:37:55.135943+0000","last_active":"2024-03-04T07:39:27.499265+0000","last_peered":"2024-03-04T07:39:27.499265+0000","last_clean":"2024-03-04T07:39:27.499265+0000","last_became_active":"2024-03-04T07:37:55.135549+0000","last_became_peered":"2024-03-04T07:37:55.135549+0000","last_unstale":"2024-03-04T07:39:27.499265+0000","last_undegraded":"2024-03-04T07:39:27.499265+0000","last_fullsized":"2024-03-04T07:39:27.499265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:35:53.953771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487475+0000","last_change":"2024-03-04T07:37:55.116949+0000","last_active":"2024-03-04T07:39:27.487475+0000","last_peered":"2024-03-04T07:39:27.487475+0000","last_clean":"2024-03-04T07:39:27.487475+0000","last_became_active":"2024-03-04T07:37:55.116620+0000","last_became_peered":"2024-03-04T07:37:55.116620+0000","last_unstale":"2024-03-04T07:39:27.487475+0000","last_undegraded":"2024-03-04T07:39:27.487475+0000","last_fullsized":"2024-03-04T07:39:27.487475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:02:39.655217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489760+0000","last_change":"2024-03-04T07:37:55.100975+0000","last_active":"2024-03-04T07:39:27.489760+0000","last_peered":"2024-03-04T07:39:27.489760+0000","last_clean":"2024-03-04T07:39:27.489760+0000","last_became_active":"2024-03-04T07:37:55.100636+0000","last_became_peered":"2024-03-04T07:37:55.100636+0000","last_unstale":"2024-03-04T07:39:27.489760+0000","last_undegraded":"2024-03-04T07:39:27.489760+0000","last_fullsized":"2024-03-04T07:39:27.489760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:29.593763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523017+0000","last_change":"2024-03-04T07:37:55.131895+0000","last_active":"2024-03-04T07:39:27.523017+0000","last_peered":"2024-03-04T07:39:27.523017+0000","last_clean":"2024-03-04T07:39:27.523017+0000","last_became_active":"2024-03-04T07:37:55.129682+0000","last_became_peered":"2024-03-04T07:37:55.129682+0000","last_unstale":"2024-03-04T07:39:27.523017+0000","last_undegraded":"2024-03-04T07:39:27.523017+0000","last_fullsized":"2024-03-04T07:39:27.523017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:41:54.179894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489551+0000","last_change":"2024-03-04T07:37:55.114475+0000","last_active":"2024-03-04T07:39:27.489551+0000","last_peered":"2024-03-04T07:39:27.489551+0000","last_clean":"2024-03-04T07:39:27.489551+0000","last_became_active":"2024-03-04T07:37:55.114068+0000","last_became_peered":"2024-03-04T07:37:55.114068+0000","last_unstale":"2024-03-04T07:39:27.489551+0000","last_undegraded":"2024-03-04T07:39:27.489551+0000","last_fullsized":"2024-03-04T07:39:27.489551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:21.466143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.477511+0000","last_change":"2024-03-04T07:37:55.115660+0000","last_active":"2024-03-04T07:39:27.477511+0000","last_peered":"2024-03-04T07:39:27.477511+0000","last_clean":"2024-03-04T07:39:27.477511+0000","last_became_active":"2024-03-04T07:37:55.115324+0000","last_became_peered":"2024-03-04T07:37:55.115324+0000","last_unstale":"2024-03-04T07:39:27.477511+0000","last_undegraded":"2024-03-04T07:39:27.477511+0000","last_fullsized":"2024-03-04T07:39:27.477511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:26:42.119728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489282+0000","last_change":"2024-03-04T07:37:55.119834+0000","last_active":"2024-03-04T07:39:27.489282+0000","last_peered":"2024-03-04T07:39:27.489282+0000","last_clean":"2024-03-04T07:39:27.489282+0000","last_became_active":"2024-03-04T07:37:55.119467+0000","last_became_peered":"2024-03-04T07:37:55.119467+0000","last_unstale":"2024-03-04T07:39:27.489282+0000","last_undegraded":"2024-03-04T07:39:27.489282+0000","last_fullsized":"2024-03-04T07:39:27.489282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:49:19.564508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.479644+0000","last_change":"2024-03-04T07:37:55.118154+0000","last_active":"2024-03-04T07:39:27.479644+0000","last_peered":"2024-03-04T07:39:27.479644+0000","last_clean":"2024-03-04T07:39:27.479644+0000","last_became_active":"2024-03-04T07:37:55.117809+0000","last_became_peered":"2024-03-04T07:37:55.117809+0000","last_unstale":"2024-03-04T07:39:27.479644+0000","last_undegraded":"2024-03-04T07:39:27.479644+0000","last_fullsized":"2024-03-04T07:39:27.479644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:36:50.195946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503528+0000","last_change":"2024-03-04T07:37:55.123325+0000","last_active":"2024-03-04T07:39:27.503528+0000","last_peered":"2024-03-04T07:39:27.503528+0000","last_clean":"2024-03-04T07:39:27.503528+0000","last_became_active":"2024-03-04T07:37:55.122848+0000","last_became_peered":"2024-03-04T07:37:55.122848+0000","last_unstale":"2024-03-04T07:39:27.503528+0000","last_undegraded":"2024-03-04T07:39:27.503528+0000","last_fullsized":"2024-03-04T07:39:27.503528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:18.842011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503051+0000","last_change":"2024-03-04T07:37:55.132447+0000","last_active":"2024-03-04T07:39:27.503051+0000","last_peered":"2024-03-04T07:39:27.503051+0000","last_clean":"2024-03-04T07:39:27.503051+0000","last_became_active":"2024-03-04T07:37:55.131512+0000","last_became_peered":"2024-03-04T07:37:55.131512+0000","last_unstale":"2024-03-04T07:39:27.503051+0000","last_undegraded":"2024-03-04T07:39:27.503051+0000","last_fullsized":"2024-03-04T07:39:27.503051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:30:23.960029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504478+0000","last_change":"2024-03-04T07:37:55.131715+0000","last_active":"2024-03-04T07:39:27.504478+0000","last_peered":"2024-03-04T07:39:27.504478+0000","last_clean":"2024-03-04T07:39:27.504478+0000","last_became_active":"2024-03-04T07:37:55.131326+0000","last_became_peered":"2024-03-04T07:37:55.131326+0000","last_unstale":"2024-03-04T07:39:27.504478+0000","last_undegraded":"2024-03-04T07:39:27.504478+0000","last_fullsized":"2024-03-04T07:39:27.504478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:56.231578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502604+0000","last_change":"2024-03-04T07:37:55.137395+0000","last_active":"2024-03-04T07:39:27.502604+0000","last_peered":"2024-03-04T07:39:27.502604+0000","last_clean":"2024-03-04T07:39:27.502604+0000","last_became_active":"2024-03-04T07:37:55.137054+0000","last_became_peered":"2024-03-04T07:37:55.137054+0000","last_unstale":"2024-03-04T07:39:27.502604+0000","last_undegraded":"2024-03-04T07:39:27.502604+0000","last_fullsized":"2024-03-04T07:39:27.502604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:19.222481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509061+0000","last_change":"2024-03-04T07:37:55.137414+0000","last_active":"2024-03-04T07:39:27.509061+0000","last_peered":"2024-03-04T07:39:27.509061+0000","last_clean":"2024-03-04T07:39:27.509061+0000","last_became_active":"2024-03-04T07:37:55.137157+0000","last_became_peered":"2024-03-04T07:37:55.137157+0000","last_unstale":"2024-03-04T07:39:27.509061+0000","last_undegraded":"2024-03-04T07:39:27.509061+0000","last_fullsized":"2024-03-04T07:39:27.509061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:57:36.593491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.502012+0000","last_change":"2024-03-04T07:37:55.121960+0000","last_active":"2024-03-04T07:39:27.502012+0000","last_peered":"2024-03-04T07:39:27.502012+0000","last_clean":"2024-03-04T07:39:27.502012+0000","last_became_active":"2024-03-04T07:37:55.115543+0000","last_became_peered":"2024-03-04T07:37:55.115543+0000","last_unstale":"2024-03-04T07:39:27.502012+0000","last_undegraded":"2024-03-04T07:39:27.502012+0000","last_fullsized":"2024-03-04T07:39:27.502012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:46:36.320475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.506619+0000","last_change":"2024-03-04T07:37:55.128908+0000","last_active":"2024-03-04T07:39:27.506619+0000","last_peered":"2024-03-04T07:39:27.506619+0000","last_clean":"2024-03-04T07:39:27.506619+0000","last_became_active":"2024-03-04T07:37:55.128500+0000","last_became_peered":"2024-03-04T07:37:55.128500+0000","last_unstale":"2024-03-04T07:39:27.506619+0000","last_undegraded":"2024-03-04T07:39:27.506619+0000","last_fullsized":"2024-03-04T07:39:27.506619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:10.598424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805031+0000","last_change":"2024-03-04T07:37:54.121610+0000","last_active":"2024-03-04T07:38:58.805031+0000","last_peered":"2024-03-04T07:38:58.805031+0000","last_clean":"2024-03-04T07:38:58.805031+0000","last_became_active":"2024-03-04T07:37:54.121150+0000","last_became_peered":"2024-03-04T07:37:54.121150+0000","last_unstale":"2024-03-04T07:38:58.805031+0000","last_undegraded":"2024-03-04T07:38:58.805031+0000","last_fullsized":"2024-03-04T07:38:58.805031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:05:03.965626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.518629+0000","last_change":"2024-03-04T07:37:55.138904+0000","last_active":"2024-03-04T07:39:27.518629+0000","last_peered":"2024-03-04T07:39:27.518629+0000","last_clean":"2024-03-04T07:39:27.518629+0000","last_became_active":"2024-03-04T07:37:55.138376+0000","last_became_peered":"2024-03-04T07:37:55.138376+0000","last_unstale":"2024-03-04T07:39:27.518629+0000","last_undegraded":"2024-03-04T07:39:27.518629+0000","last_fullsized":"2024-03-04T07:39:27.518629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:28.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807860+0000","last_change":"2024-03-04T07:37:54.139218+0000","last_active":"2024-03-04T07:38:58.807860+0000","last_peered":"2024-03-04T07:38:58.807860+0000","last_clean":"2024-03-04T07:38:58.807860+0000","last_became_active":"2024-03-04T07:37:54.138893+0000","last_became_peered":"2024-03-04T07:37:54.138893+0000","last_unstale":"2024-03-04T07:38:58.807860+0000","last_undegraded":"2024-03-04T07:38:58.807860+0000","last_fullsized":"2024-03-04T07:38:58.807860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:34:01.729635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511166+0000","last_change":"2024-03-04T07:37:55.135953+0000","last_active":"2024-03-04T07:39:27.511166+0000","last_peered":"2024-03-04T07:39:27.511166+0000","last_clean":"2024-03-04T07:39:27.511166+0000","last_became_active":"2024-03-04T07:37:55.132880+0000","last_became_peered":"2024-03-04T07:37:55.132880+0000","last_unstale":"2024-03-04T07:39:27.511166+0000","last_undegraded":"2024-03-04T07:39:27.511166+0000","last_fullsized":"2024-03-04T07:39:27.511166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:59:03.848428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808631+0000","last_change":"2024-03-04T07:37:54.122493+0000","last_active":"2024-03-04T07:38:58.808631+0000","last_peered":"2024-03-04T07:38:58.808631+0000","last_clean":"2024-03-04T07:38:58.808631+0000","last_became_active":"2024-03-04T07:37:54.122058+0000","last_became_peered":"2024-03-04T07:37:54.122058+0000","last_unstale":"2024-03-04T07:38:58.808631+0000","last_undegraded":"2024-03-04T07:38:58.808631+0000","last_fullsized":"2024-03-04T07:38:58.808631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:38:02.516175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.491455+0000","last_change":"2024-03-04T07:37:55.136155+0000","last_active":"2024-03-04T07:39:27.491455+0000","last_peered":"2024-03-04T07:39:27.491455+0000","last_clean":"2024-03-04T07:39:27.491455+0000","last_became_active":"2024-03-04T07:37:55.135796+0000","last_became_peered":"2024-03-04T07:37:55.135796+0000","last_unstale":"2024-03-04T07:39:27.491455+0000","last_undegraded":"2024-03-04T07:39:27.491455+0000","last_fullsized":"2024-03-04T07:39:27.491455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:37.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808949+0000","last_change":"2024-03-04T07:37:54.118780+0000","last_active":"2024-03-04T07:38:58.808949+0000","last_peered":"2024-03-04T07:38:58.808949+0000","last_clean":"2024-03-04T07:38:58.808949+0000","last_became_active":"2024-03-04T07:37:54.118448+0000","last_became_peered":"2024-03-04T07:37:54.118448+0000","last_unstale":"2024-03-04T07:38:58.808949+0000","last_undegraded":"2024-03-04T07:38:58.808949+0000","last_fullsized":"2024-03-04T07:38:58.808949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:34:25.547046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501807+0000","last_change":"2024-03-04T07:38:59.021729+0000","last_active":"2024-03-04T07:39:27.501807+0000","last_peered":"2024-03-04T07:39:27.501807+0000","last_clean":"2024-03-04T07:39:27.501807+0000","last_became_active":"2024-03-04T07:38:59.021127+0000","last_became_peered":"2024-03-04T07:38:59.021127+0000","last_unstale":"2024-03-04T07:39:27.501807+0000","last_undegraded":"2024-03-04T07:39:27.501807+0000","last_fullsized":"2024-03-04T07:39:27.501807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:08:07.825530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807893+0000","last_change":"2024-03-04T07:37:54.122241+0000","last_active":"2024-03-04T07:38:58.807893+0000","last_peered":"2024-03-04T07:38:58.807893+0000","last_clean":"2024-03-04T07:38:58.807893+0000","last_became_active":"2024-03-04T07:37:54.120990+0000","last_became_peered":"2024-03-04T07:37:54.120990+0000","last_unstale":"2024-03-04T07:38:58.807893+0000","last_undegraded":"2024-03-04T07:38:58.807893+0000","last_fullsized":"2024-03-04T07:38:58.807893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:35:03.233754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501726+0000","last_change":"2024-03-04T07:37:55.121692+0000","last_active":"2024-03-04T07:39:27.501726+0000","last_peered":"2024-03-04T07:39:27.501726+0000","last_clean":"2024-03-04T07:39:27.501726+0000","last_became_active":"2024-03-04T07:37:55.121345+0000","last_became_peered":"2024-03-04T07:37:55.121345+0000","last_unstale":"2024-03-04T07:39:27.501726+0000","last_undegraded":"2024-03-04T07:39:27.501726+0000","last_fullsized":"2024-03-04T07:39:27.501726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:52:42.045265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807447+0000","last_change":"2024-03-04T07:37:54.133489+0000","last_active":"2024-03-04T07:38:58.807447+0000","last_peered":"2024-03-04T07:38:58.807447+0000","last_clean":"2024-03-04T07:38:58.807447+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807447+0000","last_undegraded":"2024-03-04T07:38:58.807447+0000","last_fullsized":"2024-03-04T07:38:58.807447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:49:03.600907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.510134+0000","last_change":"2024-03-04T07:37:55.128278+0000","last_active":"2024-03-04T07:39:27.510134+0000","last_peered":"2024-03-04T07:39:27.510134+0000","last_clean":"2024-03-04T07:39:27.510134+0000","last_became_active":"2024-03-04T07:37:55.127913+0000","last_became_peered":"2024-03-04T07:37:55.127913+0000","last_unstale":"2024-03-04T07:39:27.510134+0000","last_undegraded":"2024-03-04T07:39:27.510134+0000","last_fullsized":"2024-03-04T07:39:27.510134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:54:18.623855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803866+0000","last_change":"2024-03-04T07:37:54.132195+0000","last_active":"2024-03-04T07:38:58.803866+0000","last_peered":"2024-03-04T07:38:58.803866+0000","last_clean":"2024-03-04T07:38:58.803866+0000","last_became_active":"2024-03-04T07:37:54.131830+0000","last_became_peered":"2024-03-04T07:37:54.131830+0000","last_unstale":"2024-03-04T07:38:58.803866+0000","last_undegraded":"2024-03-04T07:38:58.803866+0000","last_fullsized":"2024-03-04T07:38:58.803866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.744342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522878+0000","last_change":"2024-03-04T07:37:55.138281+0000","last_active":"2024-03-04T07:39:27.522878+0000","last_peered":"2024-03-04T07:39:27.522878+0000","last_clean":"2024-03-04T07:39:27.522878+0000","last_became_active":"2024-03-04T07:37:55.137919+0000","last_became_peered":"2024-03-04T07:37:55.137919+0000","last_unstale":"2024-03-04T07:39:27.522878+0000","last_undegraded":"2024-03-04T07:39:27.522878+0000","last_fullsized":"2024-03-04T07:39:27.522878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:53:06.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807612+0000","last_change":"2024-03-04T07:37:54.122067+0000","last_active":"2024-03-04T07:38:58.807612+0000","last_peered":"2024-03-04T07:38:58.807612+0000","last_clean":"2024-03-04T07:38:58.807612+0000","last_became_active":"2024-03-04T07:37:54.121503+0000","last_became_peered":"2024-03-04T07:37:54.121503+0000","last_unstale":"2024-03-04T07:38:58.807612+0000","last_undegraded":"2024-03-04T07:38:58.807612+0000","last_fullsized":"2024-03-04T07:38:58.807612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:03:07.531246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519901+0000","last_change":"2024-03-04T07:37:55.126924+0000","last_active":"2024-03-04T07:39:27.519901+0000","last_peered":"2024-03-04T07:39:27.519901+0000","last_clean":"2024-03-04T07:39:27.519901+0000","last_became_active":"2024-03-04T07:37:55.126266+0000","last_became_peered":"2024-03-04T07:37:55.126266+0000","last_unstale":"2024-03-04T07:39:27.519901+0000","last_undegraded":"2024-03-04T07:39:27.519901+0000","last_fullsized":"2024-03-04T07:39:27.519901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:37:01.603430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807917+0000","last_change":"2024-03-04T07:37:54.119864+0000","last_active":"2024-03-04T07:38:58.807917+0000","last_peered":"2024-03-04T07:38:58.807917+0000","last_clean":"2024-03-04T07:38:58.807917+0000","last_became_active":"2024-03-04T07:37:54.119515+0000","last_became_peered":"2024-03-04T07:37:54.119515+0000","last_unstale":"2024-03-04T07:38:58.807917+0000","last_undegraded":"2024-03-04T07:38:58.807917+0000","last_fullsized":"2024-03-04T07:38:58.807917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:16:10.422458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.490884+0000","last_change":"2024-03-04T07:37:55.120135+0000","last_active":"2024-03-04T07:39:27.490884+0000","last_peered":"2024-03-04T07:39:27.490884+0000","last_clean":"2024-03-04T07:39:27.490884+0000","last_became_active":"2024-03-04T07:37:55.119670+0000","last_became_peered":"2024-03-04T07:37:55.119670+0000","last_unstale":"2024-03-04T07:39:27.490884+0000","last_undegraded":"2024-03-04T07:39:27.490884+0000","last_fullsized":"2024-03-04T07:39:27.490884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:09.688447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807249+0000","last_change":"2024-03-04T07:37:54.120222+0000","last_active":"2024-03-04T07:38:58.807249+0000","last_peered":"2024-03-04T07:38:58.807249+0000","last_clean":"2024-03-04T07:38:58.807249+0000","last_became_active":"2024-03-04T07:37:54.119802+0000","last_became_peered":"2024-03-04T07:37:54.119802+0000","last_unstale":"2024-03-04T07:38:58.807249+0000","last_undegraded":"2024-03-04T07:38:58.807249+0000","last_fullsized":"2024-03-04T07:38:58.807249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:12:40.811870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.509588+0000","last_change":"2024-03-04T07:37:55.136371+0000","last_active":"2024-03-04T07:39:27.509588+0000","last_peered":"2024-03-04T07:39:27.509588+0000","last_clean":"2024-03-04T07:39:27.509588+0000","last_became_active":"2024-03-04T07:37:55.132587+0000","last_became_peered":"2024-03-04T07:37:55.132587+0000","last_unstale":"2024-03-04T07:39:27.509588+0000","last_undegraded":"2024-03-04T07:39:27.509588+0000","last_fullsized":"2024-03-04T07:39:27.509588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:24.397557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.809061+0000","last_change":"2024-03-04T07:37:54.123041+0000","last_active":"2024-03-04T07:38:58.809061+0000","last_peered":"2024-03-04T07:38:58.809061+0000","last_clean":"2024-03-04T07:38:58.809061+0000","last_became_active":"2024-03-04T07:37:54.122723+0000","last_became_peered":"2024-03-04T07:37:54.122723+0000","last_unstale":"2024-03-04T07:38:58.809061+0000","last_undegraded":"2024-03-04T07:38:58.809061+0000","last_fullsized":"2024-03-04T07:38:58.809061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:06:30.845376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488470+0000","last_change":"2024-03-04T07:37:55.115387+0000","last_active":"2024-03-04T07:39:27.488470+0000","last_peered":"2024-03-04T07:39:27.488470+0000","last_clean":"2024-03-04T07:39:27.488470+0000","last_became_active":"2024-03-04T07:37:55.114947+0000","last_became_peered":"2024-03-04T07:37:55.114947+0000","last_unstale":"2024-03-04T07:39:27.488470+0000","last_undegraded":"2024-03-04T07:39:27.488470+0000","last_fullsized":"2024-03-04T07:39:27.488470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:09:11.840989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":361,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:40:27.069018+0000","last_change":"2024-03-04T07:37:54.117767+0000","last_active":"2024-03-04T07:40:27.069018+0000","last_peered":"2024-03-04T07:40:27.069018+0000","last_clean":"2024-03-04T07:40:27.069018+0000","last_became_active":"2024-03-04T07:37:54.117371+0000","last_became_peered":"2024-03-04T07:37:54.117371+0000","last_unstale":"2024-03-04T07:40:27.069018+0000","last_undegraded":"2024-03-04T07:40:27.069018+0000","last_fullsized":"2024-03-04T07:40:27.069018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:08:32.164636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61605,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":310,"num_read_kb":215,"num_write":31,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488421+0000","last_change":"2024-03-04T07:37:55.112707+0000","last_active":"2024-03-04T07:39:27.488421+0000","last_peered":"2024-03-04T07:39:27.488421+0000","last_clean":"2024-03-04T07:39:27.488421+0000","last_became_active":"2024-03-04T07:37:55.112327+0000","last_became_peered":"2024-03-04T07:37:55.112327+0000","last_unstale":"2024-03-04T07:39:27.488421+0000","last_undegraded":"2024-03-04T07:39:27.488421+0000","last_fullsized":"2024-03-04T07:39:27.488421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:03.901687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811773+0000","last_change":"2024-03-04T07:37:54.125286+0000","last_active":"2024-03-04T07:38:58.811773+0000","last_peered":"2024-03-04T07:38:58.811773+0000","last_clean":"2024-03-04T07:38:58.811773+0000","last_became_active":"2024-03-04T07:37:54.124966+0000","last_became_peered":"2024-03-04T07:37:54.124966+0000","last_unstale":"2024-03-04T07:38:58.811773+0000","last_undegraded":"2024-03-04T07:38:58.811773+0000","last_fullsized":"2024-03-04T07:38:58.811773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:19.880681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.524159+0000","last_change":"2024-03-04T07:37:55.137952+0000","last_active":"2024-03-04T07:39:27.524159+0000","last_peered":"2024-03-04T07:39:27.524159+0000","last_clean":"2024-03-04T07:39:27.524159+0000","last_became_active":"2024-03-04T07:37:55.137583+0000","last_became_peered":"2024-03-04T07:37:55.137583+0000","last_unstale":"2024-03-04T07:39:27.524159+0000","last_undegraded":"2024-03-04T07:39:27.524159+0000","last_fullsized":"2024-03-04T07:39:27.524159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:30.145453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807594+0000","last_change":"2024-03-04T07:37:54.134733+0000","last_active":"2024-03-04T07:38:58.807594+0000","last_peered":"2024-03-04T07:38:58.807594+0000","last_clean":"2024-03-04T07:38:58.807594+0000","last_became_active":"2024-03-04T07:37:54.134400+0000","last_became_peered":"2024-03-04T07:37:54.134400+0000","last_unstale":"2024-03-04T07:38:58.807594+0000","last_undegraded":"2024-03-04T07:38:58.807594+0000","last_fullsized":"2024-03-04T07:38:58.807594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:24.192479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511483+0000","last_change":"2024-03-04T07:37:55.124991+0000","last_active":"2024-03-04T07:39:27.511483+0000","last_peered":"2024-03-04T07:39:27.511483+0000","last_clean":"2024-03-04T07:39:27.511483+0000","last_became_active":"2024-03-04T07:37:55.124621+0000","last_became_peered":"2024-03-04T07:37:55.124621+0000","last_unstale":"2024-03-04T07:39:27.511483+0000","last_undegraded":"2024-03-04T07:39:27.511483+0000","last_fullsized":"2024-03-04T07:39:27.511483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:05.252145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021001+0000","last_change":"2024-03-04T07:37:54.118743+0000","last_active":"2024-03-04T07:38:59.021001+0000","last_peered":"2024-03-04T07:38:59.021001+0000","last_clean":"2024-03-04T07:38:59.021001+0000","last_became_active":"2024-03-04T07:37:54.118386+0000","last_became_peered":"2024-03-04T07:37:54.118386+0000","last_unstale":"2024-03-04T07:38:59.021001+0000","last_undegraded":"2024-03-04T07:38:59.021001+0000","last_fullsized":"2024-03-04T07:38:59.021001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:32:27.073614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.473563+0000","last_change":"2024-03-04T07:37:55.115431+0000","last_active":"2024-03-04T07:39:27.473563+0000","last_peered":"2024-03-04T07:39:27.473563+0000","last_clean":"2024-03-04T07:39:27.473563+0000","last_became_active":"2024-03-04T07:37:55.115064+0000","last_became_peered":"2024-03-04T07:37:55.115064+0000","last_unstale":"2024-03-04T07:39:27.473563+0000","last_undegraded":"2024-03-04T07:39:27.473563+0000","last_fullsized":"2024-03-04T07:39:27.473563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:46:57.831154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.029383+0000","last_change":"2024-03-04T07:38:59.028943+0000","last_active":"2024-03-04T07:38:59.029383+0000","last_peered":"2024-03-04T07:38:59.029383+0000","last_clean":"2024-03-04T07:38:59.029383+0000","last_became_active":"2024-03-04T07:38:59.022232+0000","last_became_peered":"2024-03-04T07:38:59.022232+0000","last_unstale":"2024-03-04T07:38:59.029383+0000","last_undegraded":"2024-03-04T07:38:59.029383+0000","last_fullsized":"2024-03-04T07:38:59.029383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:49.281601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.485427+0000","last_change":"2024-03-04T07:37:55.114882+0000","last_active":"2024-03-04T07:39:27.485427+0000","last_peered":"2024-03-04T07:39:27.485427+0000","last_clean":"2024-03-04T07:39:27.485427+0000","last_became_active":"2024-03-04T07:37:55.113139+0000","last_became_peered":"2024-03-04T07:37:55.113139+0000","last_unstale":"2024-03-04T07:39:27.485427+0000","last_undegraded":"2024-03-04T07:39:27.485427+0000","last_fullsized":"2024-03-04T07:39:27.485427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:28.133710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808775+0000","last_change":"2024-03-04T07:37:54.119350+0000","last_active":"2024-03-04T07:38:58.808775+0000","last_peered":"2024-03-04T07:38:58.808775+0000","last_clean":"2024-03-04T07:38:58.808775+0000","last_became_active":"2024-03-04T07:37:54.119017+0000","last_became_peered":"2024-03-04T07:37:54.119017+0000","last_unstale":"2024-03-04T07:38:58.808775+0000","last_undegraded":"2024-03-04T07:38:58.808775+0000","last_fullsized":"2024-03-04T07:38:58.808775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:00.503583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.490355+0000","last_change":"2024-03-04T07:38:58.812598+0000","last_active":"2024-03-04T07:39:27.490355+0000","last_peered":"2024-03-04T07:39:27.490355+0000","last_clean":"2024-03-04T07:39:27.490355+0000","last_became_active":"2024-03-04T07:38:58.812187+0000","last_became_peered":"2024-03-04T07:38:58.812187+0000","last_unstale":"2024-03-04T07:39:27.490355+0000","last_undegraded":"2024-03-04T07:39:27.490355+0000","last_fullsized":"2024-03-04T07:39:27.490355+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:26:43.753007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.804447+0000","last_change":"2024-03-04T07:37:54.125548+0000","last_active":"2024-03-04T07:38:58.804447+0000","last_peered":"2024-03-04T07:38:58.804447+0000","last_clean":"2024-03-04T07:38:58.804447+0000","last_became_active":"2024-03-04T07:37:54.125127+0000","last_became_peered":"2024-03-04T07:37:54.125127+0000","last_unstale":"2024-03-04T07:38:58.804447+0000","last_undegraded":"2024-03-04T07:38:58.804447+0000","last_fullsized":"2024-03-04T07:38:58.804447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:26.256247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.506689+0000","last_change":"2024-03-04T07:37:55.128542+0000","last_active":"2024-03-04T07:39:27.506689+0000","last_peered":"2024-03-04T07:39:27.506689+0000","last_clean":"2024-03-04T07:39:27.506689+0000","last_became_active":"2024-03-04T07:37:55.128239+0000","last_became_peered":"2024-03-04T07:37:55.128239+0000","last_unstale":"2024-03-04T07:39:27.506689+0000","last_undegraded":"2024-03-04T07:39:27.506689+0000","last_fullsized":"2024-03-04T07:39:27.506689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:17.078095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807386+0000","last_change":"2024-03-04T07:37:54.133672+0000","last_active":"2024-03-04T07:38:58.807386+0000","last_peered":"2024-03-04T07:38:58.807386+0000","last_clean":"2024-03-04T07:38:58.807386+0000","last_became_active":"2024-03-04T07:37:54.133103+0000","last_became_peered":"2024-03-04T07:37:54.133103+0000","last_unstale":"2024-03-04T07:38:58.807386+0000","last_undegraded":"2024-03-04T07:38:58.807386+0000","last_fullsized":"2024-03-04T07:38:58.807386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:06:52.345158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.501983+0000","last_change":"2024-03-04T07:38:58.815766+0000","last_active":"2024-03-04T07:39:27.501983+0000","last_peered":"2024-03-04T07:39:27.501983+0000","last_clean":"2024-03-04T07:39:27.501983+0000","last_became_active":"2024-03-04T07:38:58.815387+0000","last_became_peered":"2024-03-04T07:38:58.815387+0000","last_unstale":"2024-03-04T07:39:27.501983+0000","last_undegraded":"2024-03-04T07:39:27.501983+0000","last_fullsized":"2024-03-04T07:39:27.501983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:21.582879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488110+0000","last_change":"2024-03-04T07:38:58.814581+0000","last_active":"2024-03-04T07:39:27.488110+0000","last_peered":"2024-03-04T07:39:27.488110+0000","last_clean":"2024-03-04T07:39:27.488110+0000","last_became_active":"2024-03-04T07:38:58.814186+0000","last_became_peered":"2024-03-04T07:38:58.814186+0000","last_unstale":"2024-03-04T07:39:27.488110+0000","last_undegraded":"2024-03-04T07:39:27.488110+0000","last_fullsized":"2024-03-04T07:39:27.488110+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:03:33.824292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.803774+0000","last_change":"2024-03-04T07:37:54.139335+0000","last_active":"2024-03-04T07:38:58.803774+0000","last_peered":"2024-03-04T07:38:58.803774+0000","last_clean":"2024-03-04T07:38:58.803774+0000","last_became_active":"2024-03-04T07:37:54.138969+0000","last_became_peered":"2024-03-04T07:37:54.138969+0000","last_unstale":"2024-03-04T07:38:58.803774+0000","last_undegraded":"2024-03-04T07:38:58.803774+0000","last_fullsized":"2024-03-04T07:38:58.803774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:41:15.204738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523815+0000","last_change":"2024-03-04T07:38:58.810554+0000","last_active":"2024-03-04T07:39:27.523815+0000","last_peered":"2024-03-04T07:39:27.523815+0000","last_clean":"2024-03-04T07:39:27.523815+0000","last_became_active":"2024-03-04T07:38:58.809978+0000","last_became_peered":"2024-03-04T07:38:58.809978+0000","last_unstale":"2024-03-04T07:39:27.523815+0000","last_undegraded":"2024-03-04T07:39:27.523815+0000","last_fullsized":"2024-03-04T07:39:27.523815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:33.417630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021011+0000","last_change":"2024-03-04T07:37:54.119906+0000","last_active":"2024-03-04T07:38:59.021011+0000","last_peered":"2024-03-04T07:38:59.021011+0000","last_clean":"2024-03-04T07:38:59.021011+0000","last_became_active":"2024-03-04T07:37:54.119592+0000","last_became_peered":"2024-03-04T07:37:54.119592+0000","last_unstale":"2024-03-04T07:38:59.021011+0000","last_undegraded":"2024-03-04T07:38:59.021011+0000","last_fullsized":"2024-03-04T07:38:59.021011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:46:29.184010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":343,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807128+0000","last_change":"2024-03-04T07:36:35.278528+0000","last_active":"2024-03-04T07:38:58.807128+0000","last_peered":"2024-03-04T07:38:58.807128+0000","last_clean":"2024-03-04T07:38:58.807128+0000","last_became_active":"2024-03-04T07:36:35.277953+0000","last_became_peered":"2024-03-04T07:36:35.277953+0000","last_unstale":"2024-03-04T07:38:58.807128+0000","last_undegraded":"2024-03-04T07:38:58.807128+0000","last_fullsized":"2024-03-04T07:38:58.807128+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:32:42.283525+0000","last_clean_scrub_stamp":"2024-03-04T07:32:42.283525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:58:47.223527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.496346+0000","last_change":"2024-03-04T07:37:55.124204+0000","last_active":"2024-03-04T07:39:27.496346+0000","last_peered":"2024-03-04T07:39:27.496346+0000","last_clean":"2024-03-04T07:39:27.496346+0000","last_became_active":"2024-03-04T07:37:55.123863+0000","last_became_peered":"2024-03-04T07:37:55.123863+0000","last_unstale":"2024-03-04T07:39:27.496346+0000","last_undegraded":"2024-03-04T07:39:27.496346+0000","last_fullsized":"2024-03-04T07:39:27.496346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:33.830367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.807256+0000","last_change":"2024-03-04T07:37:54.121538+0000","last_active":"2024-03-04T07:38:58.807256+0000","last_peered":"2024-03-04T07:38:58.807256+0000","last_clean":"2024-03-04T07:38:58.807256+0000","last_became_active":"2024-03-04T07:37:54.120939+0000","last_became_peered":"2024-03-04T07:37:54.120939+0000","last_unstale":"2024-03-04T07:38:58.807256+0000","last_undegraded":"2024-03-04T07:38:58.807256+0000","last_fullsized":"2024-03-04T07:38:58.807256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:29.597247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808527+0000","last_change":"2024-03-04T07:37:54.128180+0000","last_active":"2024-03-04T07:38:58.808527+0000","last_peered":"2024-03-04T07:38:58.808527+0000","last_clean":"2024-03-04T07:38:58.808527+0000","last_became_active":"2024-03-04T07:37:54.127800+0000","last_became_peered":"2024-03-04T07:37:54.127800+0000","last_unstale":"2024-03-04T07:38:58.808527+0000","last_undegraded":"2024-03-04T07:38:58.808527+0000","last_fullsized":"2024-03-04T07:38:58.808527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:54.046873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.504475+0000","last_change":"2024-03-04T07:37:55.133482+0000","last_active":"2024-03-04T07:39:27.504475+0000","last_peered":"2024-03-04T07:39:27.504475+0000","last_clean":"2024-03-04T07:39:27.504475+0000","last_became_active":"2024-03-04T07:37:55.133137+0000","last_became_peered":"2024-03-04T07:37:55.133137+0000","last_unstale":"2024-03-04T07:39:27.504475+0000","last_undegraded":"2024-03-04T07:39:27.504475+0000","last_fullsized":"2024-03-04T07:39:27.504475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:37:53.509564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:40:04.733943+0000","last_change":"2024-03-04T07:37:54.134294+0000","last_active":"2024-03-04T07:40:04.733943+0000","last_peered":"2024-03-04T07:40:04.733943+0000","last_clean":"2024-03-04T07:40:04.733943+0000","last_became_active":"2024-03-04T07:37:54.133677+0000","last_became_peered":"2024-03-04T07:37:54.133677+0000","last_unstale":"2024-03-04T07:40:04.733943+0000","last_undegraded":"2024-03-04T07:40:04.733943+0000","last_fullsized":"2024-03-04T07:40:04.733943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T07:57:54.082419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522888+0000","last_change":"2024-03-04T07:37:55.138305+0000","last_active":"2024-03-04T07:39:27.522888+0000","last_peered":"2024-03-04T07:39:27.522888+0000","last_clean":"2024-03-04T07:39:27.522888+0000","last_became_active":"2024-03-04T07:37:55.129937+0000","last_became_peered":"2024-03-04T07:37:55.129937+0000","last_unstale":"2024-03-04T07:39:27.522888+0000","last_undegraded":"2024-03-04T07:39:27.522888+0000","last_fullsized":"2024-03-04T07:39:27.522888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:54:45.136539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:59.021306+0000","last_change":"2024-03-04T07:37:54.133817+0000","last_active":"2024-03-04T07:38:59.021306+0000","last_peered":"2024-03-04T07:38:59.021306+0000","last_clean":"2024-03-04T07:38:59.021306+0000","last_became_active":"2024-03-04T07:37:54.133471+0000","last_became_peered":"2024-03-04T07:37:54.133471+0000","last_unstale":"2024-03-04T07:38:59.021306+0000","last_undegraded":"2024-03-04T07:38:59.021306+0000","last_fullsized":"2024-03-04T07:38:59.021306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:00:51.483859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.517445+0000","last_change":"2024-03-04T07:37:55.144639+0000","last_active":"2024-03-04T07:39:27.517445+0000","last_peered":"2024-03-04T07:39:27.517445+0000","last_clean":"2024-03-04T07:39:27.517445+0000","last_became_active":"2024-03-04T07:37:55.144194+0000","last_became_peered":"2024-03-04T07:37:55.144194+0000","last_unstale":"2024-03-04T07:39:27.517445+0000","last_undegraded":"2024-03-04T07:39:27.517445+0000","last_fullsized":"2024-03-04T07:39:27.517445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:42.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.816116+0000","last_change":"2024-03-04T07:38:58.815642+0000","last_active":"2024-03-04T07:38:58.816116+0000","last_peered":"2024-03-04T07:38:58.816116+0000","last_clean":"2024-03-04T07:38:58.816116+0000","last_became_active":"2024-03-04T07:38:58.815361+0000","last_became_peered":"2024-03-04T07:38:58.815361+0000","last_unstale":"2024-03-04T07:38:58.816116+0000","last_undegraded":"2024-03-04T07:38:58.816116+0000","last_fullsized":"2024-03-04T07:38:58.816116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:48:15.751929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487964+0000","last_change":"2024-03-04T07:37:55.115145+0000","last_active":"2024-03-04T07:39:27.487964+0000","last_peered":"2024-03-04T07:39:27.487964+0000","last_clean":"2024-03-04T07:39:27.487964+0000","last_became_active":"2024-03-04T07:37:55.114791+0000","last_became_peered":"2024-03-04T07:37:55.114791+0000","last_unstale":"2024-03-04T07:39:27.487964+0000","last_undegraded":"2024-03-04T07:39:27.487964+0000","last_fullsized":"2024-03-04T07:39:27.487964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:19.685977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.806430+0000","last_change":"2024-03-04T07:37:54.141630+0000","last_active":"2024-03-04T07:38:58.806430+0000","last_peered":"2024-03-04T07:38:58.806430+0000","last_clean":"2024-03-04T07:38:58.806430+0000","last_became_active":"2024-03-04T07:37:54.139820+0000","last_became_peered":"2024-03-04T07:37:54.139820+0000","last_unstale":"2024-03-04T07:38:58.806430+0000","last_undegraded":"2024-03-04T07:38:58.806430+0000","last_fullsized":"2024-03-04T07:38:58.806430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:18:22.419437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.503153+0000","last_change":"2024-03-04T07:37:55.133270+0000","last_active":"2024-03-04T07:39:27.503153+0000","last_peered":"2024-03-04T07:39:27.503153+0000","last_clean":"2024-03-04T07:39:27.503153+0000","last_became_active":"2024-03-04T07:37:55.132909+0000","last_became_peered":"2024-03-04T07:37:55.132909+0000","last_unstale":"2024-03-04T07:39:27.503153+0000","last_undegraded":"2024-03-04T07:39:27.503153+0000","last_fullsized":"2024-03-04T07:39:27.503153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:21.026141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.812572+0000","last_change":"2024-03-04T07:37:54.135988+0000","last_active":"2024-03-04T07:38:58.812572+0000","last_peered":"2024-03-04T07:38:58.812572+0000","last_clean":"2024-03-04T07:38:58.812572+0000","last_became_active":"2024-03-04T07:37:54.135627+0000","last_became_peered":"2024-03-04T07:37:54.135627+0000","last_unstale":"2024-03-04T07:38:58.812572+0000","last_undegraded":"2024-03-04T07:38:58.812572+0000","last_fullsized":"2024-03-04T07:38:58.812572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:06.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.522153+0000","last_change":"2024-03-04T07:37:55.144551+0000","last_active":"2024-03-04T07:39:27.522153+0000","last_peered":"2024-03-04T07:39:27.522153+0000","last_clean":"2024-03-04T07:39:27.522153+0000","last_became_active":"2024-03-04T07:37:55.143882+0000","last_became_peered":"2024-03-04T07:37:55.143882+0000","last_unstale":"2024-03-04T07:39:27.522153+0000","last_undegraded":"2024-03-04T07:39:27.522153+0000","last_fullsized":"2024-03-04T07:39:27.522153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:07.254941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.817917+0000","last_change":"2024-03-04T07:38:58.817440+0000","last_active":"2024-03-04T07:38:58.817917+0000","last_peered":"2024-03-04T07:38:58.817917+0000","last_clean":"2024-03-04T07:38:58.817917+0000","last_became_active":"2024-03-04T07:38:58.813620+0000","last_became_peered":"2024-03-04T07:38:58.813620+0000","last_unstale":"2024-03-04T07:38:58.817917+0000","last_undegraded":"2024-03-04T07:38:58.817917+0000","last_fullsized":"2024-03-04T07:38:58.817917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:10:41.554608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.505218+0000","last_change":"2024-03-04T07:37:55.133870+0000","last_active":"2024-03-04T07:39:27.505218+0000","last_peered":"2024-03-04T07:39:27.505218+0000","last_clean":"2024-03-04T07:39:27.505218+0000","last_became_active":"2024-03-04T07:37:55.133542+0000","last_became_peered":"2024-03-04T07:37:55.133542+0000","last_unstale":"2024-03-04T07:39:27.505218+0000","last_undegraded":"2024-03-04T07:39:27.505218+0000","last_fullsized":"2024-03-04T07:39:27.505218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:17:44.373426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.811749+0000","last_change":"2024-03-04T07:37:54.126326+0000","last_active":"2024-03-04T07:38:58.811749+0000","last_peered":"2024-03-04T07:38:58.811749+0000","last_clean":"2024-03-04T07:38:58.811749+0000","last_became_active":"2024-03-04T07:37:54.125989+0000","last_became_peered":"2024-03-04T07:37:54.125989+0000","last_unstale":"2024-03-04T07:38:58.811749+0000","last_undegraded":"2024-03-04T07:38:58.811749+0000","last_fullsized":"2024-03-04T07:38:58.811749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:41.108528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488757+0000","last_change":"2024-03-04T07:37:55.120052+0000","last_active":"2024-03-04T07:39:27.488757+0000","last_peered":"2024-03-04T07:39:27.488757+0000","last_clean":"2024-03-04T07:39:27.488757+0000","last_became_active":"2024-03-04T07:37:55.119708+0000","last_became_peered":"2024-03-04T07:37:55.119708+0000","last_unstale":"2024-03-04T07:39:27.488757+0000","last_undegraded":"2024-03-04T07:39:27.488757+0000","last_fullsized":"2024-03-04T07:39:27.488757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T08:25:17.679416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.808512+0000","last_change":"2024-03-04T07:37:54.134018+0000","last_active":"2024-03-04T07:38:58.808512+0000","last_peered":"2024-03-04T07:38:58.808512+0000","last_clean":"2024-03-04T07:38:58.808512+0000","last_became_active":"2024-03-04T07:37:54.133673+0000","last_became_peered":"2024-03-04T07:37:54.133673+0000","last_unstale":"2024-03-04T07:38:58.808512+0000","last_undegraded":"2024-03-04T07:38:58.808512+0000","last_fullsized":"2024-03-04T07:38:58.808512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:50:49.438338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489514+0000","last_change":"2024-03-04T07:38:58.817071+0000","last_active":"2024-03-04T07:39:27.489514+0000","last_peered":"2024-03-04T07:39:27.489514+0000","last_clean":"2024-03-04T07:39:27.489514+0000","last_became_active":"2024-03-04T07:38:58.812139+0000","last_became_peered":"2024-03-04T07:38:58.812139+0000","last_unstale":"2024-03-04T07:39:27.489514+0000","last_undegraded":"2024-03-04T07:39:27.489514+0000","last_fullsized":"2024-03-04T07:39:27.489514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:37.557255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:38:58.805106+0000","last_change":"2024-03-04T07:37:54.140806+0000","last_active":"2024-03-04T07:38:58.805106+0000","last_peered":"2024-03-04T07:38:58.805106+0000","last_clean":"2024-03-04T07:38:58.805106+0000","last_became_active":"2024-03-04T07:37:54.140419+0000","last_became_peered":"2024-03-04T07:37:54.140419+0000","last_unstale":"2024-03-04T07:38:58.805106+0000","last_undegraded":"2024-03-04T07:38:58.805106+0000","last_fullsized":"2024-03-04T07:38:58.805106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:53.084326+0000","last_clean_scrub_stamp":"2024-03-04T07:37:53.084326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:13:59.261271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.489032+0000","last_change":"2024-03-04T07:37:55.113658+0000","last_active":"2024-03-04T07:39:27.489032+0000","last_peered":"2024-03-04T07:39:27.489032+0000","last_clean":"2024-03-04T07:39:27.489032+0000","last_became_active":"2024-03-04T07:37:55.113300+0000","last_became_peered":"2024-03-04T07:37:55.113300+0000","last_unstale":"2024-03-04T07:39:27.489032+0000","last_undegraded":"2024-03-04T07:39:27.489032+0000","last_fullsized":"2024-03-04T07:39:27.489032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:43:56.907387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487646+0000","last_change":"2024-03-04T07:37:55.117769+0000","last_active":"2024-03-04T07:39:27.487646+0000","last_peered":"2024-03-04T07:39:27.487646+0000","last_clean":"2024-03-04T07:39:27.487646+0000","last_became_active":"2024-03-04T07:37:55.117408+0000","last_became_peered":"2024-03-04T07:37:55.117408+0000","last_unstale":"2024-03-04T07:39:27.487646+0000","last_undegraded":"2024-03-04T07:39:27.487646+0000","last_fullsized":"2024-03-04T07:39:27.487646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:42:22.143108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.483985+0000","last_change":"2024-03-04T07:37:55.115581+0000","last_active":"2024-03-04T07:39:27.483985+0000","last_peered":"2024-03-04T07:39:27.483985+0000","last_clean":"2024-03-04T07:39:27.483985+0000","last_became_active":"2024-03-04T07:37:55.113387+0000","last_became_peered":"2024-03-04T07:37:55.113387+0000","last_unstale":"2024-03-04T07:39:27.483985+0000","last_undegraded":"2024-03-04T07:39:27.483985+0000","last_fullsized":"2024-03-04T07:39:27.483985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T09:58:59.273650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.492691+0000","last_change":"2024-03-04T07:37:55.135175+0000","last_active":"2024-03-04T07:39:27.492691+0000","last_peered":"2024-03-04T07:39:27.492691+0000","last_clean":"2024-03-04T07:39:27.492691+0000","last_became_active":"2024-03-04T07:37:55.134787+0000","last_became_peered":"2024-03-04T07:37:55.134787+0000","last_unstale":"2024-03-04T07:39:27.492691+0000","last_undegraded":"2024-03-04T07:39:27.492691+0000","last_fullsized":"2024-03-04T07:39:27.492691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:46:04.227886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511558+0000","last_change":"2024-03-04T07:37:55.138146+0000","last_active":"2024-03-04T07:39:27.511558+0000","last_peered":"2024-03-04T07:39:27.511558+0000","last_clean":"2024-03-04T07:39:27.511558+0000","last_became_active":"2024-03-04T07:37:55.137806+0000","last_became_peered":"2024-03-04T07:37:55.137806+0000","last_unstale":"2024-03-04T07:39:27.511558+0000","last_undegraded":"2024-03-04T07:39:27.511558+0000","last_fullsized":"2024-03-04T07:39:27.511558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:04:10.596224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.507386+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.507386+0000","last_peered":"2024-03-04T07:39:27.507386+0000","last_clean":"2024-03-04T07:39:27.507386+0000","last_became_active":"2024-03-04T07:37:55.140670+0000","last_became_peered":"2024-03-04T07:37:55.140670+0000","last_unstale":"2024-03-04T07:39:27.507386+0000","last_undegraded":"2024-03-04T07:39:27.507386+0000","last_fullsized":"2024-03-04T07:39:27.507386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:29:54.439417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.487788+0000","last_change":"2024-03-04T07:37:55.115015+0000","last_active":"2024-03-04T07:39:27.487788+0000","last_peered":"2024-03-04T07:39:27.487788+0000","last_clean":"2024-03-04T07:39:27.487788+0000","last_became_active":"2024-03-04T07:37:55.114668+0000","last_became_peered":"2024-03-04T07:37:55.114668+0000","last_unstale":"2024-03-04T07:39:27.487788+0000","last_undegraded":"2024-03-04T07:39:27.487788+0000","last_fullsized":"2024-03-04T07:39:27.487788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:29:03.309234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.482820+0000","last_change":"2024-03-04T07:37:55.119514+0000","last_active":"2024-03-04T07:39:27.482820+0000","last_peered":"2024-03-04T07:39:27.482820+0000","last_clean":"2024-03-04T07:39:27.482820+0000","last_became_active":"2024-03-04T07:37:55.119166+0000","last_became_peered":"2024-03-04T07:37:55.119166+0000","last_unstale":"2024-03-04T07:39:27.482820+0000","last_undegraded":"2024-03-04T07:39:27.482820+0000","last_fullsized":"2024-03-04T07:39:27.482820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:33.468311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.520592+0000","last_change":"2024-03-04T07:37:55.141134+0000","last_active":"2024-03-04T07:39:27.520592+0000","last_peered":"2024-03-04T07:39:27.520592+0000","last_clean":"2024-03-04T07:39:27.520592+0000","last_became_active":"2024-03-04T07:37:55.140723+0000","last_became_peered":"2024-03-04T07:37:55.140723+0000","last_unstale":"2024-03-04T07:39:27.520592+0000","last_undegraded":"2024-03-04T07:39:27.520592+0000","last_fullsized":"2024-03-04T07:39:27.520592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:29.751190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.510302+0000","last_change":"2024-03-04T07:37:55.127627+0000","last_active":"2024-03-04T07:39:27.510302+0000","last_peered":"2024-03-04T07:39:27.510302+0000","last_clean":"2024-03-04T07:39:27.510302+0000","last_became_active":"2024-03-04T07:37:55.127217+0000","last_became_peered":"2024-03-04T07:37:55.127217+0000","last_unstale":"2024-03-04T07:39:27.510302+0000","last_undegraded":"2024-03-04T07:39:27.510302+0000","last_fullsized":"2024-03-04T07:39:27.510302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:47.826970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.519782+0000","last_change":"2024-03-04T07:37:55.129445+0000","last_active":"2024-03-04T07:39:27.519782+0000","last_peered":"2024-03-04T07:39:27.519782+0000","last_clean":"2024-03-04T07:39:27.519782+0000","last_became_active":"2024-03-04T07:37:55.129013+0000","last_became_peered":"2024-03-04T07:37:55.129013+0000","last_unstale":"2024-03-04T07:39:27.519782+0000","last_undegraded":"2024-03-04T07:39:27.519782+0000","last_fullsized":"2024-03-04T07:39:27.519782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:15.578436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.523619+0000","last_change":"2024-03-04T07:37:55.126397+0000","last_active":"2024-03-04T07:39:27.523619+0000","last_peered":"2024-03-04T07:39:27.523619+0000","last_clean":"2024-03-04T07:39:27.523619+0000","last_became_active":"2024-03-04T07:37:55.126092+0000","last_became_peered":"2024-03-04T07:37:55.126092+0000","last_unstale":"2024-03-04T07:39:27.523619+0000","last_undegraded":"2024-03-04T07:39:27.523619+0000","last_fullsized":"2024-03-04T07:39:27.523619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:47.968250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.511142+0000","last_change":"2024-03-04T07:37:55.140265+0000","last_active":"2024-03-04T07:39:27.511142+0000","last_peered":"2024-03-04T07:39:27.511142+0000","last_clean":"2024-03-04T07:39:27.511142+0000","last_became_active":"2024-03-04T07:37:55.139768+0000","last_became_peered":"2024-03-04T07:37:55.139768+0000","last_unstale":"2024-03-04T07:39:27.511142+0000","last_undegraded":"2024-03-04T07:39:27.511142+0000","last_fullsized":"2024-03-04T07:39:27.511142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:40:52.924188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-04T07:39:27.488738+0000","last_change":"2024-03-04T07:37:55.118929+0000","last_active":"2024-03-04T07:39:27.488738+0000","last_peered":"2024-03-04T07:39:27.488738+0000","last_clean":"2024-03-04T07:39:27.488738+0000","last_became_active":"2024-03-04T07:37:55.118549+0000","last_became_peered":"2024-03-04T07:37:55.118549+0000","last_unstale":"2024-03-04T07:39:27.488738+0000","last_undegraded":"2024-03-04T07:39:27.488738+0000","last_fullsized":"2024-03-04T07:39:27.488738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T07:37:54.092505+0000","last_clean_scrub_stamp":"2024-03-04T07:37:54.092505+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:17:15.599080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244954,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62403,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":314,"num_read_kb":219,"num_write":84,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":187209,"data_compressed":5550,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383725,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":455726,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645362,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1108010,"data_compressed":43363,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456425,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070525,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":456330,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332162,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1046793,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455843,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":517021,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455816,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":517365,"data_compressed":7665,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182489,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":455947,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920853,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":455748,"data_compressed":5815,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":1046393,"data_compressed":41513,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35698,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61605,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61639,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61799,"data_compressed":1850,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T07:40:29.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:40:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 -- 172.21.15.116:0/3646383796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4104a00 msgr2=0x7f98b4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 --2- 172.21.15.116:0/3646383796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4104a00 0x7f98b4104e80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009900 tx=0x7f98a002bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 -- 172.21.15.116:0/3646383796 shutdown_connections 2024-03-04T07:40:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 --2- 172.21.15.116:0/3646383796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98b41053c0 0x7f98b410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 --2- 172.21.15.116:0/3646383796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4104a00 0x7f98b4104e80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 --2- 172.21.15.116:0/3646383796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98b4103810 0x7f98b4103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 -- 172.21.15.116:0/3646383796 >> 172.21.15.116:0/3646383796 conn(0x7f98b40fedc0 msgr2=0x7f98b41011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 -- 172.21.15.116:0/3646383796 shutdown_connections 2024-03-04T07:40:29.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.440+0000 7f98bc200640 1 -- 172.21.15.116:0/3646383796 wait complete. 2024-03-04T07:40:29.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 Processor -- start 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 -- start start 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4103810 0x7f98b4199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98b4104a00 0x7f98b419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98b41053c0 0x7f98b41a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b410f490 con 0x7f98b41053c0 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b410f310 con 0x7f98b4103810 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98bc200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b410f610 con 0x7f98b4104a00 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98b9800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4103810 0x7f98b4199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98b9800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4103810 0x7f98b4199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56956/0 (socket says 172.21.15.116:56956) 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.441+0000 7f98b9800640 1 -- 172.21.15.116:0/1576448545 learned_addr learned my addr 172.21.15.116:0/1576448545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98ba200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98b41053c0 0x7f98b41a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b8e00640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98b4104a00 0x7f98b419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b9800640 1 -- 172.21.15.116:0/1576448545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98b4104a00 msgr2=0x7f98b419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b9800640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98b4104a00 0x7f98b419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b9800640 1 -- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98b41053c0 msgr2=0x7f98b41a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b9800640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98b41053c0 0x7f98b41a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b9800640 1 -- 172.21.15.116:0/1576448545 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b41a1720 con 0x7f98b4103810 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98ba200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98b41053c0 0x7f98b41a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b9800640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4103810 0x7f98b4199b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e9c0 tx=0x7f98a800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b2000640 1 -- 172.21.15.116:0/1576448545 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a800cd80 con 0x7f98b4103810 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b2000640 1 -- 172.21.15.116:0/1576448545 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98a80035f0 con 0x7f98b4103810 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b41a1a10 con 0x7f98b4103810 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.442+0000 7f98b2000640 1 -- 172.21.15.116:0/1576448545 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8010690 con 0x7f98b4103810 2024-03-04T07:40:29.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.443+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b41a1f30 con 0x7f98b4103810 2024-03-04T07:40:29.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.444+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b4068810 con 0x7f98b4103810 2024-03-04T07:40:29.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.445+0000 7f98b2000640 1 -- 172.21.15.116:0/1576448545 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f98a80037a0 con 0x7f98b4103810 2024-03-04T07:40:29.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.447+0000 7f98b2000640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f989c0a1d80 0x7f989c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.447+0000 7f98b2000640 1 -- 172.21.15.116:0/1576448545 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f98a80ce200 con 0x7f98b4103810 2024-03-04T07:40:29.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.447+0000 7f98b8e00640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f989c0a1d80 0x7f989c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.447+0000 7f98b8e00640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f989c0a1d80 0x7f989c0a4240 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f98a002c7b0 tx=0x7f98a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:29.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.449+0000 7f98b2000640 1 -- 172.21.15.116:0/1576448545 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f98a809b5f0 con 0x7f98b4103810 2024-03-04T07:40:29.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.645+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98b410ab30 con 0x7f98b4103810 2024-03-04T07:40:29.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.648+0000 7f98b2000640 1 -- 172.21.15.116:0/1576448545 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7f98a809aef0 con 0x7f98b4103810 2024-03-04T07:40:29.649 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:29.649 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:40:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f989c0a1d80 msgr2=0x7f989c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f989c0a1d80 0x7f989c0a4240 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f98a002c7b0 tx=0x7f98a0005fd0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4103810 msgr2=0x7f98b4199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4103810 0x7f98b4199b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e9c0 tx=0x7f98a800ee80 comp rx=0 tx=0).stop 2024-03-04T07:40:29.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 shutdown_connections 2024-03-04T07:40:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98b41053c0 0x7f98b41a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f989c0a1d80 0x7f989c0a4240 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98b4104a00 0x7f98b419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 --2- 172.21.15.116:0/1576448545 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98b4103810 0x7f98b4199b00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.650+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 >> 172.21.15.116:0/1576448545 conn(0x7f98b40fedc0 msgr2=0x7f98b4107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.651+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 shutdown_connections 2024-03-04T07:40:29.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.651+0000 7f98bc200640 1 -- 172.21.15.116:0/1576448545 wait complete. 2024-03-04T07:40:29.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:40:29.669 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.854+0000 7f2b1a600640 1 -- 172.21.15.116:0/545804455 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141027d0 msgr2=0x7f2b14102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.854+0000 7f2b1a600640 1 --2- 172.21.15.116:0/545804455 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141027d0 0x7f2b14102c50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009900 tx=0x7f2b0002bc50 comp rx=0 tx=0).stop 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 -- 172.21.15.116:0/545804455 shutdown_connections 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 --2- 172.21.15.116:0/545804455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b14103190 0x7f2b14109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 --2- 172.21.15.116:0/545804455 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141027d0 0x7f2b14102c50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 --2- 172.21.15.116:0/545804455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b141015e0 0x7f2b14101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 -- 172.21.15.116:0/545804455 >> 172.21.15.116:0/545804455 conn(0x7f2b140fcb70 msgr2=0x7f2b140fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 -- 172.21.15.116:0/545804455 shutdown_connections 2024-03-04T07:40:29.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 -- 172.21.15.116:0/545804455 wait complete. 2024-03-04T07:40:29.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.855+0000 7f2b1a600640 1 Processor -- start 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b1a600640 1 -- start start 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b1a600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141015e0 0x7f2b14197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b1a600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b141027d0 0x7f2b14197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b1a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b14103190 0x7f2b1419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b1a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1410d260 con 0x7f2b14103190 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b1a600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1410d0e0 con 0x7f2b141015e0 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b1a600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1410d3e0 con 0x7f2b141027d0 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b13400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141015e0 0x7f2b14197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b13400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141015e0 0x7f2b14197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56974/0 (socket says 172.21.15.116:56974) 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b13400640 1 -- 172.21.15.116:0/1737193042 learned_addr learned my addr 172.21.15.116:0/1737193042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b12a00640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b141027d0 0x7f2b14197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.856+0000 7f2b13e00640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b14103190 0x7f2b1419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b13400640 1 -- 172.21.15.116:0/1737193042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b141027d0 msgr2=0x7f2b14197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b13400640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b141027d0 0x7f2b14197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b13400640 1 -- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b14103190 msgr2=0x7f2b1419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b13400640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b14103190 0x7f2b1419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b13400640 1 -- 172.21.15.116:0/1737193042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1419f5b0 con 0x7f2b141015e0 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b13e00640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b14103190 0x7f2b1419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b13400640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141015e0 0x7f2b14197990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400e9c0 tx=0x7f2b0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b0be00640 1 -- 172.21.15.116:0/1737193042 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0400cd80 con 0x7f2b141015e0 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1419f8a0 con 0x7f2b141015e0 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b0be00640 1 -- 172.21.15.116:0/1737193042 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b040035f0 con 0x7f2b141015e0 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b0be00640 1 -- 172.21.15.116:0/1737193042 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04010690 con 0x7f2b141015e0 2024-03-04T07:40:29.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.857+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1419fd70 con 0x7f2b141015e0 2024-03-04T07:40:29.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.859+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b141089a0 con 0x7f2b141015e0 2024-03-04T07:40:29.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.860+0000 7f2b0be00640 1 -- 172.21.15.116:0/1737193042 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2b040037a0 con 0x7f2b141015e0 2024-03-04T07:40:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.861+0000 7f2b0be00640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2af80a1d80 0x7f2af80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.861+0000 7f2b0be00640 1 -- 172.21.15.116:0/1737193042 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2b040ce1b0 con 0x7f2b141015e0 2024-03-04T07:40:29.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.862+0000 7f2b12a00640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2af80a1d80 0x7f2af80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:29.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.862+0000 7f2b12a00640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2af80a1d80 0x7f2af80a4240 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2b0002c7d0 tx=0x7f2b00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:29.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:29.865+0000 7f2b0be00640 1 -- 172.21.15.116:0/1737193042 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2b0409b5e0 con 0x7f2b141015e0 2024-03-04T07:40:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:29 smithi116 ceph-mon[32150]: from='client.25181 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:29 smithi116 ceph-mon[32150]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:29 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:29 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1576448545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:29 smithi152 ceph-mon[37154]: from='client.25181 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:29 smithi152 ceph-mon[37154]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:29 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:29 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1576448545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:30.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.068+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b1402cff0 con 0x7f2b141015e0 2024-03-04T07:40:30.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.069+0000 7f2b0be00640 1 -- 172.21.15.116:0/1737193042 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b0409aee0 con 0x7f2b141015e0 2024-03-04T07:40:30.070 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.071+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2af80a1d80 msgr2=0x7f2af80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.071+0000 7f2b1a600640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2af80a1d80 0x7f2af80a4240 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2b0002c7d0 tx=0x7f2b00005fd0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.071+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141015e0 msgr2=0x7f2b14197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.071+0000 7f2b1a600640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141015e0 0x7f2b14197990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400e9c0 tx=0x7f2b0400ee80 comp rx=0 tx=0).stop 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 shutdown_connections 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b14103190 0x7f2b1419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2af80a1d80 0x7f2af80a4240 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b141027d0 0x7f2b14197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 --2- 172.21.15.116:0/1737193042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b141015e0 0x7f2b14197990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 >> 172.21.15.116:0/1737193042 conn(0x7f2b140fcb70 msgr2=0x7f2b14105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:30.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 shutdown_connections 2024-03-04T07:40:30.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.072+0000 7f2b1a600640 1 -- 172.21.15.116:0/1737193042 wait complete. 2024-03-04T07:40:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:29 smithi133 ceph-mon[36505]: from='client.25181 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:29 smithi133 ceph-mon[36505]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:29 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:30.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:29 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1576448545' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:30.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.288+0000 7f4435e00640 1 -- 172.21.15.116:0/4025878742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4430103810 msgr2=0x7f4430103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.288+0000 7f4435e00640 1 --2- 172.21.15.116:0/4025878742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4430103810 0x7f4430103c30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4420009900 tx=0x7f442002bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.288+0000 7f4435e00640 1 -- 172.21.15.116:0/4025878742 shutdown_connections 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.288+0000 7f4435e00640 1 --2- 172.21.15.116:0/4025878742 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f44301053c0 0x7f443010bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.288+0000 7f4435e00640 1 --2- 172.21.15.116:0/4025878742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430104a00 0x7f4430104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.288+0000 7f4435e00640 1 --2- 172.21.15.116:0/4025878742 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4430103810 0x7f4430103c30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.289+0000 7f4435e00640 1 -- 172.21.15.116:0/4025878742 >> 172.21.15.116:0/4025878742 conn(0x7f44300fed80 msgr2=0x7f44301011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.289+0000 7f4435e00640 1 -- 172.21.15.116:0/4025878742 shutdown_connections 2024-03-04T07:40:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.289+0000 7f4435e00640 1 -- 172.21.15.116:0/4025878742 wait complete. 2024-03-04T07:40:30.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.289+0000 7f4435e00640 1 Processor -- start 2024-03-04T07:40:30.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f4435e00640 1 -- start start 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f4435e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430103810 0x7f4430199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f4435e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4430104a00 0x7f443019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f4435e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44301053c0 0x7f44301a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f4435e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443010f4b0 con 0x7f4430103810 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f4435e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443010f330 con 0x7f4430104a00 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f4435e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443010f630 con 0x7f44301053c0 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f442e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4430104a00 0x7f443019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f442ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430103810 0x7f4430199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f442f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44301053c0 0x7f44301a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f442e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4430104a00 0x7f443019a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57004/0 (socket says 172.21.15.116:57004) 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.290+0000 7f442e000640 1 -- 172.21.15.116:0/3506766106 learned_addr learned my addr 172.21.15.116:0/3506766106 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f442e000640 1 -- 172.21.15.116:0/3506766106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44301053c0 msgr2=0x7f44301a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f442e000640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44301053c0 0x7f44301a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f442e000640 1 -- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430103810 msgr2=0x7f4430199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f442e000640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430103810 0x7f4430199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f442e000640 1 -- 172.21.15.116:0/3506766106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44301a1770 con 0x7f4430104a00 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f442f400640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44301053c0 0x7f44301a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f442e000640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4430104a00 0x7f443019a090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f441c00e9f0 tx=0x7f441c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f4434a00640 1 -- 172.21.15.116:0/3506766106 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c00cd80 con 0x7f4430104a00 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f4434a00640 1 -- 172.21.15.116:0/3506766106 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f441c010910 con 0x7f4430104a00 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44301a1a60 con 0x7f4430104a00 2024-03-04T07:40:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f4434a00640 1 -- 172.21.15.116:0/3506766106 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c0188f0 con 0x7f4430104a00 2024-03-04T07:40:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.291+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44301a1fb0 con 0x7f4430104a00 2024-03-04T07:40:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.292+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443010ab30 con 0x7f4430104a00 2024-03-04T07:40:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.295+0000 7f4434a00640 1 -- 172.21.15.116:0/3506766106 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f441c010ac0 con 0x7f4430104a00 2024-03-04T07:40:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.296+0000 7f4434a00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44140a1d80 0x7f44140a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.297+0000 7f4434a00640 1 -- 172.21.15.116:0/3506766106 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f441c0ce340 con 0x7f4430104a00 2024-03-04T07:40:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.297+0000 7f442ea00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44140a1d80 0x7f44140a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.298+0000 7f442ea00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44140a1d80 0x7f44140a4240 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f442002c170 tx=0x7f442002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.299+0000 7f4434a00640 1 -- 172.21.15.116:0/3506766106 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f441c09b0e0 con 0x7f4430104a00 2024-03-04T07:40:30.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.495+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f443002d240 con 0x7f4430104a00 2024-03-04T07:40:30.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.496+0000 7f4434a00640 1 -- 172.21.15.116:0/3506766106 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f441c09f180 con 0x7f4430104a00 2024-03-04T07:40:30.497 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:40:30.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44140a1d80 msgr2=0x7f44140a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44140a1d80 0x7f44140a4240 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f442002c170 tx=0x7f442002df90 comp rx=0 tx=0).stop 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4430104a00 msgr2=0x7f443019a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4430104a00 0x7f443019a090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f441c00e9f0 tx=0x7f441c00eeb0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 shutdown_connections 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f44140a1d80 0x7f44140a4240 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44301053c0 0x7f44301a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4430104a00 0x7f443019a090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 --2- 172.21.15.116:0/3506766106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4430103810 0x7f4430199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.498+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 >> 172.21.15.116:0/3506766106 conn(0x7f44300fed80 msgr2=0x7f4430107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.499+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 shutdown_connections 2024-03-04T07:40:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.499+0000 7f4435e00640 1 -- 172.21.15.116:0/3506766106 wait complete. 2024-03-04T07:40:30.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.711+0000 7fbe4c400640 1 -- 172.21.15.116:0/2215585842 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44074030 msgr2=0x7fbe440744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.711+0000 7fbe4c400640 1 --2- 172.21.15.116:0/2215585842 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44074030 0x7fbe440744b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009930 tx=0x7fbe3002bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.712+0000 7fbe4c400640 1 -- 172.21.15.116:0/2215585842 shutdown_connections 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.712+0000 7fbe4c400640 1 --2- 172.21.15.116:0/2215585842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4410dbb0 0x7fbe4410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.712+0000 7fbe4c400640 1 --2- 172.21.15.116:0/2215585842 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44074030 0x7fbe440744b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.712+0000 7fbe4c400640 1 --2- 172.21.15.116:0/2215585842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe44075610 0x7fbe44073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.712+0000 7fbe4c400640 1 -- 172.21.15.116:0/2215585842 >> 172.21.15.116:0/2215585842 conn(0x7fbe440fcb70 msgr2=0x7fbe440fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.712+0000 7fbe4c400640 1 -- 172.21.15.116:0/2215585842 shutdown_connections 2024-03-04T07:40:30.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.712+0000 7fbe4c400640 1 -- 172.21.15.116:0/2215585842 wait complete. 2024-03-04T07:40:30.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 Processor -- start 2024-03-04T07:40:30.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 -- start start 2024-03-04T07:40:30.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe44074030 0x7fbe4419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44075610 0x7fbe4419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4410dbb0 0x7fbe441a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe44113830 con 0x7fbe4410dbb0 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe441136b0 con 0x7fbe44075610 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4c400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe441139b0 con 0x7fbe44074030 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4410dbb0 0x7fbe441a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe49000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44075610 0x7fbe4419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe49a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe44074030 0x7fbe4419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe49000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44075610 0x7fbe4419e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57026/0 (socket says 172.21.15.116:57026) 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe4a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4410dbb0 0x7fbe441a5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44576/0 (socket says 172.21.15.116:44576) 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.713+0000 7fbe49000640 1 -- 172.21.15.116:0/991132527 learned_addr learned my addr 172.21.15.116:0/991132527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe49000640 1 -- 172.21.15.116:0/991132527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe44074030 msgr2=0x7fbe4419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe49000640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe44074030 0x7fbe4419df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe49000640 1 -- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4410dbb0 msgr2=0x7fbe441a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe49000640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4410dbb0 0x7fbe441a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe49000640 1 -- 172.21.15.116:0/991132527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe441a5b70 con 0x7fbe44075610 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe49000640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44075610 0x7fbe4419e490 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe30000c00 tx=0x7fbe30005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe42000640 1 -- 172.21.15.116:0/991132527 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30041070 con 0x7fbe44075610 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe42000640 1 -- 172.21.15.116:0/991132527 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe300044d0 con 0x7fbe44075610 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe441a5e00 con 0x7fbe44075610 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.714+0000 7fbe42000640 1 -- 172.21.15.116:0/991132527 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30034440 con 0x7fbe44075610 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.715+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe441a6260 con 0x7fbe44075610 2024-03-04T07:40:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.715+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe44103d80 con 0x7fbe44075610 2024-03-04T07:40:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.717+0000 7fbe42000640 1 -- 172.21.15.116:0/991132527 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fbe30044070 con 0x7fbe44075610 2024-03-04T07:40:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.719+0000 7fbe42000640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbe2c0a1d80 0x7fbe2c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.719+0000 7fbe42000640 1 -- 172.21.15.116:0/991132527 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fbe300ef830 con 0x7fbe44075610 2024-03-04T07:40:30.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.719+0000 7fbe49a00640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbe2c0a1d80 0x7fbe2c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:30.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.720+0000 7fbe49a00640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbe2c0a1d80 0x7fbe2c0a4240 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbe38005fd0 tx=0x7fbe38009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:30.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.721+0000 7fbe42000640 1 -- 172.21.15.116:0/991132527 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbe300bc610 con 0x7fbe44075610 2024-03-04T07:40:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.929+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe4402d180 con 0x7fbe44075610 2024-03-04T07:40:30.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.930+0000 7fbe42000640 1 -- 172.21.15.116:0/991132527 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe300c06b0 con 0x7fbe44075610 2024-03-04T07:40:30.931 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.932+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbe2c0a1d80 msgr2=0x7fbe2c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.932+0000 7fbe4c400640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbe2c0a1d80 0x7fbe2c0a4240 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbe38005fd0 tx=0x7fbe38009760 comp rx=0 tx=0).stop 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44075610 msgr2=0x7fbe4419e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44075610 0x7fbe4419e490 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe30000c00 tx=0x7fbe30005d50 comp rx=0 tx=0).stop 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 shutdown_connections 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe4410dbb0 0x7fbe441a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe44075610 0x7fbe4419e490 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbe2c0a1d80 0x7fbe2c0a4240 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 --2- 172.21.15.116:0/991132527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbe44074030 0x7fbe4419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 >> 172.21.15.116:0/991132527 conn(0x7fbe440fcb70 msgr2=0x7fbe44100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:30.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 shutdown_connections 2024-03-04T07:40:30.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:30.933+0000 7fbe4c400640 1 -- 172.21.15.116:0/991132527 wait complete. 2024-03-04T07:40:30.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: pgmap v393: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1737193042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3506766106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:30 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: pgmap v393: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1737193042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3506766106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:31.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:30 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T07:40:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: pgmap v393: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1737193042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:40:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3506766106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:40:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:30 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:31.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 -- 172.21.15.116:0/613399247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c1053c0 msgr2=0x7f9d9c10bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 --2- 172.21.15.116:0/613399247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c10bc50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009ca0 tx=0x7f9d9402c0f0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 -- 172.21.15.116:0/613399247 shutdown_connections 2024-03-04T07:40:31.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 --2- 172.21.15.116:0/613399247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c10bc50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 --2- 172.21.15.116:0/613399247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d9c104a00 0x7f9d9c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 --2- 172.21.15.116:0/613399247 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c103810 0x7f9d9c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 -- 172.21.15.116:0/613399247 >> 172.21.15.116:0/613399247 conn(0x7f9d9c0fedc0 msgr2=0x7f9d9c1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 -- 172.21.15.116:0/613399247 shutdown_connections 2024-03-04T07:40:31.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.152+0000 7f9da3000640 1 -- 172.21.15.116:0/613399247 wait complete. 2024-03-04T07:40:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 Processor -- start 2024-03-04T07:40:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 -- start start 2024-03-04T07:40:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d9c103810 0x7f9d9c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c104a00 0x7f9d9c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c10f490 con 0x7f9d9c103810 2024-03-04T07:40:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c10f310 con 0x7f9d9c1053c0 2024-03-04T07:40:31.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.154+0000 7f9da3000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c10f610 con 0x7f9d9c104a00 2024-03-04T07:40:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9d9be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d9c103810 0x7f9d9c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c1a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57044/0 (socket says 172.21.15.116:57044) 2024-03-04T07:40:31.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 -- 172.21.15.116:0/706351929 learned_addr learned my addr 172.21.15.116:0/706351929 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9d9b400640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c104a00 0x7f9d9c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 -- 172.21.15.116:0/706351929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c104a00 msgr2=0x7f9d9c199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c104a00 0x7f9d9c199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 -- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d9c103810 msgr2=0x7f9d9c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d9c103810 0x7f9d9c199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 -- 172.21.15.116:0/706351929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c1a16c0 con 0x7f9d9c1053c0 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.155+0000 7f9da1000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c1a1050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9d94037040 tx=0x7f9d94032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.156+0000 7f9d98c00640 1 -- 172.21.15.116:0/706351929 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94041020 con 0x7f9d9c1053c0 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.156+0000 7f9d98c00640 1 -- 172.21.15.116:0/706351929 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d94034cd0 con 0x7f9d9c1053c0 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.156+0000 7f9d98c00640 1 -- 172.21.15.116:0/706351929 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94039040 con 0x7f9d9c1053c0 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.156+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9c1a1950 con 0x7f9d9c1053c0 2024-03-04T07:40:31.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.156+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9c1a1e90 con 0x7f9d9c1053c0 2024-03-04T07:40:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.157+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9c10ab30 con 0x7f9d9c1053c0 2024-03-04T07:40:31.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.159+0000 7f9d98c00640 1 -- 172.21.15.116:0/706351929 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9d94044020 con 0x7f9d9c1053c0 2024-03-04T07:40:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.160+0000 7f9d98c00640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d800a1e00 0x7f9d800a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.161+0000 7f9d98c00640 1 -- 172.21.15.116:0/706351929 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9d940efc30 con 0x7f9d9c1053c0 2024-03-04T07:40:31.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.161+0000 7f9d9be00640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d800a1e00 0x7f9d800a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.162+0000 7f9d9be00640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d800a1e00 0x7f9d800a42c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c104860 tx=0x7f9d8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:31.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.165+0000 7f9d98c00640 1 -- 172.21.15.116:0/706351929 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9d9404b020 con 0x7f9d9c1053c0 2024-03-04T07:40:31.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.435+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d9c02cff0 con 0x7f9d9c1053c0 2024-03-04T07:40:31.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.436+0000 7f9d98c00640 1 -- 172.21.15.116:0/706351929 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f9d940bd0f0 con 0x7f9d9c1053c0 2024-03-04T07:40:31.437 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:31.438 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:40:31.438 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:40:31.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d800a1e00 msgr2=0x7f9d800a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d800a1e00 0x7f9d800a42c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c104860 tx=0x7f9d8c005f90 comp rx=0 tx=0).stop 2024-03-04T07:40:31.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c1053c0 msgr2=0x7f9d9c1a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c1a1050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9d94037040 tx=0x7f9d94032f40 comp rx=0 tx=0).stop 2024-03-04T07:40:31.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 shutdown_connections 2024-03-04T07:40:31.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d9c1053c0 0x7f9d9c1a1050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d800a1e00 0x7f9d800a42c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d9c104a00 0x7f9d9c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 --2- 172.21.15.116:0/706351929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d9c103810 0x7f9d9c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 >> 172.21.15.116:0/706351929 conn(0x7f9d9c0fedc0 msgr2=0x7f9d9c107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:31.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 shutdown_connections 2024-03-04T07:40:31.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.438+0000 7f9da3000640 1 -- 172.21.15.116:0/706351929 wait complete. 2024-03-04T07:40:31.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:40:31.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.640+0000 7ffb01400640 1 -- 172.21.15.116:0/3840143756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 msgr2=0x7ffafc104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.640+0000 7ffb01400640 1 --2- 172.21.15.116:0/3840143756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 0x7ffafc104e80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009900 tx=0x7ffae802bc30 comp rx=0 tx=0).stop 2024-03-04T07:40:31.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.641+0000 7ffb01400640 1 -- 172.21.15.116:0/3840143756 shutdown_connections 2024-03-04T07:40:31.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.641+0000 7ffb01400640 1 --2- 172.21.15.116:0/3840143756 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 0x7ffafc10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.641+0000 7ffb01400640 1 --2- 172.21.15.116:0/3840143756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 0x7ffafc104e80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.641+0000 7ffb01400640 1 --2- 172.21.15.116:0/3840143756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffafc103810 0x7ffafc103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.641+0000 7ffb01400640 1 -- 172.21.15.116:0/3840143756 >> 172.21.15.116:0/3840143756 conn(0x7ffafc0fedc0 msgr2=0x7ffafc1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:31.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.641+0000 7ffb01400640 1 -- 172.21.15.116:0/3840143756 shutdown_connections 2024-03-04T07:40:31.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.641+0000 7ffb01400640 1 -- 172.21.15.116:0/3840143756 wait complete. 2024-03-04T07:40:31.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 Processor -- start 2024-03-04T07:40:31.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 -- start start 2024-03-04T07:40:31.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffafc103810 0x7ffafc199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 0x7ffafc19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 0x7ffafc1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc10f4e0 con 0x7ffafc104a00 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc10f360 con 0x7ffafc1053c0 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.642+0000 7ffb01400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffafc10f660 con 0x7ffafc103810 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 0x7ffafc1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 0x7ffafc1a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57052/0 (socket says 172.21.15.116:57052) 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 -- 172.21.15.116:0/4209843620 learned_addr learned my addr 172.21.15.116:0/4209843620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffaf9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 0x7ffafc19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafa800640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffafc103810 0x7ffafc199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 -- 172.21.15.116:0/4209843620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffafc103810 msgr2=0x7ffafc199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffafc103810 0x7ffafc199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 -- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 msgr2=0x7ffafc19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 0x7ffafc19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 -- 172.21.15.116:0/4209843620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc1a17a0 con 0x7ffafc1053c0 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafa800640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffafc103810 0x7ffafc199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffafb200640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 0x7ffafc1a1130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400efd0 tx=0x7ffaf400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffaf3400640 1 -- 172.21.15.116:0/4209843620 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4003b70 con 0x7ffafc1053c0 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffaf3400640 1 -- 172.21.15.116:0/4209843620 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaf4003d10 con 0x7ffafc1053c0 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.643+0000 7ffaf3400640 1 -- 172.21.15.116:0/4209843620 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40109f0 con 0x7ffafc1053c0 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.644+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffafc1a1a90 con 0x7ffafc1053c0 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.644+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffafc1a1f60 con 0x7ffafc1053c0 2024-03-04T07:40:31.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.645+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffafc10ab30 con 0x7ffafc1053c0 2024-03-04T07:40:31.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.647+0000 7ffaf3400640 1 -- 172.21.15.116:0/4209843620 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ffaf4010b90 con 0x7ffafc1053c0 2024-03-04T07:40:31.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.648+0000 7ffaf3400640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffae00a1e00 0x7ffae00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:31.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.649+0000 7ffaf3400640 1 -- 172.21.15.116:0/4209843620 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ffaf40ce230 con 0x7ffafc1053c0 2024-03-04T07:40:31.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.649+0000 7ffafa800640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffae00a1e00 0x7ffae00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:31.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.649+0000 7ffafa800640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffae00a1e00 0x7ffae00a42c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffafc104860 tx=0x7ffaec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:31.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.650+0000 7ffaf3400640 1 -- 172.21.15.116:0/4209843620 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffaf409b620 con 0x7ffafc1053c0 2024-03-04T07:40:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.846+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffafc02d1c0 con 0x7ffafc1053c0 2024-03-04T07:40:31.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.849+0000 7ffaf3400640 1 -- 172.21.15.116:0/4209843620 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7ffaf409af20 con 0x7ffafc1053c0 2024-03-04T07:40:31.850 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:40:31.850 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:40:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffae00a1e00 msgr2=0x7ffae00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffae00a1e00 0x7ffae00a42c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffafc104860 tx=0x7ffaec005f90 comp rx=0 tx=0).stop 2024-03-04T07:40:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 msgr2=0x7ffafc1a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 0x7ffafc1a1130 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400efd0 tx=0x7ffaf400c6a0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 shutdown_connections 2024-03-04T07:40:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffafc1053c0 0x7ffafc1a1130 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffafc104a00 0x7ffafc19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffae00a1e00 0x7ffae00a42c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 --2- 172.21.15.116:0/4209843620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffafc103810 0x7ffafc199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 >> 172.21.15.116:0/4209843620 conn(0x7ffafc0fedc0 msgr2=0x7ffafc107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 shutdown_connections 2024-03-04T07:40:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:31.851+0000 7ffb01400640 1 -- 172.21.15.116:0/4209843620 wait complete. 2024-03-04T07:40:31.868 DEBUG:tasks.fwd_scrub:Yielding 2024-03-04T07:40:31.868 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-04T07:40:31.880 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T07:40:31.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-04T07:40:31.880 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-04T07:40:31.880 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:40:31.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:40:31.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:40:31.886 INFO:teuthology.run_tasks:Running task workunit... 2024-03-04T07:40:31.899 INFO:tasks.workunit:Pulling workunits from ref 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:40:31.900 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-04T07:40:31.900 INFO:tasks.workunit:timeout=3h 2024-03-04T07:40:31.900 INFO:tasks.workunit:cleanup=True 2024-03-04T07:40:31.900 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:31.942 INFO:teuthology.orchestra.run.smithi116.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:31.942 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-04T07:40:31.942 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-04T07:40:31.942 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T07:40:31.942 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-04T07:40:31.942 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-04 07:38:01.063575214 +0000 2024-03-04T07:40:31.942 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-04 07:38:01.063575214 +0000 2024-03-04T07:40:31.943 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-04 07:40:23.323750775 +0000 2024-03-04T07:40:31.943 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-04 07:38:01.063575214 +0000 2024-03-04T07:40:31.943 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-04T07:40:31.943 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-04T07:40:32.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:31 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/991132527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:32.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:31 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/706351929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:32.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:31 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:32.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:31 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:32.042 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:31 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/991132527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:32.043 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:31 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/706351929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:32.043 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:32.043 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:31 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:32.064 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-04T07:40:32.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 -- 172.21.15.116:0/1038294517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a641007a0 msgr2=0x7f9a64100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1038294517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a641007a0 0x7f9a64100c00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009900 tx=0x7f9a5002bc50 comp rx=0 tx=0).stop 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 -- 172.21.15.116:0/1038294517 shutdown_connections 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1038294517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a64101140 0x7f9a641015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1038294517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a641007a0 0x7f9a64100c00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1038294517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a640ffe40 0x7f9a64100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 -- 172.21.15.116:0/1038294517 >> 172.21.15.116:0/1038294517 conn(0x7f9a640fba60 msgr2=0x7f9a640fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 -- 172.21.15.116:0/1038294517 shutdown_connections 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.063+0000 7f9a6be00640 1 -- 172.21.15.116:0/1038294517 wait complete. 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.064+0000 7f9a6be00640 1 Processor -- start 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.064+0000 7f9a6be00640 1 -- start start 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.064+0000 7f9a6be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a640ffe40 0x7f9a64199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a6be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a641007a0 0x7f9a6419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a6be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a64101140 0x7f9a641a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a6be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6410f490 con 0x7f9a641007a0 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a6be00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6410f310 con 0x7f9a640ffe40 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a6be00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6410f610 con 0x7f9a64101140 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a68a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a641007a0 0x7f9a6419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a69400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a640ffe40 0x7f9a64199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a68a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a641007a0 0x7f9a6419a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44634/0 (socket says 172.21.15.116:44634) 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a68a00640 1 -- 172.21.15.116:0/1063150832 learned_addr learned my addr 172.21.15.116:0/1063150832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a69e00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a64101140 0x7f9a641a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a69400640 1 -- 172.21.15.116:0/1063150832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a64101140 msgr2=0x7f9a641a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a69400640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a64101140 0x7f9a641a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a69400640 1 -- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a641007a0 msgr2=0x7f9a6419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a69400640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a641007a0 0x7f9a6419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a69400640 1 -- 172.21.15.116:0/1063150832 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a641a1730 con 0x7f9a640ffe40 2024-03-04T07:40:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.065+0000 7f9a68a00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a641007a0 0x7f9a6419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.066+0000 7f9a69400640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a640ffe40 0x7f9a64199b10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800e9f0 tx=0x7f9a5800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.066+0000 7f9a62000640 1 -- 172.21.15.116:0/1063150832 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5800cd80 con 0x7f9a640ffe40 2024-03-04T07:40:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.066+0000 7f9a62000640 1 -- 172.21.15.116:0/1063150832 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a580035f0 con 0x7f9a640ffe40 2024-03-04T07:40:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.066+0000 7f9a62000640 1 -- 172.21.15.116:0/1063150832 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58010690 con 0x7f9a640ffe40 2024-03-04T07:40:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.066+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a641a1a20 con 0x7f9a640ffe40 2024-03-04T07:40:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.066+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a641a1fc0 con 0x7f9a640ffe40 2024-03-04T07:40:32.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.070+0000 7f9a6ebb6740 1 Processor -- start 2024-03-04T07:40:32.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.070+0000 7f9a6ebb6740 1 -- start start 2024-03-04T07:40:32.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.070+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d59724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.070+0000 7f9a62000640 1 -- 172.21.15.116:0/1063150832 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9a58003b30 con 0x7f9a640ffe40 2024-03-04T07:40:32.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.070+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59775c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d597a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578d5885e20 con 0x5578d5972ac0 2024-03-04T07:40:32.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578d58d7a20 con 0x5578d5977c30 2024-03-04T07:40:32.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578d5880b00 con 0x5578d59720d0 2024-03-04T07:40:32.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a69400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59775c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a68a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d597a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a68a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d597a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57074/0 (socket says 172.21.15.116:57074) 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a68a00640 1 -- 172.21.15.116:0/1771648067 learned_addr learned my addr 172.21.15.116:0/1771648067 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a69e00640 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d59724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a69400640 1 -- 172.21.15.116:0/1771648067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 msgr2=0x5578d59724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a69400640 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d59724f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a69400640 1 -- 172.21.15.116:0/1771648067 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 msgr2=0x5578d597a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a69400640 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d597a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.071+0000 7f9a69400640 1 -- 172.21.15.116:0/1771648067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578d5895e10 con 0x5578d5972ac0 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.072+0000 7f9a69400640 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59775c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9a580cba30 tx=0x7f9a580d64b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447985befa92b30c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.072+0000 7f9a62000640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a4c0a1e00 0x7f9a4c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.072+0000 7f9a56600640 1 -- 172.21.15.116:0/1771648067 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580e1070 con 0x5578d5972ac0 2024-03-04T07:40:32.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.072+0000 7f9a56600640 1 -- 172.21.15.116:0/1771648067 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a580708d0 con 0x5578d5972ac0 2024-03-04T07:40:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.072+0000 7f9a56600640 1 -- 172.21.15.116:0/1771648067 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5809a620 con 0x5578d5972ac0 2024-03-04T07:40:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.072+0000 7f9a62000640 1 -- 172.21.15.116:0/1063150832 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9a580cdf90 con 0x7f9a640ffe40 2024-03-04T07:40:32.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.072+0000 7f9a69e00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a4c0a1e00 0x7f9a4c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/1771648067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 msgr2=0x5578d59775c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a69e00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a4c0a1e00 0x7f9a4c0a42c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5578d5973320 tx=0x7f9a5c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59775c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9a580cba30 tx=0x7f9a580d64b0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/1771648067 shutdown_connections 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d597a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59775c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/1771648067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d59724f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/1771648067 >> 172.21.15.116:0/1771648067 conn(0x5578d596d640 msgr2=0x5578d596da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/1771648067 shutdown_connections 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.073+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/1771648067 wait complete. 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.074+0000 7f9a6ebb6740 1 -- start start 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.074+0000 7f9a6ebb6740 1 Processor -- start 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.074+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d597c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d597c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d59839e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578d5893f40 con 0x5578d5972ac0 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578d58d3ac0 con 0x5578d5977c30 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578d597abe0 con 0x5578d59720d0 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a6ebb6740 10 client.0 _ll_register_callbacks cb 0x5578d58d2aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d59839e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d59839e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57080/0 (socket says 172.21.15.116:57080) 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 -- 172.21.15.116:0/317103839 learned_addr learned my addr 172.21.15.116:0/317103839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a68a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d597c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69400640 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d597c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 -- 172.21.15.116:0/317103839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 msgr2=0x5578d597c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d597c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 -- 172.21.15.116:0/317103839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 msgr2=0x5578d597c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d597c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.075+0000 7f9a69e00640 1 -- 172.21.15.116:0/317103839 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578d56b47e0 con 0x5578d5977c30 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a69e00640 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d59839e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009cd0 tx=0x7f9a5c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9872422408e1f252 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a4aa00640 1 -- 172.21.15.116:0/317103839 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00f830 con 0x5578d5977c30 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a4aa00640 1 -- 172.21.15.116:0/317103839 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9a5c00f9d0 con 0x5578d5977c30 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a580708d0 con 0x5578d5977c30 2024-03-04T07:40:32.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a4aa00640 1 -- 172.21.15.116:0/317103839 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a5c00fbd0 con 0x5578d5977c30 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a6ebb6740 10 client.25241 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.076+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5578d5714f70 con 0x5578d5977c30 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a4aa00640 1 -- 172.21.15.116:0/317103839 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a580708d0 con 0x5578d5977c30 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a4aa00640 1 -- 172.21.15.116:0/317103839 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9a5c026070 con 0x5578d5977c30 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a6ebb6740 10 client.25241 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a6ebb6740 10 client.25241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a6ebb6740 20 client.25241 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d5988280 0x5578d598a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a6ebb6740 4 client.25241 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578d58e4f60 con 0x5578d5988280 2024-03-04T07:40:32.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.077+0000 7f9a69400640 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d5988280 0x5578d598a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.078+0000 7f9a69400640 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d5988280 0x5578d598a6c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c85b18c00802c581 server_cookie=b5fab278e3ce49d5 in_seq=0 out_seq=0 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.078+0000 7f9a4aa00640 10 client.25241 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.078+0000 7f9a4aa00640 1 -- 172.21.15.116:0/317103839 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5578d58e4f60 con 0x5578d5988280 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.078+0000 7f9a4aa00640 10 client.25241 handle_command_reply: tid=0 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.078+0000 7f9a6ebb6740 1 client.25241 shutdown 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 msgr2=0x5578d59839e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d59839e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009cd0 tx=0x7f9a5c004ab0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 shutdown_connections 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d5988280 0x5578d598a6c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d59839e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d597c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/317103839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d597c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a4aa00640 0 client.25241 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 >> 172.21.15.116:0/317103839 conn(0x5578d596d640 msgr2=0x5578d596e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 shutdown_connections 2024-03-04T07:40:32.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/317103839 wait complete. 2024-03-04T07:40:32.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.079+0000 7f9a6ebb6740 20 client.25241 trim_cache size 0 max 16384 2024-03-04T07:40:32.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:31 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/991132527' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:40:32.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:31 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/706351929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:40:32.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:31 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:32.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:31 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:32.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 Processor -- start 2024-03-04T07:40:32.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 -- start start 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d59720d0 0x5578d596c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59839a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d5977c30 0x5578d598ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641a8cb0 con 0x5578d5972ac0 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641ab290 con 0x5578d59720d0 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641a92b0 con 0x5578d5977c30 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a69400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d59720d0 0x5578d596c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a68a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59839a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a69400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d59720d0 0x5578d596c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57088/0 (socket says 172.21.15.116:57088) 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a69400640 1 -- 172.21.15.116:0/669598218 learned_addr learned my addr 172.21.15.116:0/669598218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.096+0000 7f9a69e00640 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d5977c30 0x5578d598ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a69400640 1 -- 172.21.15.116:0/669598218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d5977c30 msgr2=0x5578d598ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a69400640 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d5977c30 0x5578d598ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a69400640 1 -- 172.21.15.116:0/669598218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 msgr2=0x5578d59839a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a69400640 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59839a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a69400640 1 -- 172.21.15.116:0/669598218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578d55ac170 con 0x5578d59720d0 2024-03-04T07:40:32.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a69400640 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d59720d0 0x5578d596c8e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a5808f690 tx=0x7f9a5800c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66bfc409d4c138e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a55c00640 1 -- 172.21.15.116:0/669598218 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58090400 con 0x5578d59720d0 2024-03-04T07:40:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a55c00640 1 -- 172.21.15.116:0/669598218 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a5809b3c0 con 0x5578d59720d0 2024-03-04T07:40:32.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.097+0000 7f9a55c00640 1 -- 172.21.15.116:0/669598218 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58090e30 con 0x5578d59720d0 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/669598218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d59720d0 msgr2=0x5578d596c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d59720d0 0x5578d596c8e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a5808f690 tx=0x7f9a5800c7f0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/669598218 shutdown_connections 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d5977c30 0x5578d598ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d59839a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/669598218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d59720d0 0x5578d596c8e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/669598218 >> 172.21.15.116:0/669598218 conn(0x5578d596d640 msgr2=0x5578d596c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/669598218 shutdown_connections 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.098+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/669598218 wait complete. 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 -- start start 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 Processor -- start 2024-03-04T07:40:32.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d5983330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d5983870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d5981990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641a9910 con 0x5578d5972ac0 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641aac30 con 0x5578d5977c30 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641a9f70 con 0x5578d59720d0 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a6ebb6740 10 client.0 _ll_register_callbacks cb 0x7f9a5c009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a69e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d5981990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a68a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d5983870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a69e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d5981990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57104/0 (socket says 172.21.15.116:57104) 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.099+0000 7f9a69e00640 1 -- 172.21.15.116:0/3692551251 learned_addr learned my addr 172.21.15.116:0/3692551251 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a69400640 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d5983330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a69e00640 1 -- 172.21.15.116:0/3692551251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 msgr2=0x5578d5983330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a69e00640 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d5983330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a69e00640 1 -- 172.21.15.116:0/3692551251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 msgr2=0x5578d5983870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a69e00640 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d5983870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a69e00640 1 -- 172.21.15.116:0/3692551251 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578d59820d0 con 0x5578d5977c30 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a68a00640 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d5983870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a69e00640 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d5981990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009ca0 tx=0x7f9a5c011fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d47a62210b9aaafa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a4aa00640 1 -- 172.21.15.116:0/3692551251 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c011250 con 0x5578d5977c30 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.100+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a5809b3c0 con 0x5578d5977c30 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a4aa00640 1 -- 172.21.15.116:0/3692551251 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9a5c028800 con 0x5578d5977c30 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a4aa00640 1 -- 172.21.15.116:0/3692551251 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a5c042050 con 0x5578d5977c30 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 10 client.25247 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5578d59e4620 con 0x5578d5977c30 2024-03-04T07:40:32.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a4aa00640 1 -- 172.21.15.116:0/3692551251 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a5809b3c0 con 0x5578d5977c30 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a4aa00640 1 -- 172.21.15.116:0/3692551251 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9a5c026040 con 0x5578d5977c30 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 10 client.25247 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 10 client.25247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 20 client.25247 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d59e4f10 0x5578d59e7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 4 client.25247 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.101+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5578d58df2b0 con 0x5578d59e4f10 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.102+0000 7f9a69400640 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d59e4f10 0x5578d59e7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.102+0000 7f9a69400640 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d59e4f10 0x5578d59e7330 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18ad4d89d0fb782 server_cookie=cd8896af4e9ac08c in_seq=0 out_seq=0 2024-03-04T07:40:32.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.102+0000 7f9a4aa00640 10 client.25247 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a4aa00640 1 -- 172.21.15.116:0/3692551251 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5578d58df2b0 con 0x5578d59e4f10 2024-03-04T07:40:32.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a4aa00640 10 client.25247 handle_command_reply: tid=0 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 client.25247 shutdown 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 msgr2=0x5578d5981990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d5981990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009ca0 tx=0x7f9a5c011fb0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 shutdown_connections 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5578d59e4f10 0x5578d59e7330 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5578d5977c30 0x5578d5981990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d5972ac0 0x5578d5983870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a4aa00640 0 client.25247 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 --2- 172.21.15.116:0/3692551251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578d59720d0 0x5578d5983330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.103+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 >> 172.21.15.116:0/3692551251 conn(0x5578d596d640 msgr2=0x5578d5971130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.104+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 shutdown_connections 2024-03-04T07:40:32.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.104+0000 7f9a6ebb6740 1 -- 172.21.15.116:0/3692551251 wait complete. 2024-03-04T07:40:32.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.104+0000 7f9a6ebb6740 20 client.25247 trim_cache size 0 max 16384 2024-03-04T07:40:32.109 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:40:32.109 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:40:32.109 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "9d106acd-4f57-4896-bb18-b905433c1bec", 2024-03-04T07:40:32.109 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:40:32.109 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:40:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.104+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a4c0a1e00 msgr2=0x7f9a4c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.104+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a4c0a1e00 0x7f9a4c0a42c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5578d5973320 tx=0x7f9a5c009500 comp rx=0 tx=0).stop 2024-03-04T07:40:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.104+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a640ffe40 msgr2=0x7f9a64199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.104+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a640ffe40 0x7f9a64199b10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800e9f0 tx=0x7f9a5800eeb0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 shutdown_connections 2024-03-04T07:40:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9a4c0a1e00 0x7f9a4c0a42c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a64101140 0x7f9a641a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a641007a0 0x7f9a6419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 --2- 172.21.15.116:0/1063150832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9a640ffe40 0x7f9a64199b10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 >> 172.21.15.116:0/1063150832 conn(0x7f9a640fba60 msgr2=0x7f9a640fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 shutdown_connections 2024-03-04T07:40:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.105+0000 7f9a6be00640 1 -- 172.21.15.116:0/1063150832 wait complete. 2024-03-04T07:40:32.111 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-04T07:40:32.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:40:32.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:40:32.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 -- 172.21.15.116:0/3843492106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc101f30 msgr2=0x7f9dcc104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 --2- 172.21.15.116:0/3843492106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc101f30 0x7f9dcc104320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009900 tx=0x7f9dc002bc50 comp rx=0 tx=0).stop 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 -- 172.21.15.116:0/3843492106 shutdown_connections 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 --2- 172.21.15.116:0/3843492106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc107190 0x7f9dcc1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 --2- 172.21.15.116:0/3843492106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dcc104860 0x7f9dcc106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 --2- 172.21.15.116:0/3843492106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc101f30 0x7f9dcc104320 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 -- 172.21.15.116:0/3843492106 >> 172.21.15.116:0/3843492106 conn(0x7f9dcc0fb940 msgr2=0x7f9dcc0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 -- 172.21.15.116:0/3843492106 shutdown_connections 2024-03-04T07:40:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.375+0000 7f9dd2800640 1 -- 172.21.15.116:0/3843492106 wait complete. 2024-03-04T07:40:32.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.376+0000 7f9dd2800640 1 Processor -- start 2024-03-04T07:40:32.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.376+0000 7f9dd2800640 1 -- start start 2024-03-04T07:40:32.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dcc101f30 0x7f9dcc1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc104860 0x7f9dcc0ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc107190 0x7f9dcc100420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd2800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc10d190 con 0x7f9dcc107190 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd2800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc10d010 con 0x7f9dcc104860 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd2800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc10d310 con 0x7f9dcc101f30 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc104860 0x7f9dcc0ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc104860 0x7f9dcc0ffee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57130/0 (socket says 172.21.15.116:57130) 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 -- 172.21.15.116:0/2572752670 learned_addr learned my addr 172.21.15.116:0/2572752670 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd1400640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dcc101f30 0x7f9dcc1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd1e00640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc107190 0x7f9dcc100420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 -- 172.21.15.116:0/2572752670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dcc101f30 msgr2=0x7f9dcc1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dcc101f30 0x7f9dcc1018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 -- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc107190 msgr2=0x7f9dcc100420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc107190 0x7f9dcc100420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.377+0000 7f9dd0a00640 1 -- 172.21.15.116:0/2572752670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc100cb0 con 0x7f9dcc104860 2024-03-04T07:40:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.378+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc104860 0x7f9dcc0ffee0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9db8005fd0 tx=0x7f9db800dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.378+0000 7f9dca000640 1 -- 172.21.15.116:0/2572752670 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8017070 con 0x7f9dcc104860 2024-03-04T07:40:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.378+0000 7f9dca000640 1 -- 172.21.15.116:0/2572752670 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9db800de50 con 0x7f9dcc104860 2024-03-04T07:40:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.378+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc1a1920 con 0x7f9dcc104860 2024-03-04T07:40:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.378+0000 7f9dca000640 1 -- 172.21.15.116:0/2572752670 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8003e00 con 0x7f9dcc104860 2024-03-04T07:40:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.378+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc1a1e70 con 0x7f9dcc104860 2024-03-04T07:40:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.381+0000 7f9dca000640 1 -- 172.21.15.116:0/2572752670 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9db8003fa0 con 0x7f9dcc104860 2024-03-04T07:40:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 Processor -- start 2024-03-04T07:40:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 -- start start 2024-03-04T07:40:32.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e9700 0x556c907ee200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907ee870 0x556c907f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c90716090 con 0x556c907e8d10 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c9074dd30 con 0x556c907ee870 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c906f6a30 con 0x556c907e9700 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907ee870 0x556c907f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44682/0 (socket says 172.21.15.116:44682) 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd1e00640 1 -- 172.21.15.116:0/4091964810 learned_addr learned my addr 172.21.15.116:0/4091964810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.382+0000 7f9dd1400640 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e9700 0x556c907ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dd1e00640 1 -- 172.21.15.116:0/4091964810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e9700 msgr2=0x556c907ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dd1e00640 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e9700 0x556c907ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dd1e00640 1 -- 172.21.15.116:0/4091964810 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907ee870 msgr2=0x556c907f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dd1e00640 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907ee870 0x556c907f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dd1e00640 1 -- 172.21.15.116:0/4091964810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c9052c7d0 con 0x556c907e8d10 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907ee870 0x556c907f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dd1e00640 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e9130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009f60 tx=0x7f9dc4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cee386443b92079 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dca000640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db40a1d80 0x7f9db40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dbe600640 1 -- 172.21.15.116:0/4091964810 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc401e070 con 0x556c907e8d10 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dbe600640 1 -- 172.21.15.116:0/4091964810 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc4003c30 con 0x556c907e8d10 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.383+0000 7f9dbe600640 1 -- 172.21.15.116:0/4091964810 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc400fa20 con 0x556c907e8d10 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dca000640 1 -- 172.21.15.116:0/2572752670 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9db80cda10 con 0x7f9dcc104860 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd1400640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db40a1d80 0x7f9db40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd6bad740 1 -- 172.21.15.116:0/4091964810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 msgr2=0x556c907e9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e9130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009f60 tx=0x7f9dc4009f90 comp rx=0 tx=0).stop 2024-03-04T07:40:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd6bad740 1 -- 172.21.15.116:0/4091964810 shutdown_connections 2024-03-04T07:40:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907ee870 0x556c907f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e9700 0x556c907ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/4091964810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e9130 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.384+0000 7f9dd6bad740 1 -- 172.21.15.116:0/4091964810 >> 172.21.15.116:0/4091964810 conn(0x556c907e4280 msgr2=0x556c907e4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.385+0000 7f9dd6bad740 1 -- 172.21.15.116:0/4091964810 shutdown_connections 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.385+0000 7f9dd1400640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db40a1d80 0x7f9db40a4240 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9dc002c260 tx=0x7f9dc0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.385+0000 7f9dd6bad740 1 -- 172.21.15.116:0/4091964810 wait complete. 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.385+0000 7f9dd6bad740 1 -- start start 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.385+0000 7f9dd6bad740 1 Processor -- start 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e8d10 0x556c907f3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e9700 0x556c907f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c907fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c906ec0c0 con 0x556c907e9700 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c90749dd0 con 0x556c907e8d10 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c907f1820 con 0x556c907ee870 2024-03-04T07:40:32.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd6bad740 10 client.0 _ll_register_callbacks cb 0x556c90709860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e8d10 0x556c907f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e9700 0x556c907f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e9700 0x556c907f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44690/0 (socket says 172.21.15.116:44690) 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd0a00640 1 -- 172.21.15.116:0/1392356749 learned_addr learned my addr 172.21.15.116:0/1392356749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.386+0000 7f9dd1e00640 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c907fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd0a00640 1 -- 172.21.15.116:0/1392356749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 msgr2=0x556c907fa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c907fa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd0a00640 1 -- 172.21.15.116:0/1392356749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e8d10 msgr2=0x556c907f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e8d10 0x556c907f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd0a00640 1 -- 172.21.15.116:0/1392356749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c904623f0 con 0x556c907e9700 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e9700 0x556c907f35c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9db8096b60 tx=0x7f9db8009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5422be5f1977015c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9db2a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9db2a00640 1 -- 172.21.15.116:0/1392356749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db809a240 con 0x556c907e9700 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9db2a00640 1 -- 172.21.15.116:0/1392356749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9db8009c30 con 0x556c907e9700 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9dc4003c30 con 0x556c907e9700 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9db2a00640 1 -- 172.21.15.116:0/1392356749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9db809a060 con 0x556c907e9700 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd6bad740 10 client.15150 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.387+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556c904220f0 con 0x556c907e9700 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9db2a00640 1 -- 172.21.15.116:0/1392356749 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc4003c30 con 0x556c907e9700 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9db2a00640 1 -- 172.21.15.116:0/1392356749 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9db8096d70 con 0x556c907e9700 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9dd6bad740 10 client.15150 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9dd6bad740 10 client.15150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9dd6bad740 20 client.15150 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:32.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c907feab0 0x556c90800ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9dd6bad740 4 client.15150 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556c9075b270 con 0x556c907feab0 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.388+0000 7f9dd1400640 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c907feab0 0x556c90800ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.389+0000 7f9dd1400640 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c907feab0 0x556c90800ea0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bac63ef16f9844e server_cookie=445de4da4f4101bd in_seq=0 out_seq=0 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.389+0000 7f9db2a00640 10 client.15150 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.389+0000 7f9db2a00640 1 -- 172.21.15.116:0/1392356749 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556c9075b270 con 0x556c907feab0 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.389+0000 7f9db2a00640 10 client.15150 handle_command_reply: tid=0 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.389+0000 7f9dd6bad740 1 client.15150 shutdown 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e9700 msgr2=0x556c907f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e9700 0x556c907f35c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9db8096b60 tx=0x7f9db8009ed0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 shutdown_connections 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c907feab0 0x556c90800ea0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c907fa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e9700 0x556c907f35c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/1392356749 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e8d10 0x556c907f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 >> 172.21.15.116:0/1392356749 conn(0x556c907e4280 msgr2=0x556c907e34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9db2a00640 0 client.15150 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 shutdown_connections 2024-03-04T07:40:32.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 1 -- 172.21.15.116:0/1392356749 wait complete. 2024-03-04T07:40:32.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.390+0000 7f9dd6bad740 20 client.15150 trim_cache size 0 max 16384 2024-03-04T07:40:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 Processor -- start 2024-03-04T07:40:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 -- start start 2024-03-04T07:40:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c908057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1a8a60 con 0x556c907e8d10 2024-03-04T07:40:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1ab040 con 0x556c907e9700 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1a9060 con 0x556c907ee870 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907fa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57166/0 (socket says 172.21.15.116:57166) 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd0a00640 1 -- 172.21.15.116:0/869606498 learned_addr learned my addr 172.21.15.116:0/869606498 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.407+0000 7f9dd1e00640 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c908057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dd0a00640 1 -- 172.21.15.116:0/869606498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 msgr2=0x556c908057a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c908057a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dd0a00640 1 -- 172.21.15.116:0/869606498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 msgr2=0x556c907e7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dd0a00640 1 -- 172.21.15.116:0/869606498 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c907ea6a0 con 0x556c907e9700 2024-03-04T07:40:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dd1e00640 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c908057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907fa5c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9db80a2970 tx=0x7f9db8004290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f570d531b3aca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dbdc00640 1 -- 172.21.15.116:0/869606498 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8004450 con 0x556c907e9700 2024-03-04T07:40:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dbdc00640 1 -- 172.21.15.116:0/869606498 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9db8015070 con 0x556c907e9700 2024-03-04T07:40:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.408+0000 7f9dbdc00640 1 -- 172.21.15.116:0/869606498 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80969a0 con 0x556c907e9700 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.409+0000 7f9dd6bad740 1 -- 172.21.15.116:0/869606498 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 msgr2=0x556c907fa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.409+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907fa5c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9db80a2970 tx=0x7f9db8004290 comp rx=0 tx=0).stop 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 -- 172.21.15.116:0/869606498 shutdown_connections 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907ee870 0x556c908057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907fa5c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/869606498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907e8d10 0x556c907e7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 -- 172.21.15.116:0/869606498 >> 172.21.15.116:0/869606498 conn(0x556c907e4280 msgr2=0x556c907e74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 -- 172.21.15.116:0/869606498 shutdown_connections 2024-03-04T07:40:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 -- 172.21.15.116:0/869606498 wait complete. 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 -- start start 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.410+0000 7f9dd6bad740 1 Processor -- start 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e8d10 0x556c907f8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd6bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907ee870 0x556c907f99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1a96c0 con 0x556c907ee870 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1aa9e0 con 0x556c907e9700 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd6bad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc1a9d20 con 0x556c907e8d10 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd6bad740 10 client.0 _ll_register_callbacks cb 0x7f9db80962e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907ee870 0x556c907f99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:57180/0 (socket says 172.21.15.116:57180) 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 -- 172.21.15.116:0/3276437296 learned_addr learned my addr 172.21.15.116:0/3276437296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd1400640 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e8d10 0x556c907f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 -- 172.21.15.116:0/3276437296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e8d10 msgr2=0x556c907f8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e8d10 0x556c907f8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 -- 172.21.15.116:0/3276437296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907ee870 msgr2=0x556c907f99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907ee870 0x556c907f99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 -- 172.21.15.116:0/3276437296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c907f9fb0 con 0x556c907e9700 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9dd0a00640 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907f89b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9db8096b60 tx=0x7f9db809a770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8d785aa37b5ad94 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.411+0000 7f9db2a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9db2a00640 1 -- 172.21.15.116:0/3276437296 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80b7020 con 0x556c907e9700 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9db2a00640 1 -- 172.21.15.116:0/3276437296 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9db8099b50 con 0x556c907e9700 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9db8015070 con 0x556c907e9700 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9db2a00640 1 -- 172.21.15.116:0/3276437296 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9db80969a0 con 0x556c907e9700 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 10 client.25265 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556c9085aae0 con 0x556c907e9700 2024-03-04T07:40:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9db2a00640 1 -- 172.21.15.116:0/3276437296 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9db8015070 con 0x556c907e9700 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9db2a00640 1 -- 172.21.15.116:0/3276437296 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9db80995c0 con 0x556c907e9700 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 10 client.25265 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 10 client.25265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 20 client.25265 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c9085b240 0x556c9085d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 4 client.25265 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.412+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556c907555c0 con 0x556c9085b240 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd1400640 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c9085b240 0x556c9085d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd1400640 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c9085b240 0x556c9085d660 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e6dc93179b0ddac server_cookie=b8cfeb3cd17d0165 in_seq=0 out_seq=0 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9db2a00640 10 client.25265 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9db2a00640 1 -- 172.21.15.116:0/3276437296 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556c907555c0 con 0x556c9085b240 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9db2a00640 10 client.25265 handle_command_reply: tid=0 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 client.25265 shutdown 2024-03-04T07:40:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 msgr2=0x556c907f89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907f89b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9db8096b60 tx=0x7f9db809a770 comp rx=0 tx=0).stop 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 shutdown_connections 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556c9085b240 0x556c9085d660 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556c907ee870 0x556c907f99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556c907e9700 0x556c907f89b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 --2- 172.21.15.116:0/3276437296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c907e8d10 0x556c907f8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9db2a00640 0 client.25265 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.413+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 >> 172.21.15.116:0/3276437296 conn(0x556c907e4280 msgr2=0x556c907e7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 shutdown_connections 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd6bad740 1 -- 172.21.15.116:0/3276437296 wait complete. 2024-03-04T07:40:32.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd6bad740 20 client.25265 trim_cache size 0 max 16384 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: "9d106acd-4f57-4896-bb18-b905433c1bec": { 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "9d106acd-4f57-4896-bb18-b905433c1bec", 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:40:32.420 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:40:32.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db40a1d80 msgr2=0x7f9db40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db40a1d80 0x7f9db40a4240 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9dc002c260 tx=0x7f9dc0000bc0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc104860 msgr2=0x7f9dcc0ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:32.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc104860 0x7f9dcc0ffee0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9db8005fd0 tx=0x7f9db800dd20 comp rx=0 tx=0).stop 2024-03-04T07:40:32.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 shutdown_connections 2024-03-04T07:40:32.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dcc107190 0x7f9dcc100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dcc104860 0x7f9dcc0ffee0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db40a1d80 0x7f9db40a4240 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 --2- 172.21.15.116:0/2572752670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dcc101f30 0x7f9dcc1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 >> 172.21.15.116:0/2572752670 conn(0x7f9dcc0fb940 msgr2=0x7f9dcc102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 shutdown_connections 2024-03-04T07:40:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:40:32.414+0000 7f9dd2800640 1 -- 172.21.15.116:0/2572752670 wait complete. 2024-03-04T07:40:32.430 INFO:tasks.cephfs.filesystem:scrub status for tag:9d106acd-4f57-4896-bb18-b905433c1bec - {'path': '/', 'tag': '9d106acd-4f57-4896-bb18-b905433c1bec', 'options': 'recursive,force'} 2024-03-04T07:40:33.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:32 smithi152 ceph-mon[37154]: pgmap v394: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:33.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:32 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4209843620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:33.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:32 smithi133 ceph-mon[36505]: pgmap v394: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:33.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:32 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4209843620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:32 smithi116 ceph-mon[32150]: pgmap v394: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:40:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:32 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4209843620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:40:34.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:40:34.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:40:34.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:40:34.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:40:34.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:40:34.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:40:34.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:40:34.022 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:40:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:40:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:40:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:40:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:40:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:40:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:40:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:40:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:40:35.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:34 smithi133 ceph-mon[36505]: pgmap v395: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-04T07:40:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:34 smithi116 ceph-mon[32150]: pgmap v395: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-04T07:40:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:34 smithi152 ceph-mon[37154]: pgmap v395: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-04T07:40:37.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:36 smithi133 ceph-mon[36505]: pgmap v396: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-04T07:40:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:36 smithi116 ceph-mon[32150]: pgmap v396: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-04T07:40:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:36 smithi152 ceph-mon[37154]: pgmap v396: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-04T07:40:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:38 smithi133 ceph-mon[36505]: pgmap v397: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-04T07:40:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:38 smithi116 ceph-mon[32150]: pgmap v397: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-04T07:40:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:38 smithi152 ceph-mon[37154]: pgmap v397: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-04T07:40:41.073 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:40 smithi152 ceph-mon[37154]: pgmap v398: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-04T07:40:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:40 smithi133 ceph-mon[36505]: pgmap v398: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-04T07:40:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:40 smithi116 ceph-mon[32150]: pgmap v398: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-04T07:40:43.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:42 smithi133 ceph-mon[36505]: pgmap v399: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-04T07:40:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:42 smithi116 ceph-mon[32150]: pgmap v399: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-04T07:40:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:42 smithi152 ceph-mon[37154]: pgmap v399: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-04T07:40:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:44 smithi133 ceph-mon[36505]: pgmap v400: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-04T07:40:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 -- 172.21.15.116:0/1947965060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081071e0 msgr2=0x7f06081095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 --2- 172.21.15.116:0/1947965060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081071e0 0x7f06081095d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0600009ca0 tx=0x7f060002c100 comp rx=0 tx=0).stop 2024-03-04T07:40:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 -- 172.21.15.116:0/1947965060 shutdown_connections 2024-03-04T07:40:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 --2- 172.21.15.116:0/1947965060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081071e0 0x7f06081095d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 --2- 172.21.15.116:0/1947965060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06081048b0 0x7f0608106ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 --2- 172.21.15.116:0/1947965060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0608101f80 0x7f0608104370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 -- 172.21.15.116:0/1947965060 >> 172.21.15.116:0/1947965060 conn(0x7f06080fb8d0 msgr2=0x7f06080fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 -- 172.21.15.116:0/1947965060 shutdown_connections 2024-03-04T07:40:45.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.120+0000 7f060d600640 1 -- 172.21.15.116:0/1947965060 wait complete. 2024-03-04T07:40:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.121+0000 7f060d600640 1 Processor -- start 2024-03-04T07:40:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.121+0000 7f060d600640 1 -- start start 2024-03-04T07:40:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f060d600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0608101f80 0x7f0608197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f060d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081048b0 0x7f0608197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f060d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06081071e0 0x7f060819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f060d600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060810d0e0 con 0x7f06081071e0 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f060d600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060810cf60 con 0x7f06081048b0 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f060d600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060810d260 con 0x7f0608101f80 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f0607400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081048b0 0x7f0608197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f0607400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081048b0 0x7f0608197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60072/0 (socket says 172.21.15.116:60072) 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f0607400640 1 -- 172.21.15.116:0/3999172471 learned_addr learned my addr 172.21.15.116:0/3999172471 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f0607e00640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0608101f80 0x7f0608197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.122+0000 7f060cc00640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06081071e0 0x7f060819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0607400640 1 -- 172.21.15.116:0/3999172471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0608101f80 msgr2=0x7f0608197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0607400640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0608101f80 0x7f0608197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0607400640 1 -- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06081071e0 msgr2=0x7f060819ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0607400640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06081071e0 0x7f060819ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0607400640 1 -- 172.21.15.116:0/3999172471 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060819f320 con 0x7f06081048b0 2024-03-04T07:40:45.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0607e00640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0608101f80 0x7f0608197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0607400640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081048b0 0x7f0608197c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05f400d770 tx=0x7f05f400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0604c00640 1 -- 172.21.15.116:0/3999172471 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4017070 con 0x7f06081048b0 2024-03-04T07:40:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0604c00640 1 -- 172.21.15.116:0/3999172471 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f4009c30 con 0x7f06081048b0 2024-03-04T07:40:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f0604c00640 1 -- 172.21.15.116:0/3999172471 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f400f620 con 0x7f06081048b0 2024-03-04T07:40:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.123+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060819f610 con 0x7f06081048b0 2024-03-04T07:40:45.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.124+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060819fb10 con 0x7f06081048b0 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 Processor -- start 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 -- start start 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acab3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acab8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acabba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aca19370 con 0x5649acab3a10 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ac9c2550 con 0x5649acab9570 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f061184e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aca15410 con 0x5649acab4400 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f060cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acab3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f060cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acab3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55372/0 (socket says 172.21.15.116:55372) 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.127+0000 7f060cc00640 1 -- 172.21.15.116:0/3466719322 learned_addr learned my addr 172.21.15.116:0/3466719322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f0607e00640 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acab8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f0604c00640 1 -- 172.21.15.116:0/3999172471 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f05f401a070 con 0x7f06081048b0 2024-03-04T07:40:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f060cc00640 1 -- 172.21.15.116:0/3466719322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 msgr2=0x5649acab8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f060cc00640 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acab8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f060cc00640 1 -- 172.21.15.116:0/3466719322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 msgr2=0x5649acabba00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T07:40:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f060cc00640 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acabba00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f060cc00640 1 -- 172.21.15.116:0/3466719322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649ac9d63c0 con 0x5649acab3a10 2024-03-04T07:40:45.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f060cc00640 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acab3e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f060002c710 tx=0x7f0600005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6cb754453e9f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.128+0000 7f05fd200640 1 -- 172.21.15.116:0/3466719322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600043070 con 0x5649acab3a10 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f05fd200640 1 -- 172.21.15.116:0/3466719322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0600003970 con 0x5649acab3a10 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f05fd200640 1 -- 172.21.15.116:0/3466719322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600039040 con 0x5649acab3a10 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 -- 172.21.15.116:0/3466719322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 msgr2=0x5649acab3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acab3e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f060002c710 tx=0x7f0600005fa0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 -- 172.21.15.116:0/3466719322 shutdown_connections 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acabba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acab8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 --2- 172.21.15.116:0/3466719322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acab3e30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 -- 172.21.15.116:0/3466719322 >> 172.21.15.116:0/3466719322 conn(0x5649acaaef80 msgr2=0x5649acaaf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 -- 172.21.15.116:0/3466719322 shutdown_connections 2024-03-04T07:40:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.129+0000 7f061184e740 1 -- 172.21.15.116:0/3466719322 wait complete. 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.130+0000 7f0604c00640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05ec0a1d00 0x7f05ec0a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.130+0000 7f0604c00640 1 -- 172.21.15.116:0/3999172471 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f05f40cd7c0 con 0x7f06081048b0 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.130+0000 7f0607e00640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05ec0a1d00 0x7f05ec0a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.130+0000 7f061184e740 1 -- start start 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.130+0000 7f061184e740 1 Processor -- start 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.130+0000 7f0607e00640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05ec0a1d00 0x7f05ec0a41c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5649acaafef0 tx=0x7f05f800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f061184e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab3a10 0x5649acabdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f061184e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab4400 0x5649acabe2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f061184e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f061184e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649acabc4c0 con 0x5649acab4400 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f061184e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649ac9d59b0 con 0x5649acab9570 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f061184e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649acabc640 con 0x5649acab3a10 2024-03-04T07:40:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f061184e740 10 client.0 _ll_register_callbacks cb 0x5649aca11340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab3a10 0x5649acabdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab4400 0x5649acabe2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f060cc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab3a10 0x5649acabdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:49152/0 (socket says 172.21.15.116:49152) 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f060cc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60100/0 (socket says 172.21.15.116:60100) 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab4400 0x5649acabe2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55378/0 (socket says 172.21.15.116:55378) 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 -- 172.21.15.116:0/1510763164 learned_addr learned my addr 172.21.15.116:0/1510763164 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 -- 172.21.15.116:0/1510763164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 msgr2=0x5649acac52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 -- 172.21.15.116:0/1510763164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab4400 msgr2=0x5649acabe2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab4400 0x5649acabe2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.131+0000 7f0607e00640 1 -- 172.21.15.116:0/1510763164 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649ac72e210 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f0607e00640 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab3a10 0x5649acabdd60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05f8005d60 tx=0x7f05f8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78b6c091e9602445 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f05f1600640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f05f1600640 1 -- 172.21.15.116:0/1510763164 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f801fe00 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0600003970 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f05f1600640 1 -- 172.21.15.116:0/1510763164 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f05f801f260 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f05f1600640 1 -- 172.21.15.116:0/1510763164 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f80109d0 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f061184e740 10 client.25261 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.132+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5649ac7fa620 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f05f1600640 1 -- 172.21.15.116:0/1510763164 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0600003970 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f05f1600640 1 -- 172.21.15.116:0/1510763164 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f05f8026070 con 0x5649acab3a10 2024-03-04T07:40:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f061184e740 10 client.25261 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f061184e740 10 client.25261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f061184e740 20 client.25261 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f061184e740 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acac9bd0 0x5649acacbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f061184e740 4 client.25261 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:40:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649aca268b0 con 0x5649acac9bd0 2024-03-04T07:40:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.133+0000 7f0607400640 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acac9bd0 0x5649acacbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.134+0000 7f0607400640 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acac9bd0 0x5649acacbfc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd1f37bea48fa965 server_cookie=f16a3956a40fa045 in_seq=0 out_seq=0 2024-03-04T07:40:45.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.134+0000 7f05f1600640 10 client.25261 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:45.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f05f1600640 1 -- 172.21.15.116:0/1510763164 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649aca268b0 con 0x5649acac9bd0 2024-03-04T07:40:45.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f05f1600640 10 client.25261 handle_command_reply: tid=0 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 client.25261 shutdown 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab3a10 msgr2=0x5649acabdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab3a10 0x5649acabdd60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05f8005d60 tx=0x7f05f8004730 comp rx=0 tx=0).stop 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 shutdown_connections 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acac9bd0 0x5649acacbfc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab4400 0x5649acabe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f05f1600640 0 client.25261 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 --2- 172.21.15.116:0/1510763164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab3a10 0x5649acabdd60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 >> 172.21.15.116:0/1510763164 conn(0x5649acaaef80 msgr2=0x5649acaae200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 shutdown_connections 2024-03-04T07:40:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 1 -- 172.21.15.116:0/1510763164 wait complete. 2024-03-04T07:40:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.135+0000 7f061184e740 20 client.25261 trim_cache size 0 max 16384 2024-03-04T07:40:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.150+0000 7f061184e740 1 Processor -- start 2024-03-04T07:40:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.150+0000 7f061184e740 1 -- start start 2024-03-04T07:40:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f061184e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab3a10 0x5649acaad970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f061184e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f061184e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab9570 0x5649acaadeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f061184e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06081a6960 con 0x5649acab9570 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f061184e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06081a8f40 con 0x5649acab3a10 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f061184e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06081a6f60 con 0x5649acab4400 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f060cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab9570 0x5649acaadeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab3a10 0x5649acaad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab3a10 0x5649acaad970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60104/0 (socket says 172.21.15.116:60104) 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 -- 172.21.15.116:0/3829816292 learned_addr learned my addr 172.21.15.116:0/3829816292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607400640 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 -- 172.21.15.116:0/3829816292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 msgr2=0x5649acac5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 -- 172.21.15.116:0/3829816292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab9570 msgr2=0x5649acaadeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab9570 0x5649acaadeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f0607e00640 1 -- 172.21.15.116:0/3829816292 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649acacc330 con 0x5649acab3a10 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.151+0000 7f060cc00640 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab9570 0x5649acaadeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.152+0000 7f0607e00640 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab3a10 0x5649acaad970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f05f8028e30 tx=0x7f05f80059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c170d3e4270e42d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.152+0000 7f05f3e00640 1 -- 172.21.15.116:0/3829816292 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f801f2e0 con 0x5649acab3a10 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.152+0000 7f05f3e00640 1 -- 172.21.15.116:0/3829816292 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f8010b70 con 0x5649acab3a10 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.152+0000 7f05f3e00640 1 -- 172.21.15.116:0/3829816292 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80288d0 con 0x5649acab3a10 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.152+0000 7f061184e740 1 -- 172.21.15.116:0/3829816292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab3a10 msgr2=0x5649acaad970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.152+0000 7f061184e740 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab3a10 0x5649acaad970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f05f8028e30 tx=0x7f05f80059b0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 -- 172.21.15.116:0/3829816292 shutdown_connections 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab9570 0x5649acaadeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 --2- 172.21.15.116:0/3829816292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab3a10 0x5649acaad970 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 -- 172.21.15.116:0/3829816292 >> 172.21.15.116:0/3829816292 conn(0x5649acaaef80 msgr2=0x5649acaaf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 -- 172.21.15.116:0/3829816292 shutdown_connections 2024-03-04T07:40:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 -- 172.21.15.116:0/3829816292 wait complete. 2024-03-04T07:40:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 -- start start 2024-03-04T07:40:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.153+0000 7f061184e740 1 Processor -- start 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f061184e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acac2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f061184e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f061184e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f061184e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06081a75c0 con 0x5649acab3a10 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f061184e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06081a88e0 con 0x5649acab9570 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f061184e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06081a7c20 con 0x5649acab4400 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f061184e740 10 client.0 _ll_register_callbacks cb 0x7f05f800bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acac2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acac2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55402/0 (socket says 172.21.15.116:55402) 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f060cc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 -- 172.21.15.116:0/1473115682 learned_addr learned my addr 172.21.15.116:0/1473115682 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607400640 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 -- 172.21.15.116:0/1473115682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 msgr2=0x5649acac3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 -- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 msgr2=0x5649acac4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f0607e00640 1 -- 172.21.15.116:0/1473115682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649acac4a60 con 0x5649acab3a10 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.154+0000 7f060cc00640 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f0607e00640 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acac2f20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05f8000c00 tx=0x7f05f8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3752252706c26e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f05f1600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f05f1600640 1 -- 172.21.15.116:0/1473115682 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f8041030 con 0x5649acab3a10 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f05f1600640 1 -- 172.21.15.116:0/1473115682 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f05f8039050 con 0x5649acab3a10 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05f8010b70 con 0x5649acab3a10 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f05f1600640 1 -- 172.21.15.116:0/1473115682 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f804d050 con 0x5649acab3a10 2024-03-04T07:40:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f061184e740 10 client.15174 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5649acb25a60 con 0x5649acab3a10 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f05f1600640 1 -- 172.21.15.116:0/1473115682 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f8010b70 con 0x5649acab3a10 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.155+0000 7f05f1600640 1 -- 172.21.15.116:0/1473115682 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f05f8026040 con 0x5649acab3a10 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f061184e740 10 client.15174 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f061184e740 10 client.15174 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f061184e740 20 client.15174 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f061184e740 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acb26330 0x5649acb28750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f061184e740 4 client.15174 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5649aca20c00 con 0x5649acb26330 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f0607400640 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acb26330 0x5649acb28750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f0607400640 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acb26330 0x5649acb28750 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9ee39eca684d803 server_cookie=f8c285d8f2bc8d07 in_seq=0 out_seq=0 2024-03-04T07:40:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.156+0000 7f05f1600640 10 client.15174 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f05f1600640 1 -- 172.21.15.116:0/1473115682 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5649aca20c00 con 0x5649acb26330 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f05f1600640 10 client.15174 handle_command_reply: tid=0 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 client.15174 shutdown 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 msgr2=0x5649acac2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acac2f20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05f8000c00 tx=0x7f05f8016000 comp rx=0 tx=0).stop 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 shutdown_connections 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5649acb26330 0x5649acb28750 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5649acab9570 0x5649acac4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649acab4400 0x5649acac3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 --2- 172.21.15.116:0/1473115682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649acab3a10 0x5649acac2f20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f05f1600640 0 client.15174 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.167+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 >> 172.21.15.116:0/1473115682 conn(0x5649acaaef80 msgr2=0x5649acab21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.168+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 shutdown_connections 2024-03-04T07:40:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.168+0000 7f061184e740 1 -- 172.21.15.116:0/1473115682 wait complete. 2024-03-04T07:40:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.168+0000 7f061184e740 20 client.15174 trim_cache size 0 max 16384 2024-03-04T07:40:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:40:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:40:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:40:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.168+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05ec0a1d00 msgr2=0x7f05ec0a41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.168+0000 7f060d600640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05ec0a1d00 0x7f05ec0a41c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5649acaafef0 tx=0x7f05f800b390 comp rx=0 tx=0).stop 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.168+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081048b0 msgr2=0x7f0608197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081048b0 0x7f0608197c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05f400d770 tx=0x7f05f400dc30 comp rx=0 tx=0).stop 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 shutdown_connections 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06081071e0 0x7f060819ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f06081048b0 0x7f0608197c40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f05ec0a1d00 0x7f05ec0a41c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 --2- 172.21.15.116:0/3999172471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0608101f80 0x7f0608197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 >> 172.21.15.116:0/3999172471 conn(0x7f06080fb8d0 msgr2=0x7f06081028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 shutdown_connections 2024-03-04T07:40:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:45.169+0000 7f060d600640 1 -- 172.21.15.116:0/3999172471 wait complete. 2024-03-04T07:40:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:44 smithi116 ceph-mon[32150]: pgmap v400: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-04T07:40:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:44 smithi152 ceph-mon[37154]: pgmap v400: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-04T07:40:47.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:46 smithi133 ceph-mon[36505]: pgmap v401: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:40:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:46 smithi116 ceph-mon[32150]: pgmap v401: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:40:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:46 smithi152 ceph-mon[37154]: pgmap v401: 97 pgs: 97 active+clean; 648 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:40:49.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:48 smithi133 ceph-mon[36505]: pgmap v402: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-04T07:40:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:48 smithi116 ceph-mon[32150]: pgmap v402: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-04T07:40:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:48 smithi152 ceph-mon[37154]: pgmap v402: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-04T07:40:51.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:50 smithi133 ceph-mon[36505]: pgmap v403: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-04T07:40:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:50 smithi116 ceph-mon[32150]: pgmap v403: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-04T07:40:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:50 smithi152 ceph-mon[37154]: pgmap v403: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-04T07:40:53.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:52 smithi133 ceph-mon[36505]: pgmap v404: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:52 smithi116 ceph-mon[32150]: pgmap v404: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:52 smithi152 ceph-mon[37154]: pgmap v404: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:54 smithi116 ceph-mon[32150]: pgmap v405: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:54 smithi152 ceph-mon[37154]: pgmap v405: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:55.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:54 smithi133 ceph-mon[36505]: pgmap v405: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:56.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:55 smithi116 ceph-mon[32150]: pgmap v406: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:56.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:55 smithi152 ceph-mon[37154]: pgmap v406: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:56.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:55 smithi133 ceph-mon[36505]: pgmap v406: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T07:40:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 -- 172.21.15.116:0/3663125087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 msgr2=0x7f1470104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 --2- 172.21.15.116:0/3663125087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 0x7f1470104ce0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f145c009860 tx=0x7f145c02bb70 comp rx=0 tx=0).stop 2024-03-04T07:40:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 -- 172.21.15.116:0/3663125087 shutdown_connections 2024-03-04T07:40:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 --2- 172.21.15.116:0/3663125087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1470105220 0x7f147010bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 --2- 172.21.15.116:0/3663125087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 0x7f1470104ce0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 --2- 172.21.15.116:0/3663125087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 0x7f1470103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 -- 172.21.15.116:0/3663125087 >> 172.21.15.116:0/3663125087 conn(0x7f14700febe0 msgr2=0x7f1470101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 -- 172.21.15.116:0/3663125087 shutdown_connections 2024-03-04T07:40:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.309+0000 7f1476000640 1 -- 172.21.15.116:0/3663125087 wait complete. 2024-03-04T07:40:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 Processor -- start 2024-03-04T07:40:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 -- start start 2024-03-04T07:40:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 0x7f1470199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 0x7f1470199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1470105220 0x7f14701a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010f2e0 con 0x7f1470104860 2024-03-04T07:40:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010f160 con 0x7f1470103670 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.310+0000 7f1476000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010f460 con 0x7f1470105220 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 0x7f1470199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 0x7f1470199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48092/0 (socket says 172.21.15.116:48092) 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 -- 172.21.15.116:0/1247522285 learned_addr learned my addr 172.21.15.116:0/1247522285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f146fe00640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 0x7f1470199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1475600640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1470105220 0x7f14701a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 -- 172.21.15.116:0/1247522285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1470105220 msgr2=0x7f14701a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1470105220 0x7f14701a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 -- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 msgr2=0x7f1470199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 0x7f1470199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 -- 172.21.15.116:0/1247522285 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14701a1570 con 0x7f1470103670 2024-03-04T07:40:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f1474c00640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 0x7f1470199950 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f146400e9a0 tx=0x7f146400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.311+0000 7f146d600640 1 -- 172.21.15.116:0/1247522285 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400cd80 con 0x7f1470103670 2024-03-04T07:40:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.312+0000 7f146d600640 1 -- 172.21.15.116:0/1247522285 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1464010910 con 0x7f1470103670 2024-03-04T07:40:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.312+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14701a1860 con 0x7f1470103670 2024-03-04T07:40:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.312+0000 7f146d600640 1 -- 172.21.15.116:0/1247522285 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464018a30 con 0x7f1470103670 2024-03-04T07:40:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.312+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701a1dd0 con 0x7f1470103670 2024-03-04T07:40:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 Processor -- start 2024-03-04T07:40:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 -- start start 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef155ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef155ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1562a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ef14c0370 con 0x562ef1560570 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ef1469550 con 0x562ef155aa10 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f147a37e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ef14bc410 con 0x562ef155b400 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f146d600640 1 -- 172.21.15.116:0/1247522285 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1464010ac0 con 0x7f1470103670 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.316+0000 7f1475600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef155ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef155ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48098/0 (socket says 172.21.15.116:48098) 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 -- 172.21.15.116:0/723507403 learned_addr learned my addr 172.21.15.116:0/723507403 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f146fe00640 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1562a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 -- 172.21.15.116:0/723507403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 msgr2=0x562ef155ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef155ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 -- 172.21.15.116:0/723507403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 msgr2=0x562ef1562a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1562a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 -- 172.21.15.116:0/723507403 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ef147d3c0 con 0x562ef155aa10 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1475600640 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef155ae30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1468009fc0 tx=0x7f1468010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1a1be27fd132924 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1461c00640 1 -- 172.21.15.116:0/723507403 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468003b70 con 0x562ef155aa10 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1461c00640 1 -- 172.21.15.116:0/723507403 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1468003d10 con 0x562ef155aa10 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.317+0000 7f1461c00640 1 -- 172.21.15.116:0/723507403 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468011b60 con 0x562ef155aa10 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 -- 172.21.15.116:0/723507403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 msgr2=0x562ef155ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef155ae30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1468009fc0 tx=0x7f1468010860 comp rx=0 tx=0).stop 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f146d600640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14580a1df0 0x7f14580a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 -- 172.21.15.116:0/723507403 shutdown_connections 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1562a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef155ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 --2- 172.21.15.116:0/723507403 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef155ae30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 -- 172.21.15.116:0/723507403 >> 172.21.15.116:0/723507403 conn(0x562ef1555f80 msgr2=0x562ef1556380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f146d600640 1 -- 172.21.15.116:0/1247522285 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f14640ce180 con 0x7f1470103670 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 -- 172.21.15.116:0/723507403 shutdown_connections 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f1474c00640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14580a1df0 0x7f14580a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.318+0000 7f147a37e740 1 -- 172.21.15.116:0/723507403 wait complete. 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.319+0000 7f147a37e740 1 -- start start 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.319+0000 7f1474c00640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14580a1df0 0x7f14580a42b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f146409ae00 tx=0x7f1464010e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:58.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.319+0000 7f147a37e740 1 Processor -- start 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.319+0000 7f147a37e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef1564d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.319+0000 7f147a37e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef1565270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f147a37e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f147a37e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ef1563520 con 0x562ef1560570 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f147a37e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ef147c9b0 con 0x562ef155aa10 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f147a37e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ef15636a0 con 0x562ef155b400 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f147a37e740 10 client.0 _ll_register_callbacks cb 0x562ef14b8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f1475600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef1564d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f146fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef1565270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f1475600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef1564d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48112/0 (socket says 172.21.15.116:48112) 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f1475600640 1 -- 172.21.15.116:0/1714236407 learned_addr learned my addr 172.21.15.116:0/1714236407 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f1475600640 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f146fe00640 1 -- 172.21.15.116:0/1714236407 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 msgr2=0x562ef1564d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f146fe00640 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef1564d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f146fe00640 1 -- 172.21.15.116:0/1714236407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 msgr2=0x562ef156c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f146fe00640 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.320+0000 7f146fe00640 1 -- 172.21.15.116:0/1714236407 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ef11d5210 con 0x562ef155b400 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.321+0000 7f146fe00640 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef1565270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f145c02c0b0 tx=0x7f145c009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f89ee81b2ece02d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.321+0000 7f1456000640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.321+0000 7f1456000640 1 -- 172.21.15.116:0/1714236407 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c043070 con 0x562ef155b400 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.321+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1468003d10 con 0x562ef155b400 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.321+0000 7f1456000640 1 -- 172.21.15.116:0/1714236407 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f145c004040 con 0x562ef155b400 2024-03-04T07:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.321+0000 7f1456000640 1 -- 172.21.15.116:0/1714236407 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f145c04f050 con 0x562ef155b400 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 10 client.25276 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562ef12a1620 con 0x562ef155b400 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f1456000640 1 -- 172.21.15.116:0/1714236407 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1468003d10 con 0x562ef155b400 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f1456000640 1 -- 172.21.15.116:0/1714236407 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f145c032640 con 0x562ef155b400 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 10 client.25276 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 10 client.25276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 20 client.25276 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef1570b90 0x562ef1572f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 4 client.25276 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ef14cd8b0 con 0x562ef1570b90 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.322+0000 7f1475600640 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef1570b90 0x562ef1572f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.323+0000 7f1475600640 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef1570b90 0x562ef1572f80 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4e4ae93af73ae0f server_cookie=ecd952022bcc7fa9 in_seq=0 out_seq=0 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.323+0000 7f1456000640 10 client.25276 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f1456000640 1 -- 172.21.15.116:0/1714236407 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562ef14cd8b0 con 0x562ef1570b90 2024-03-04T07:40:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f1456000640 10 client.25276 handle_command_reply: tid=0 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 client.25276 shutdown 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 msgr2=0x562ef1565270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef1565270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f145c02c0b0 tx=0x7f145c009490 comp rx=0 tx=0).stop 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 shutdown_connections 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef1570b90 0x562ef1572f80 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f1456000640 0 client.25276 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155b400 0x562ef1565270 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 --2- 172.21.15.116:0/1714236407 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155aa10 0x562ef1564d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 >> 172.21.15.116:0/1714236407 conn(0x562ef1555f80 msgr2=0x562ef1555200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 shutdown_connections 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 1 -- 172.21.15.116:0/1714236407 wait complete. 2024-03-04T07:40:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.324+0000 7f147a37e740 20 client.25276 trim_cache size 0 max 16384 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.339+0000 7f147a37e740 1 Processor -- start 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f147a37e740 1 -- start start 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f147a37e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef15591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f147a37e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f147a37e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1559720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f147a37e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701a8b40 con 0x562ef1560570 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f147a37e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701ab120 con 0x562ef155b400 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f147a37e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701a9140 con 0x562ef155aa10 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48114/0 (socket says 172.21.15.116:48114) 2024-03-04T07:40:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 -- 172.21.15.116:0/2305619823 learned_addr learned my addr 172.21.15.116:0/2305619823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f1475600640 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef15591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f1475600640 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1559720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 -- 172.21.15.116:0/2305619823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 msgr2=0x562ef15591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef15591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 -- 172.21.15.116:0/2305619823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 msgr2=0x562ef1559720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1559720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.340+0000 7f146fe00640 1 -- 172.21.15.116:0/2305619823 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ef156d730 con 0x562ef155b400 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.341+0000 7f1475600640 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef15591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.341+0000 7f146fe00640 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156c270 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f145c005b60 tx=0x7f145c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34e2090ef3fa04e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.341+0000 7f1461200640 1 -- 172.21.15.116:0/2305619823 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c043040 con 0x562ef155b400 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.341+0000 7f1461200640 1 -- 172.21.15.116:0/2305619823 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f145c005850 con 0x562ef155b400 2024-03-04T07:40:58.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.341+0000 7f1461200640 1 -- 172.21.15.116:0/2305619823 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c039040 con 0x562ef155b400 2024-03-04T07:40:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 -- 172.21.15.116:0/2305619823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 msgr2=0x562ef156c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156c270 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f145c005b60 tx=0x7f145c000bc0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 -- 172.21.15.116:0/2305619823 shutdown_connections 2024-03-04T07:40:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef1559720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156c270 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 --2- 172.21.15.116:0/2305619823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef15591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 -- 172.21.15.116:0/2305619823 >> 172.21.15.116:0/2305619823 conn(0x562ef1555f80 msgr2=0x562ef155c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 -- 172.21.15.116:0/2305619823 shutdown_connections 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 -- 172.21.15.116:0/2305619823 wait complete. 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 -- start start 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.342+0000 7f147a37e740 1 Processor -- start 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f147a37e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef1569fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f147a37e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f147a37e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f147a37e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701a97a0 con 0x562ef1560570 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f147a37e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701aaac0 con 0x562ef155b400 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f147a37e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701a9e00 con 0x562ef155aa10 2024-03-04T07:40:58.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f147a37e740 10 client.0 _ll_register_callbacks cb 0x7f1464005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f1475600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48116/0 (socket says 172.21.15.116:48116) 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 -- 172.21.15.116:0/2780502137 learned_addr learned my addr 172.21.15.116:0/2780502137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f1475600640 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef1569fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 -- 172.21.15.116:0/2780502137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 msgr2=0x562ef1569fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef1569fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 -- 172.21.15.116:0/2780502137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 msgr2=0x562ef156b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f146fe00640 1 -- 172.21.15.116:0/2780502137 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ef156ba70 con 0x562ef155b400 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.343+0000 7f1475600640 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f146fe00640 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156a510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f145c02c0b0 tx=0x7f145c005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e42fb76e60ae5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:40:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f1456000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f1456000640 1 -- 172.21.15.116:0/2780502137 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c039070 con 0x562ef155b400 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f1456000640 1 -- 172.21.15.116:0/2780502137 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f145c02c710 con 0x562ef155b400 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f145c005850 con 0x562ef155b400 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f1456000640 1 -- 172.21.15.116:0/2780502137 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f145c043070 con 0x562ef155b400 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f147a37e740 10 client.25298 fetch_fsmap learned FSMap version 19 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562ef15cca20 con 0x562ef155b400 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f1456000640 1 -- 172.21.15.116:0/2780502137 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f145c005850 con 0x562ef155b400 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.344+0000 7f1456000640 1 -- 172.21.15.116:0/2780502137 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f145c03f450 con 0x562ef155b400 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f147a37e740 10 client.25298 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f147a37e740 10 client.25298 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f147a37e740 20 client.25298 populate_metadata read hostname 'smithi116' 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f147a37e740 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef15cd2b0 0x562ef15cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f147a37e740 4 client.25298 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:40:58.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562ef14c7c00 con 0x562ef15cd2b0 2024-03-04T07:40:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f1475600640 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef15cd2b0 0x562ef15cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:40:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f1475600640 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef15cd2b0 0x562ef15cf6d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c779c2a777d15805 server_cookie=ada6ee661e41dacc in_seq=0 out_seq=0 2024-03-04T07:40:58.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.345+0000 7f1456000640 10 client.25298 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.350+0000 7f1456000640 1 -- 172.21.15.116:0/2780502137 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562ef14c7c00 con 0x562ef15cd2b0 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.350+0000 7f1456000640 10 client.25298 handle_command_reply: tid=0 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.350+0000 7f147a37e740 1 client.25298 shutdown 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 msgr2=0x562ef156a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156a510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f145c02c0b0 tx=0x7f145c005f80 comp rx=0 tx=0).stop 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 shutdown_connections 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562ef15cd2b0 0x562ef15cf6d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ef1560570 0x562ef156b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562ef155b400 0x562ef156a510 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f1456000640 0 client.25298 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 --2- 172.21.15.116:0/2780502137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ef155aa10 0x562ef1569fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 >> 172.21.15.116:0/2780502137 conn(0x562ef1555f80 msgr2=0x562ef15649c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 shutdown_connections 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 1 -- 172.21.15.116:0/2780502137 wait complete. 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f147a37e740 20 client.25298 trim_cache size 0 max 16384 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:40:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:40:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:40:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:40:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14580a1df0 msgr2=0x7f14580a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f1476000640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14580a1df0 0x7f14580a42b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f146409ae00 tx=0x7f1464010e50 comp rx=0 tx=0).stop 2024-03-04T07:40:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.351+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 msgr2=0x7f1470199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:40:58.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 0x7f1470199950 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f146400e9a0 tx=0x7f146400ee60 comp rx=0 tx=0).stop 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 shutdown_connections 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f14580a1df0 0x7f14580a42b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1470105220 0x7f14701a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470104860 0x7f1470199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 --2- 172.21.15.116:0/1247522285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1470103670 0x7f1470199950 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 >> 172.21.15.116:0/1247522285 conn(0x7f14700febe0 msgr2=0x7f1470107b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 shutdown_connections 2024-03-04T07:40:58.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:40:58.352+0000 7f1476000640 1 -- 172.21.15.116:0/1247522285 wait complete. 2024-03-04T07:40:58.962 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:40:58 smithi116 ceph-mon[32150]: pgmap v407: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-04T07:40:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:40:58 smithi152 ceph-mon[37154]: pgmap v407: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-04T07:40:59.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:40:58 smithi133 ceph-mon[36505]: pgmap v407: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-04T07:41:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:00 smithi116 ceph-mon[32150]: pgmap v408: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:01.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:00 smithi152 ceph-mon[37154]: pgmap v408: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:00 smithi133 ceph-mon[36505]: pgmap v408: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:02.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:41:02.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:41:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 -- 172.21.15.116:0/2692477960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c1097a0 msgr2=0x7f739c110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 --2- 172.21.15.116:0/2692477960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c1097a0 0x7f739c110030 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7394009ca0 tx=0x7f739402c100 comp rx=0 tx=0).stop 2024-03-04T07:41:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 -- 172.21.15.116:0/2692477960 shutdown_connections 2024-03-04T07:41:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 --2- 172.21.15.116:0/2692477960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c1097a0 0x7f739c110030 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 --2- 172.21.15.116:0/2692477960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f739c108de0 0x7f739c109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 --2- 172.21.15.116:0/2692477960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f739c107bf0 0x7f739c108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 -- 172.21.15.116:0/2692477960 >> 172.21.15.116:0/2692477960 conn(0x7f739c076b90 msgr2=0x7f739c078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 -- 172.21.15.116:0/2692477960 shutdown_connections 2024-03-04T07:41:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.655+0000 7f73a2e00640 1 -- 172.21.15.116:0/2692477960 wait complete. 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 Processor -- start 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 -- start start 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c107bf0 0x7f739c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f739c108de0 0x7f739c19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f739c1097a0 0x7f739c1a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c113870 con 0x7f739c108de0 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1136f0 con 0x7f739c107bf0 2024-03-04T07:41:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f73a2e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1139f0 con 0x7f739c1097a0 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f739be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c107bf0 0x7f739c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f739be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c107bf0 0x7f739c19de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48136/0 (socket says 172.21.15.116:48136) 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f739be00640 1 -- 172.21.15.116:0/3171443770 learned_addr learned my addr 172.21.15.116:0/3171443770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.656+0000 7f7393e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f739c108de0 0x7f739c19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f73a0e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f739c1097a0 0x7f739c1a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f739be00640 1 -- 172.21.15.116:0/3171443770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f739c1097a0 msgr2=0x7f739c1a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f739be00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f739c1097a0 0x7f739c1a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f739be00640 1 -- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f739c108de0 msgr2=0x7f739c19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f739be00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f739c108de0 0x7f739c19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f739be00640 1 -- 172.21.15.116:0/3171443770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c1a5ab0 con 0x7f739c107bf0 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f7393e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f739c108de0 0x7f739c19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f739be00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c107bf0 0x7f739c19de90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e9c0 tx=0x7f738c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f7399600640 1 -- 172.21.15.116:0/3171443770 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c00cd80 con 0x7f739c107bf0 2024-03-04T07:41:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c1a5da0 con 0x7f739c107bf0 2024-03-04T07:41:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f7399600640 1 -- 172.21.15.116:0/3171443770 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f738c0035f0 con 0x7f739c107bf0 2024-03-04T07:41:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f7399600640 1 -- 172.21.15.116:0/3171443770 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c010690 con 0x7f739c107bf0 2024-03-04T07:41:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.657+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c1a6210 con 0x7f739c107bf0 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.660+0000 7f7399600640 1 -- 172.21.15.116:0/3171443770 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f738c0037a0 con 0x7f739c107bf0 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.660+0000 7f73a5c07740 1 Processor -- start 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a5c07740 1 -- start start 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x55663037af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630380040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x556630382b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a5c07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566302a8090 con 0x5566303806b0 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a5c07740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566302dfd30 con 0x55663037b540 2024-03-04T07:41:02.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a5c07740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556630288a30 con 0x55663037ab50 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f739be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630380040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f73a0e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x55663037af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f7393e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x556630382b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f739be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630380040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48148/0 (socket says 172.21.15.116:48148) 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.661+0000 7f739be00640 1 -- 172.21.15.116:0/2829588121 learned_addr learned my addr 172.21.15.116:0/2829588121 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f739be00640 1 -- 172.21.15.116:0/2829588121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 msgr2=0x55663037af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f739be00640 1 --2- 172.21.15.116:0/2829588121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x55663037af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f739be00640 1 -- 172.21.15.116:0/2829588121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 msgr2=0x556630382b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f739be00640 1 --2- 172.21.15.116:0/2829588121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x556630382b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f739be00640 1 -- 172.21.15.116:0/2829588121 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566300be7d0 con 0x55663037b540 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f739be00640 1 --2- 172.21.15.116:0/2829588121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630380040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f738c0d6290 tx=0x7f738c0d6380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f19e0a13b7001ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f7399600640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73800a1d80 0x7f73800a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f7391600640 1 -- 172.21.15.116:0/2829588121 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c0e1070 con 0x55663037b540 2024-03-04T07:41:02.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f73a0e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73800a1d80 0x7f73800a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f7391600640 1 -- 172.21.15.116:0/2829588121 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f738c09b670 con 0x55663037b540 2024-03-04T07:41:02.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f7391600640 1 -- 172.21.15.116:0/2829588121 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c003b10 con 0x55663037b540 2024-03-04T07:41:02.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.662+0000 7f7399600640 1 -- 172.21.15.116:0/3171443770 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f738c0ce280 con 0x7f739c107bf0 2024-03-04T07:41:02.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 -- 172.21.15.116:0/2829588121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 msgr2=0x556630380040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2829588121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630380040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f738c0d6290 tx=0x7f738c0d6380 comp rx=0 tx=0).stop 2024-03-04T07:41:02.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a0e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73800a1d80 0x7f73800a4240 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55663037bda0 tx=0x7f73940325b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 -- 172.21.15.116:0/2829588121 shutdown_connections 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2829588121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x556630382b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2829588121 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630380040 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2829588121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x55663037af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 -- 172.21.15.116:0/2829588121 >> 172.21.15.116:0/2829588121 conn(0x5566303760c0 msgr2=0x5566303764c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 -- 172.21.15.116:0/2829588121 shutdown_connections 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 -- 172.21.15.116:0/2829588121 wait complete. 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.663+0000 7f73a5c07740 1 -- start start 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 1 Processor -- start 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x556630384ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630385430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663027e0c0 con 0x5566303806b0 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566302dbdd0 con 0x55663037b540 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556630383660 con 0x55663037ab50 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a5c07740 10 client.0 _ll_register_callbacks cb 0x55663029b860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f7393e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630385430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f7393e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630385430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48160/0 (socket says 172.21.15.116:48160) 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f7393e00640 1 -- 172.21.15.116:0/1246888502 learned_addr learned my addr 172.21.15.116:0/1246888502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f73a0e00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.664+0000 7f739be00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x556630384ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f7393e00640 1 -- 172.21.15.116:0/1246888502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 msgr2=0x556630384ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f7393e00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x556630384ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f7393e00640 1 -- 172.21.15.116:0/1246888502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 msgr2=0x55663038c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f7393e00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f7393e00640 1 -- 172.21.15.116:0/1246888502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55662fff43f0 con 0x55663037b540 2024-03-04T07:41:02.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f73a0e00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f7393e00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630385430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7384012770 tx=0x7f7384012c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40fe7fd984fe8aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f738a000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f738a000640 1 -- 172.21.15.116:0/1246888502 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384004260 con 0x55663037b540 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f738a000640 1 -- 172.21.15.116:0/1246888502 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7384004400 con 0x55663037b540 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f738c09b670 con 0x55663037b540 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.665+0000 7f738a000640 1 -- 172.21.15.116:0/1246888502 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73840119a0 con 0x55663037b540 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 10 client.25310 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55662ffb40f0 con 0x55663037b540 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f738a000640 1 -- 172.21.15.116:0/1246888502 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f738c09b670 con 0x55663037b540 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f738a000640 1 -- 172.21.15.116:0/1246888502 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f738400f440 con 0x55663037b540 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 10 client.25310 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 10 client.25310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 20 client.25310 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556630390a60 0x556630392ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 4 client.25310 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566302ed270 con 0x556630390a60 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.666+0000 7f739be00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556630390a60 0x556630392ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.667+0000 7f739be00640 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556630390a60 0x556630392ec0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c838312d216b9bdf server_cookie=4babb1cfbb9fe8b2 in_seq=0 out_seq=0 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.667+0000 7f738a000640 10 client.25310 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.667+0000 7f738a000640 1 -- 172.21.15.116:0/1246888502 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5566302ed270 con 0x556630390a60 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.667+0000 7f738a000640 10 client.25310 handle_command_reply: tid=0 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.667+0000 7f73a5c07740 1 client.25310 shutdown 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 msgr2=0x556630385430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630385430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7384012770 tx=0x7f7384012c30 comp rx=0 tx=0).stop 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 shutdown_connections 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556630390a60 0x556630392ec0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x556630385430 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f738a000640 0 client.25310 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:02.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1246888502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037ab50 0x556630384ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 >> 172.21.15.116:0/1246888502 conn(0x5566303760c0 msgr2=0x556630375360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:02.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 shutdown_connections 2024-03-04T07:41:02.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 1 -- 172.21.15.116:0/1246888502 wait complete. 2024-03-04T07:41:02.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.668+0000 7f73a5c07740 20 client.25310 trim_cache size 0 max 16384 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 Processor -- start 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 -- start start 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037ab50 0x55663037ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037b540 0x55663037cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1ad070 con 0x5566303806b0 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1af650 con 0x55663037ab50 2024-03-04T07:41:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f73a5c07740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1ad670 con 0x55663037b540 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.685+0000 7f739be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037ab50 0x55663037ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037ab50 0x55663037ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48172/0 (socket says 172.21.15.116:48172) 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 -- 172.21.15.116:0/2063543893 learned_addr learned my addr 172.21.15.116:0/2063543893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f73a0e00640 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f7393e00640 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037b540 0x55663037cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 -- 172.21.15.116:0/2063543893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037b540 msgr2=0x55663037cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037b540 0x55663037cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 -- 172.21.15.116:0/2063543893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 msgr2=0x55663038c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 -- 172.21.15.116:0/2063543893 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556630376b10 con 0x55663037ab50 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f73a0e00640 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f739be00640 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037ab50 0x55663037ca00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f738c0d6880 tx=0x7f738c003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372196d29fe40c32 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f7390c00640 1 -- 172.21.15.116:0/2063543893 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c0046c0 con 0x55663037ab50 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f7390c00640 1 -- 172.21.15.116:0/2063543893 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f738c09b670 con 0x55663037ab50 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.686+0000 7f7390c00640 1 -- 172.21.15.116:0/2063543893 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c004880 con 0x55663037ab50 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 -- 172.21.15.116:0/2063543893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037ab50 msgr2=0x55663037ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037ab50 0x55663037ca00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f738c0d6880 tx=0x7f738c003f10 comp rx=0 tx=0).stop 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 -- 172.21.15.116:0/2063543893 shutdown_connections 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566303806b0 0x55663038c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55663037b540 0x55663037cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 --2- 172.21.15.116:0/2063543893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037ab50 0x55663037ca00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 -- 172.21.15.116:0/2063543893 >> 172.21.15.116:0/2063543893 conn(0x5566303760c0 msgr2=0x556630384ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 -- 172.21.15.116:0/2063543893 shutdown_connections 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 -- 172.21.15.116:0/2063543893 wait complete. 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 -- start start 2024-03-04T07:41:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.687+0000 7f73a5c07740 1 Processor -- start 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55663037ab50 0x55663038a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x55663038a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a5c07740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566303806b0 0x55663038b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a5c07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1adcd0 con 0x55663037ab50 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a5c07740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1aeff0 con 0x55663037b540 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a5c07740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c1ae330 con 0x5566303806b0 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a5c07740 10 client.0 _ll_register_callbacks cb 0x7f7384009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f7393e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x55663038a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f739be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55663037ab50 0x55663038a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f7393e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x55663038a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48184/0 (socket says 172.21.15.116:48184) 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f7393e00640 1 -- 172.21.15.116:0/1192677333 learned_addr learned my addr 172.21.15.116:0/1192677333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f73a0e00640 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566303806b0 0x55663038b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f739be00640 1 -- 172.21.15.116:0/1192677333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566303806b0 msgr2=0x55663038b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f739be00640 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566303806b0 0x55663038b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f739be00640 1 -- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 msgr2=0x55663038a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f739be00640 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x55663038a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f739be00640 1 -- 172.21.15.116:0/1192677333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55663038bce0 con 0x55663037ab50 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.688+0000 7f7393e00640 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x55663038a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f739be00640 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55663037ab50 0x55663038a1a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x556630383310 tx=0x7f738c09ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b53d01909f22a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f738a000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f738a000640 1 -- 172.21.15.116:0/1192677333 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c004a70 con 0x55663037ab50 2024-03-04T07:41:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f738c09b670 con 0x55663037ab50 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f738a000640 1 -- 172.21.15.116:0/1192677333 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f738c0d6dc0 con 0x55663037ab50 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f738a000640 1 -- 172.21.15.116:0/1192677333 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f738c004c60 con 0x55663037ab50 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f73a5c07740 10 client.15210 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5566303ec9e0 con 0x55663037ab50 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.689+0000 7f738a000640 1 -- 172.21.15.116:0/1192677333 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f738c09b670 con 0x55663037ab50 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f738a000640 1 -- 172.21.15.116:0/1192677333 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f738c0975e0 con 0x55663037ab50 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f73a5c07740 10 client.15210 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f73a5c07740 10 client.15210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f73a5c07740 20 client.15210 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566303ece90 0x5566303ef280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f73a5c07740 4 client.15210 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5566302e75c0 con 0x5566303ece90 2024-03-04T07:41:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f7393e00640 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566303ece90 0x5566303ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f7393e00640 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566303ece90 0x5566303ef280 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f77a5d8ca74201bf server_cookie=e5f25792fae1477f in_seq=0 out_seq=0 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.690+0000 7f738a000640 10 client.15210 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f738a000640 1 -- 172.21.15.116:0/1192677333 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5566302e75c0 con 0x5566303ece90 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f738a000640 10 client.15210 handle_command_reply: tid=0 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 client.15210 shutdown 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55663037ab50 msgr2=0x55663038a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55663037ab50 0x55663038a1a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x556630383310 tx=0x7f738c09ae40 comp rx=0 tx=0).stop 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 shutdown_connections 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566303ece90 0x5566303ef280 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566303806b0 0x55663038b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55663037b540 0x55663038a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 --2- 172.21.15.116:0/1192677333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55663037ab50 0x55663038a1a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f738a000640 0 client.15210 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 >> 172.21.15.116:0/1192677333 conn(0x5566303760c0 msgr2=0x556630379bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 shutdown_connections 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 1 -- 172.21.15.116:0/1192677333 wait complete. 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.691+0000 7f73a5c07740 20 client.15210 trim_cache size 0 max 16384 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:41:02.697 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:41:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73800a1d80 msgr2=0x7f73800a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73800a1d80 0x7f73800a4240 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55663037bda0 tx=0x7f73940325b0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c107bf0 msgr2=0x7f739c19de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c107bf0 0x7f739c19de90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e9c0 tx=0x7f738c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:41:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 shutdown_connections 2024-03-04T07:41:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73800a1d80 0x7f73800a4240 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f739c1097a0 0x7f739c1a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f739c108de0 0x7f739c19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 --2- 172.21.15.116:0/3171443770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f739c107bf0 0x7f739c19de90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 >> 172.21.15.116:0/3171443770 conn(0x7f739c076b90 msgr2=0x7f739c10c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 shutdown_connections 2024-03-04T07:41:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.692+0000 7f73a2e00640 1 -- 172.21.15.116:0/3171443770 wait complete. 2024-03-04T07:41:02.707 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:41:02.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:41:02.776 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:02 smithi116 ceph-mon[32150]: pgmap v409: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 341 B/s wr, 3 op/s 2024-03-04T07:41:02.836 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:02 smithi133 ceph-mon[36505]: pgmap v409: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 341 B/s wr, 3 op/s 2024-03-04T07:41:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 -- 172.21.15.116:0/1858125355 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 msgr2=0x7fb9cc108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/1858125355 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc108030 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009900 tx=0x7fb9bc02bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:02.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 -- 172.21.15.116:0/1858125355 shutdown_connections 2024-03-04T07:41:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/1858125355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc1097c0 0x7fb9cc110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/1858125355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9cc108e00 0x7fb9cc109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/1858125355 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc108030 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 -- 172.21.15.116:0/1858125355 >> 172.21.15.116:0/1858125355 conn(0x7fb9cc076be0 msgr2=0x7fb9cc079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 -- 172.21.15.116:0/1858125355 shutdown_connections 2024-03-04T07:41:02.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.906+0000 7fb9d1a00640 1 -- 172.21.15.116:0/1858125355 wait complete. 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.907+0000 7fb9d1a00640 1 Processor -- start 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.907+0000 7fb9d1a00640 1 -- start start 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9d1a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9d1a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9cc108e00 0x7fb9cc19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9d1a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc1097c0 0x7fb9cc1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9d1a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc1138b0 con 0x7fb9cc1097c0 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9d1a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc113730 con 0x7fb9cc107c10 2024-03-04T07:41:02.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9d1a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc113a30 con 0x7fb9cc108e00 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48214/0 (socket says 172.21.15.116:48214) 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 -- 172.21.15.116:0/69417093 learned_addr learned my addr 172.21.15.116:0/69417093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9cb400640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc1097c0 0x7fb9cc1a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9c2000640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9cc108e00 0x7fb9cc19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 -- 172.21.15.116:0/69417093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9cc108e00 msgr2=0x7fb9cc19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9cc108e00 0x7fb9cc19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 -- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc1097c0 msgr2=0x7fb9cc1a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc1097c0 0x7fb9cc1a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9caa00640 1 -- 172.21.15.116:0/69417093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc1a5b40 con 0x7fb9cc107c10 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.908+0000 7fb9c2000640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9cc108e00 0x7fb9cc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.909+0000 7fb9caa00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc19df20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc000c00 tx=0x7fb9bc005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.909+0000 7fb9c3e00640 1 -- 172.21.15.116:0/69417093 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc043020 con 0x7fb9cc107c10 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.909+0000 7fb9c3e00640 1 -- 172.21.15.116:0/69417093 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9bc0054e0 con 0x7fb9cc107c10 2024-03-04T07:41:02.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.909+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc1a5dd0 con 0x7fb9cc107c10 2024-03-04T07:41:02.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.909+0000 7fb9c3e00640 1 -- 172.21.15.116:0/69417093 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc003e60 con 0x7fb9cc107c10 2024-03-04T07:41:02.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.909+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc1a6290 con 0x7fb9cc107c10 2024-03-04T07:41:02.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.910+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc10ef30 con 0x7fb9cc107c10 2024-03-04T07:41:02.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.912+0000 7fb9c3e00640 1 -- 172.21.15.116:0/69417093 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb9bc046020 con 0x7fb9cc107c10 2024-03-04T07:41:02.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.913+0000 7fb9c3e00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb9b00a1e00 0x7fb9b00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:02.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.913+0000 7fb9c3e00640 1 -- 172.21.15.116:0/69417093 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb9bc0effa0 con 0x7fb9cc107c10 2024-03-04T07:41:02.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.913+0000 7fb9c2000640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb9b00a1e00 0x7fb9b00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:02.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.914+0000 7fb9c2000640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb9b00a1e00 0x7fb9b00a42c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40058c0 tx=0x7fb9b4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:02.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:02.917+0000 7fb9c3e00640 1 -- 172.21.15.116:0/69417093 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb9bc04d020 con 0x7fb9cc107c10 2024-03-04T07:41:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:02 smithi152 ceph-mon[37154]: pgmap v409: 97 pgs: 97 active+clean; 651 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 341 B/s wr, 3 op/s 2024-03-04T07:41:03.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.155+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9cc062270 con 0x7fb9cc107c10 2024-03-04T07:41:03.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.157+0000 7fb9c3e00640 1 -- 172.21.15.116:0/69417093 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fb9bc0bd390 con 0x7fb9cc107c10 2024-03-04T07:41:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:41:03.157 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:41:03.157 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:41:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.159+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb9b00a1e00 msgr2=0x7fb9b00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.159+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb9b00a1e00 0x7fb9b00a42c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40058c0 tx=0x7fb9b4005850 comp rx=0 tx=0).stop 2024-03-04T07:41:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 msgr2=0x7fb9cc19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc19df20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc000c00 tx=0x7fb9bc005dd0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 shutdown_connections 2024-03-04T07:41:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9cc1097c0 0x7fb9cc1a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb9b00a1e00 0x7fb9b00a42c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9cc108e00 0x7fb9cc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 --2- 172.21.15.116:0/69417093 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9cc107c10 0x7fb9cc19df20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.160+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 >> 172.21.15.116:0/69417093 conn(0x7fb9cc076be0 msgr2=0x7fb9cc10c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.161+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 shutdown_connections 2024-03-04T07:41:03.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.161+0000 7fb9d1a00640 1 -- 172.21.15.116:0/69417093 wait complete. 2024-03-04T07:41:03.179 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:41:03.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.394+0000 7fb389c00640 1 -- 172.21.15.116:0/2411847856 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb384107bf0 msgr2=0x7fb384108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.394+0000 7fb389c00640 1 --2- 172.21.15.116:0/2411847856 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb384107bf0 0x7fb384108010 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb374009900 tx=0x7fb37402bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.395+0000 7fb389c00640 1 -- 172.21.15.116:0/2411847856 shutdown_connections 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.395+0000 7fb389c00640 1 --2- 172.21.15.116:0/2411847856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3841097a0 0x7fb384110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.395+0000 7fb389c00640 1 --2- 172.21.15.116:0/2411847856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb384108de0 0x7fb384109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.395+0000 7fb389c00640 1 --2- 172.21.15.116:0/2411847856 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb384107bf0 0x7fb384108010 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.395+0000 7fb389c00640 1 -- 172.21.15.116:0/2411847856 >> 172.21.15.116:0/2411847856 conn(0x7fb384076b90 msgr2=0x7fb384078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.395+0000 7fb389c00640 1 -- 172.21.15.116:0/2411847856 shutdown_connections 2024-03-04T07:41:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.395+0000 7fb389c00640 1 -- 172.21.15.116:0/2411847856 wait complete. 2024-03-04T07:41:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 Processor -- start 2024-03-04T07:41:03.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 -- start start 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb384107bf0 0x7fb38419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb384108de0 0x7fb38419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3841097a0 0x7fb3841a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841138c0 con 0x7fb384107bf0 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384113740 con 0x7fb3841097a0 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.396+0000 7fb389c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384113a40 con 0x7fb384108de0 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3841097a0 0x7fb3841a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb382a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb384107bf0 0x7fb38419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3841097a0 0x7fb3841a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48242/0 (socket says 172.21.15.116:48242) 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 -- 172.21.15.116:0/943053945 learned_addr learned my addr 172.21.15.116:0/943053945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb382000640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb384108de0 0x7fb38419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 -- 172.21.15.116:0/943053945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb384108de0 msgr2=0x7fb38419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb384108de0 0x7fb38419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 -- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb384107bf0 msgr2=0x7fb38419df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb384107bf0 0x7fb38419df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 -- 172.21.15.116:0/943053945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3841a5b20 con 0x7fb3841097a0 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.397+0000 7fb383400640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3841097a0 0x7fb3841a54b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00eb10 tx=0x7fb37c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.398+0000 7fb37b400640 1 -- 172.21.15.116:0/943053945 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c004b20 con 0x7fb3841097a0 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.398+0000 7fb37b400640 1 -- 172.21.15.116:0/943053945 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37c004cc0 con 0x7fb3841097a0 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.398+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3841a5e10 con 0x7fb3841097a0 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.398+0000 7fb37b400640 1 -- 172.21.15.116:0/943053945 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c010760 con 0x7fb3841097a0 2024-03-04T07:41:03.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.398+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3841a6320 con 0x7fb3841097a0 2024-03-04T07:41:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.401+0000 7fb38ca04740 1 Processor -- start 2024-03-04T07:41:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.401+0000 7fb37b400640 1 -- 172.21.15.116:0/943053945 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb37c005020 con 0x7fb3841097a0 2024-03-04T07:41:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.401+0000 7fb38ca04740 1 -- start start 2024-03-04T07:41:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7ba190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7bcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb38ca04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098d719d60 con 0x56098d7ba800 2024-03-04T07:41:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb38ca04740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098d6c2cf0 con 0x56098d7b4ca0 2024-03-04T07:41:03.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb38ca04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098d715e00 con 0x56098d7b5690 2024-03-04T07:41:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb382000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb382000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48252/0 (socket says 172.21.15.116:48252) 2024-03-04T07:41:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb382000640 1 -- 172.21.15.116:0/1921179014 learned_addr learned my addr 172.21.15.116:0/1921179014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb382a00640 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7ba190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.402+0000 7fb382000640 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7bcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382000640 1 -- 172.21.15.116:0/1921179014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 msgr2=0x56098d7ba190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382000640 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7ba190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382000640 1 -- 172.21.15.116:0/1921179014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 msgr2=0x56098d7bcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382000640 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7bcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382000640 1 -- 172.21.15.116:0/1921179014 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56098d42e3f0 con 0x56098d7b4ca0 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382a00640 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7ba190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb37b400640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb3680a1e00 0x7fb3680a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb37b400640 1 -- 172.21.15.116:0/943053945 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb37c0ce440 con 0x7fb3841097a0 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382000640 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7b50c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb3700093e0 tx=0x7fb370009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca6ab69f1e4f8714 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb382a00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb3680a1e00 0x7fb3680a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb36fe00640 1 -- 172.21.15.116:0/1921179014 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37000f7c0 con 0x56098d7b4ca0 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb36fe00640 1 -- 172.21.15.116:0/1921179014 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37000fe40 con 0x56098d7b4ca0 2024-03-04T07:41:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.403+0000 7fb36fe00640 1 -- 172.21.15.116:0/1921179014 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37000e3e0 con 0x56098d7b4ca0 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 -- 172.21.15.116:0/1921179014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 msgr2=0x56098d7b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb382a00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb3680a1e00 0x7fb3680a42c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb37402c170 tx=0x7fb37402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7b50c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb3700093e0 tx=0x7fb370009370 comp rx=0 tx=0).stop 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 -- 172.21.15.116:0/1921179014 shutdown_connections 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7bcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7ba190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1921179014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7b50c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 -- 172.21.15.116:0/1921179014 >> 172.21.15.116:0/1921179014 conn(0x56098d7b0210 msgr2=0x56098d7b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 -- 172.21.15.116:0/1921179014 shutdown_connections 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.404+0000 7fb38ca04740 1 -- 172.21.15.116:0/1921179014 wait complete. 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 -- start start 2024-03-04T07:41:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 Processor -- start 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098d7bd7b0 con 0x56098d7ba800 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098d6b8220 con 0x56098d7b4ca0 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098d7bd930 con 0x56098d7b5690 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb38ca04740 10 client.0 _ll_register_callbacks cb 0x56098d42dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb382a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb383400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb382a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48266/0 (socket says 172.21.15.116:48266) 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.405+0000 7fb382a00640 1 -- 172.21.15.116:0/2768818843 learned_addr learned my addr 172.21.15.116:0/2768818843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb382000640 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb382a00640 1 -- 172.21.15.116:0/2768818843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 msgr2=0x56098d7bf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb382a00640 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7bf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb382a00640 1 -- 172.21.15.116:0/2768818843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 msgr2=0x56098d7c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb382a00640 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb382a00640 1 -- 172.21.15.116:0/2768818843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56098d4f87d0 con 0x56098d7b4ca0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb382a00640 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf270 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb374009900 tx=0x7fb374009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91d1592547245675 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb36cc00640 1 -- 172.21.15.116:0/2768818843 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37404f070 con 0x56098d7b4ca0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb36cc00640 1 -- 172.21.15.116:0/2768818843 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb374003800 con 0x56098d7b4ca0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.406+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb37000fe40 con 0x56098d7b4ca0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb36cc00640 1 -- 172.21.15.116:0/2768818843 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb374045070 con 0x56098d7b4ca0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 10 client.25334 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56098d7c9bd0 con 0x56098d7b4ca0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb36cc00640 1 -- 172.21.15.116:0/2768818843 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37000fe40 con 0x56098d7b4ca0 2024-03-04T07:41:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb36cc00640 1 -- 172.21.15.116:0/2768818843 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb374054b60 con 0x56098d7b4ca0 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 10 client.25334 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 10 client.25334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 20 client.25334 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d7cb220 0x56098d7cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 4 client.25334 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.407+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56098d7272a0 con 0x56098d7cb220 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.408+0000 7fb382000640 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d7cb220 0x56098d7cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.408+0000 7fb382000640 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d7cb220 0x56098d7cd6a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2710870dbfe39dc5 server_cookie=3909eef392266ab1 in_seq=0 out_seq=0 2024-03-04T07:41:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.408+0000 7fb36cc00640 10 client.25334 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb36cc00640 1 -- 172.21.15.116:0/2768818843 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56098d7272a0 con 0x56098d7cb220 2024-03-04T07:41:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb36cc00640 10 client.25334 handle_command_reply: tid=0 2024-03-04T07:41:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 client.25334 shutdown 2024-03-04T07:41:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 msgr2=0x56098d7bf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf270 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb374009900 tx=0x7fb374009260 comp rx=0 tx=0).stop 2024-03-04T07:41:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 shutdown_connections 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d7cb220 0x56098d7cd6a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 --2- 172.21.15.116:0/2768818843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf270 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb36cc00640 0 client.25334 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 >> 172.21.15.116:0/2768818843 conn(0x56098d7b0210 msgr2=0x56098d7af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 shutdown_connections 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.409+0000 7fb38ca04740 1 -- 172.21.15.116:0/2768818843 wait complete. 2024-03-04T07:41:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.410+0000 7fb38ca04740 20 client.25334 trim_cache size 0 max 16384 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 Processor -- start 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 -- start start 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841ad0c0 con 0x56098d7ba800 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841af6a0 con 0x56098d7b4ca0 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb38ca04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841ad6c0 con 0x56098d7b5690 2024-03-04T07:41:03.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.423+0000 7fb382a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb383400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48272/0 (socket says 172.21.15.116:48272) 2024-03-04T07:41:03.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 -- 172.21.15.116:0/1354233875 learned_addr learned my addr 172.21.15.116:0/1354233875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:03.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382000640 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 -- 172.21.15.116:0/1354233875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 msgr2=0x56098d7c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 -- 172.21.15.116:0/1354233875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 msgr2=0x56098d7b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 -- 172.21.15.116:0/1354233875 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56098d7b9f10 con 0x56098d7b4ca0 2024-03-04T07:41:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb382a00640 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb374036040 tx=0x7fb3740058f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4837b7c76352dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb36f400640 1 -- 172.21.15.116:0/1354233875 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374063070 con 0x56098d7b4ca0 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb36f400640 1 -- 172.21.15.116:0/1354233875 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb374034600 con 0x56098d7b4ca0 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.424+0000 7fb36f400640 1 -- 172.21.15.116:0/1354233875 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37404f070 con 0x56098d7b4ca0 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 -- 172.21.15.116:0/1354233875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 msgr2=0x56098d7bf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb374036040 tx=0x7fb3740058f0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 -- 172.21.15.116:0/1354233875 shutdown_connections 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7ba800 0x56098d7b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7b5690 0x56098d7c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1354233875 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b4ca0 0x56098d7bf710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 -- 172.21.15.116:0/1354233875 >> 172.21.15.116:0/1354233875 conn(0x56098d7b0210 msgr2=0x56098d7b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 -- 172.21.15.116:0/1354233875 shutdown_connections 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.425+0000 7fb38ca04740 1 -- 172.21.15.116:0/1354233875 wait complete. 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 -- start start 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 Processor -- start 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7b4ca0 0x56098d7c4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b5690 0x56098d7c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7ba800 0x56098d7c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841add20 con 0x56098d7b4ca0 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841af040 con 0x56098d7b5690 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841ae380 con 0x56098d7ba800 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb38ca04740 10 client.0 _ll_register_callbacks cb 0x7fb374009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:03.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.426+0000 7fb382000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b5690 0x56098d7c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7b4ca0 0x56098d7c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b5690 0x56098d7c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48282/0 (socket says 172.21.15.116:48282) 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 -- 172.21.15.116:0/1436187119 learned_addr learned my addr 172.21.15.116:0/1436187119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb383400640 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7ba800 0x56098d7c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 -- 172.21.15.116:0/1436187119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7ba800 msgr2=0x56098d7c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7ba800 0x56098d7c5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 -- 172.21.15.116:0/1436187119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7b4ca0 msgr2=0x56098d7c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7b4ca0 0x56098d7c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 -- 172.21.15.116:0/1436187119 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56098d7c6270 con 0x56098d7b5690 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382a00640 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7b4ca0 0x56098d7c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb382000640 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b5690 0x56098d7c4c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb3700117d0 tx=0x7fb370009d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d09163b6b3aa00 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:03.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb36cc00640 1 -- 172.21.15.116:0/1436187119 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37002d2d0 con 0x56098d7b5690 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb36cc00640 1 -- 172.21.15.116:0/1436187119 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb370005040 con 0x56098d7b5690 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.427+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb374034600 con 0x56098d7b5690 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb36cc00640 1 -- 172.21.15.116:0/1436187119 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb370005240 con 0x56098d7b5690 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 10 client.25340 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56098d8273e0 con 0x56098d7b5690 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb36cc00640 1 -- 172.21.15.116:0/1436187119 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb374034600 con 0x56098d7b5690 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb36cc00640 1 -- 172.21.15.116:0/1436187119 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb370035030 con 0x56098d7b5690 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 10 client.25340 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 10 client.25340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 20 client.25340 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d827bf0 0x56098d82a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 4 client.25340 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56098d7215f0 con 0x56098d827bf0 2024-03-04T07:41:03.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.428+0000 7fb382a00640 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d827bf0 0x56098d82a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb382a00640 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d827bf0 0x56098d82a010 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=586d5719832e9b2c server_cookie=31d89fb0425630f3 in_seq=0 out_seq=0 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb36cc00640 10 client.25340 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb36cc00640 1 -- 172.21.15.116:0/1436187119 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56098d7215f0 con 0x56098d827bf0 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb36cc00640 10 client.25340 handle_command_reply: tid=0 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 client.25340 shutdown 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b5690 msgr2=0x56098d7c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b5690 0x56098d7c4c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb3700117d0 tx=0x7fb370009d80 comp rx=0 tx=0).stop 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 shutdown_connections 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56098d827bf0 0x56098d82a010 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56098d7ba800 0x56098d7c5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56098d7b5690 0x56098d7c4c70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb36cc00640 0 client.25340 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 --2- 172.21.15.116:0/1436187119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56098d7b4ca0 0x56098d7c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 >> 172.21.15.116:0/1436187119 conn(0x56098d7b0210 msgr2=0x56098d7bf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:03.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 shutdown_connections 2024-03-04T07:41:03.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.429+0000 7fb38ca04740 1 -- 172.21.15.116:0/1436187119 wait complete. 2024-03-04T07:41:03.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb38ca04740 20 client.25340 trim_cache size 0 max 16384 2024-03-04T07:41:03.435 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb3680a1e00 msgr2=0x7fb3680a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb3680a1e00 0x7fb3680a42c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb37402c170 tx=0x7fb37402df90 comp rx=0 tx=0).stop 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3841097a0 msgr2=0x7fb3841a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3841097a0 0x7fb3841a54b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00eb10 tx=0x7fb37c00c2d0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 shutdown_connections 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb3841097a0 0x7fb3841a54b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb3680a1e00 0x7fb3680a42c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb384108de0 0x7fb38419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 --2- 172.21.15.116:0/943053945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb384107bf0 0x7fb38419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 >> 172.21.15.116:0/943053945 conn(0x7fb384076b90 msgr2=0x7fb38410c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 shutdown_connections 2024-03-04T07:41:03.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:03.430+0000 7fb389c00640 1 -- 172.21.15.116:0/943053945 wait complete. 2024-03-04T07:41:03.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:03 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/69417093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:41:04.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:03 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/69417093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:41:04.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:03 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/69417093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:41:04.449 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:41:04.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:41:04.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:41:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 -- 172.21.15.116:0/3824461164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 msgr2=0x7f1250104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 --2- 172.21.15.116:0/3824461164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f1250104e80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1238009900 tx=0x7f123802bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 -- 172.21.15.116:0/3824461164 shutdown_connections 2024-03-04T07:41:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 --2- 172.21.15.116:0/3824461164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12501053c0 0x7f125010bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 --2- 172.21.15.116:0/3824461164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f1250104e80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 --2- 172.21.15.116:0/3824461164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1250103810 0x7f1250103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 -- 172.21.15.116:0/3824461164 >> 172.21.15.116:0/3824461164 conn(0x7f12500fed80 msgr2=0x7f12501011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 -- 172.21.15.116:0/3824461164 shutdown_connections 2024-03-04T07:41:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.683+0000 7f1254c00640 1 -- 172.21.15.116:0/3824461164 wait complete. 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 Processor -- start 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 -- start start 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1250103810 0x7f1250199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f125019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12501053c0 0x7f12501a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125010f4e0 con 0x7f12501053c0 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125010f360 con 0x7f1250104a00 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.684+0000 7f1254c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125010f660 con 0x7f1250103810 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f125019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f125019a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48300/0 (socket says 172.21.15.116:48300) 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 -- 172.21.15.116:0/4172347024 learned_addr learned my addr 172.21.15.116:0/4172347024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124e800640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12501053c0 0x7f12501a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124de00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1250103810 0x7f1250199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 -- 172.21.15.116:0/4172347024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1250103810 msgr2=0x7f1250199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1250103810 0x7f1250199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 -- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12501053c0 msgr2=0x7f12501a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12501053c0 0x7f12501a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 -- 172.21.15.116:0/4172347024 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12501a1740 con 0x7f1250104a00 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.685+0000 7f124d400640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f125019a060 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1238000c00 tx=0x7f123802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.686+0000 7f1246a00640 1 -- 172.21.15.116:0/4172347024 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1238043020 con 0x7f1250104a00 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.686+0000 7f1246a00640 1 -- 172.21.15.116:0/4172347024 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12380054e0 con 0x7f1250104a00 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.686+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12501a19d0 con 0x7f1250104a00 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.686+0000 7f1246a00640 1 -- 172.21.15.116:0/4172347024 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1238003c40 con 0x7f1250104a00 2024-03-04T07:41:04.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.686+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12501a1f70 con 0x7f1250104a00 2024-03-04T07:41:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f1246a00640 1 -- 172.21.15.116:0/4172347024 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1238046020 con 0x7f1250104a00 2024-03-04T07:41:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 Processor -- start 2024-03-04T07:41:04.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 -- start start 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094a80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c093b3e20 con 0x561c094a5c30 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c09405a20 con 0x561c094a0ac0 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.689+0000 7f125785f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c093aeb00 con 0x561c094a00d0 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48312/0 (socket says 172.21.15.116:48312) 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094a80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124de00640 1 -- 172.21.15.116:0/2246508526 learned_addr learned my addr 172.21.15.116:0/2246508526 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124e800640 1 -- 172.21.15.116:0/2246508526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 msgr2=0x561c094a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124e800640 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124e800640 1 -- 172.21.15.116:0/2246508526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 msgr2=0x561c094a80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124e800640 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094a80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124e800640 1 -- 172.21.15.116:0/2246508526 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c093c3e10 con 0x561c094a00d0 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124de00640 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f1246a00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f12340a1e00 0x7f12340a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.690+0000 7f124e800640 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094a04f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1248009f00 tx=0x7f1248011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c80755a3931039d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f1246a00640 1 -- 172.21.15.116:0/4172347024 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f12380efe10 con 0x7f1250104a00 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f124de00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f12340a1e00 0x7f12340a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f123f000640 1 -- 172.21.15.116:0/2246508526 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248004d20 con 0x561c094a00d0 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f123f000640 1 -- 172.21.15.116:0/2246508526 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1248004030 con 0x561c094a00d0 2024-03-04T07:41:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f123f000640 1 -- 172.21.15.116:0/2246508526 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248005260 con 0x561c094a00d0 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f125785f740 1 -- 172.21.15.116:0/2246508526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 msgr2=0x561c094a04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f124de00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f12340a1e00 0x7f12340a42c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x561c094a0ec0 tx=0x7f1240005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.691+0000 7f125785f740 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094a04f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1248009f00 tx=0x7f1248011810 comp rx=0 tx=0).stop 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 -- 172.21.15.116:0/2246508526 shutdown_connections 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094a80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 --2- 172.21.15.116:0/2246508526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094a04f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 -- 172.21.15.116:0/2246508526 >> 172.21.15.116:0/2246508526 conn(0x561c0949b640 msgr2=0x561c0949ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 -- 172.21.15.116:0/2246508526 shutdown_connections 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 -- 172.21.15.116:0/2246508526 wait complete. 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 -- start start 2024-03-04T07:41:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.692+0000 7f125785f740 1 Processor -- start 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f125785f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a00d0 0x561c094aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f125785f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f125785f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a5c30 0x561c094b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f125785f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c093c1f40 con 0x561c094a00d0 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f125785f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c09401ac0 con 0x561c094a0ac0 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f125785f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c094a8be0 con 0x561c094a5c30 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f125785f740 10 client.0 _ll_register_callbacks cb 0x561c09400aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a00d0 0x561c094aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a5c30 0x561c094b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a5c30 0x561c094b1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:34448/0 (socket says 172.21.15.116:34448) 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a00d0 0x561c094aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60086/0 (socket says 172.21.15.116:60086) 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094aaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48318/0 (socket says 172.21.15.116:48318) 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124e800640 1 -- 172.21.15.116:0/50776542 learned_addr learned my addr 172.21.15.116:0/50776542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124d400640 1 -- 172.21.15.116:0/50776542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a5c30 msgr2=0x561c094b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.693+0000 7f124d400640 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a5c30 0x561c094b1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f124d400640 1 -- 172.21.15.116:0/50776542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a00d0 msgr2=0x561c094aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f124d400640 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a00d0 0x561c094aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f124d400640 1 -- 172.21.15.116:0/50776542 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c091e27e0 con 0x561c094a0ac0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f124d400640 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094aaa30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12380bcad0 tx=0x7f1238009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da3dcfcab7366208 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f1233400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f1233400640 1 -- 172.21.15.116:0/50776542 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12380bc680 con 0x561c094a0ac0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f1233400640 1 -- 172.21.15.116:0/50776542 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f12380eda20 con 0x561c094a0ac0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1248004030 con 0x561c094a0ac0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f1233400640 1 -- 172.21.15.116:0/50776542 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12380bc4a0 con 0x561c094a0ac0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f125785f740 10 client.25352 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.694+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c09242f70 con 0x561c094a0ac0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f1233400640 1 -- 172.21.15.116:0/50776542 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1248004030 con 0x561c094a0ac0 2024-03-04T07:41:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f1233400640 1 -- 172.21.15.116:0/50776542 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f12380c4e50 con 0x561c094a0ac0 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f125785f740 10 client.25352 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f125785f740 10 client.25352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f125785f740 20 client.25352 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f125785f740 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c094b6280 0x561c094b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f125785f740 4 client.25352 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c09412f60 con 0x561c094b6280 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.695+0000 7f124de00640 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c094b6280 0x561c094b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.696+0000 7f124de00640 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c094b6280 0x561c094b86e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b293f8d9d7204162 server_cookie=a9771e3970e8e424 in_seq=0 out_seq=0 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.696+0000 7f1233400640 10 client.25352 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.696+0000 7f1233400640 1 -- 172.21.15.116:0/50776542 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c09412f60 con 0x561c094b6280 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.696+0000 7f1233400640 10 client.25352 handle_command_reply: tid=0 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.696+0000 7f125785f740 1 client.25352 shutdown 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 msgr2=0x561c094aaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094aaa30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12380bcad0 tx=0x7f1238009560 comp rx=0 tx=0).stop 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 shutdown_connections 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c094b6280 0x561c094b86e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a5c30 0x561c094b1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a0ac0 0x561c094aaa30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f1233400640 0 client.25352 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 --2- 172.21.15.116:0/50776542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a00d0 0x561c094aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 >> 172.21.15.116:0/50776542 conn(0x561c0949b640 msgr2=0x561c0949c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 shutdown_connections 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 1 -- 172.21.15.116:0/50776542 wait complete. 2024-03-04T07:41:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.697+0000 7f125785f740 20 client.25352 trim_cache size 0 max 16384 2024-03-04T07:41:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.714+0000 7f125785f740 1 Processor -- start 2024-03-04T07:41:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f125785f740 1 -- start start 2024-03-04T07:41:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f125785f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a00d0 0x561c0949a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f125785f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a0ac0 0x561c094b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f125785f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094bcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f125785f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a8c70 con 0x561c094a5c30 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f125785f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501ab250 con 0x561c094a00d0 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f125785f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a9270 con 0x561c094a0ac0 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f124de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a00d0 0x561c0949a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f124de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a00d0 0x561c0949a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48328/0 (socket says 172.21.15.116:48328) 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f124de00640 1 -- 172.21.15.116:0/4008598188 learned_addr learned my addr 172.21.15.116:0/4008598188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.715+0000 7f124e800640 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094bcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f124d400640 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a0ac0 0x561c094b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f124de00640 1 -- 172.21.15.116:0/4008598188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a0ac0 msgr2=0x561c094b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f124de00640 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a0ac0 0x561c094b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f124de00640 1 -- 172.21.15.116:0/4008598188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 msgr2=0x561c094bcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f124de00640 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094bcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f124de00640 1 -- 172.21.15.116:0/4008598188 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c090da170 con 0x561c094a00d0 2024-03-04T07:41:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f124de00640 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a00d0 0x561c0949a8e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f124001f440 tx=0x7f124001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d568050bedfff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f123e600640 1 -- 172.21.15.116:0/4008598188 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124001fdd0 con 0x561c094a00d0 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f123e600640 1 -- 172.21.15.116:0/4008598188 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1240030070 con 0x561c094a00d0 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.716+0000 7f123e600640 1 -- 172.21.15.116:0/4008598188 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124001e880 con 0x561c094a00d0 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 -- 172.21.15.116:0/4008598188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a00d0 msgr2=0x561c0949a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a00d0 0x561c0949a8e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f124001f440 tx=0x7f124001a000 comp rx=0 tx=0).stop 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 -- 172.21.15.116:0/4008598188 shutdown_connections 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a5c30 0x561c094bcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a0ac0 0x561c094b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 --2- 172.21.15.116:0/4008598188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a00d0 0x561c0949a8e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 -- 172.21.15.116:0/4008598188 >> 172.21.15.116:0/4008598188 conn(0x561c0949b640 msgr2=0x561c0949a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 -- 172.21.15.116:0/4008598188 shutdown_connections 2024-03-04T07:41:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 -- 172.21.15.116:0/4008598188 wait complete. 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.717+0000 7f125785f740 1 -- start start 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 1 Processor -- start 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a0ac0 0x561c094afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a5c30 0x561c094b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a98d0 con 0x561c094a0ac0 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501aabf0 con 0x561c094a5c30 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12501a9f30 con 0x561c094a00d0 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f125785f740 10 client.0 _ll_register_callbacks cb 0x7f1238005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f124d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a0ac0 0x561c094afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f124e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a5c30 0x561c094b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f124de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f124d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a0ac0 0x561c094afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60102/0 (socket says 172.21.15.116:60102) 2024-03-04T07:41:04.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.718+0000 7f124d400640 1 -- 172.21.15.116:0/1896133011 learned_addr learned my addr 172.21.15.116:0/1896133011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f124d400640 1 -- 172.21.15.116:0/1896133011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 msgr2=0x561c094b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f124d400640 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f124d400640 1 -- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a5c30 msgr2=0x561c094b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f124d400640 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a5c30 0x561c094b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f124d400640 1 -- 172.21.15.116:0/1896133011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c094b06e0 con 0x561c094a0ac0 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f124e800640 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a5c30 0x561c094b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f124d400640 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a0ac0 0x561c094afc60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1238005c20 tx=0x7f12380bb050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceda8b99a09f4b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f1233400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f1233400640 1 -- 172.21.15.116:0/1896133011 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12380bc680 con 0x561c094a0ac0 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f1233400640 1 -- 172.21.15.116:0/1896133011 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f12380d0d90 con 0x561c094a0ac0 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1240030070 con 0x561c094a0ac0 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f1233400640 1 -- 172.21.15.116:0/1896133011 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12380b8de0 con 0x561c094a0ac0 2024-03-04T07:41:04.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f125785f740 10 client.15252 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.719+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c0951b1d0 con 0x561c094a0ac0 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f1233400640 1 -- 172.21.15.116:0/1896133011 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1240030070 con 0x561c094a0ac0 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f1233400640 1 -- 172.21.15.116:0/1896133011 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f12380bbf90 con 0x561c094a0ac0 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f125785f740 10 client.15252 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f125785f740 10 client.15252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f125785f740 20 client.15252 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f125785f740 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c0951b9f0 0x561c0951dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f125785f740 4 client.15252 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561c0940d2b0 con 0x561c0951b9f0 2024-03-04T07:41:04.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.720+0000 7f124de00640 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c0951b9f0 0x561c0951dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.721+0000 7f124de00640 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c0951b9f0 0x561c0951dde0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dbf498d7bef0b91 server_cookie=2f29ec5916109067 in_seq=0 out_seq=0 2024-03-04T07:41:04.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.721+0000 7f1233400640 10 client.15252 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f1233400640 1 -- 172.21.15.116:0/1896133011 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561c0940d2b0 con 0x561c0951b9f0 2024-03-04T07:41:04.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f1233400640 10 client.15252 handle_command_reply: tid=0 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 client.15252 shutdown 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a0ac0 msgr2=0x561c094afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a0ac0 0x561c094afc60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1238005c20 tx=0x7f12380bb050 comp rx=0 tx=0).stop 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 shutdown_connections 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c0951b9f0 0x561c0951dde0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c094a5c30 0x561c094b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c094a0ac0 0x561c094afc60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 --2- 172.21.15.116:0/1896133011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c094a00d0 0x561c094b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f1233400640 0 client.15252 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 >> 172.21.15.116:0/1896133011 conn(0x561c0949b640 msgr2=0x561c0949aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 shutdown_connections 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 1 -- 172.21.15.116:0/1896133011 wait complete. 2024-03-04T07:41:04.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.722+0000 7f125785f740 20 client.15252 trim_cache size 0 max 16384 2024-03-04T07:41:04.726 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:41:04.726 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:41:04.726 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "6c911a83-e6da-4e14-8e24-90d1f6e19894", 2024-03-04T07:41:04.726 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:41:04.726 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f12340a1e00 msgr2=0x7f12340a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f12340a1e00 0x7f12340a42c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x561c094a0ec0 tx=0x7f1240005f90 comp rx=0 tx=0).stop 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 msgr2=0x7f125019a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f125019a060 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1238000c00 tx=0x7f123802c650 comp rx=0 tx=0).stop 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 shutdown_connections 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12501053c0 0x7f12501a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1250104a00 0x7f125019a060 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f12340a1e00 0x7f12340a42c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 --2- 172.21.15.116:0/4172347024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1250103810 0x7f1250199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 >> 172.21.15.116:0/4172347024 conn(0x7f12500fed80 msgr2=0x7f1250107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 shutdown_connections 2024-03-04T07:41:04.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.723+0000 7f1254c00640 1 -- 172.21.15.116:0/4172347024 wait complete. 2024-03-04T07:41:04.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:41:04.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:41:04.816 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:04 smithi116 ceph-mon[32150]: pgmap v410: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:04.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 -- 172.21.15.116:0/4116105976 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c075610 msgr2=0x7f493c073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 --2- 172.21.15.116:0/4116105976 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c075610 0x7f493c073a60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4930009930 tx=0x7f493002bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:04.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 -- 172.21.15.116:0/4116105976 shutdown_connections 2024-03-04T07:41:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 --2- 172.21.15.116:0/4116105976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f493c10dbb0 0x7f493c10ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 --2- 172.21.15.116:0/4116105976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f493c074030 0x7f493c0744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 --2- 172.21.15.116:0/4116105976 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c075610 0x7f493c073a60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 -- 172.21.15.116:0/4116105976 >> 172.21.15.116:0/4116105976 conn(0x7f493c0fcb70 msgr2=0x7f493c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 -- 172.21.15.116:0/4116105976 shutdown_connections 2024-03-04T07:41:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.952+0000 7f4944200640 1 -- 172.21.15.116:0/4116105976 wait complete. 2024-03-04T07:41:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 Processor -- start 2024-03-04T07:41:04.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 -- start start 2024-03-04T07:41:04.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c074030 0x7f493c19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f493c075610 0x7f493c19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f493c10dbb0 0x7f493c1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c113880 con 0x7f493c10dbb0 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c113700 con 0x7f493c074030 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4944200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c113a00 con 0x7f493c075610 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.954+0000 7f4941800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c074030 0x7f493c19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c074030 0x7f493c19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48364/0 (socket says 172.21.15.116:48364) 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 -- 172.21.15.116:0/1826009377 learned_addr learned my addr 172.21.15.116:0/1826009377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4942200640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f493c10dbb0 0x7f493c1a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4940e00640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f493c075610 0x7f493c19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 -- 172.21.15.116:0/1826009377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f493c075610 msgr2=0x7f493c19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f493c075610 0x7f493c19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 -- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f493c10dbb0 msgr2=0x7f493c1a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f493c10dbb0 0x7f493c1a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 -- 172.21.15.116:0/1826009377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c1a5b40 con 0x7f493c074030 2024-03-04T07:41:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.955+0000 7f4941800640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c074030 0x7f493c19df20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4930000c00 tx=0x7f493002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.956+0000 7f493a000640 1 -- 172.21.15.116:0/1826009377 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4930043020 con 0x7f493c074030 2024-03-04T07:41:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.956+0000 7f493a000640 1 -- 172.21.15.116:0/1826009377 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49300054e0 con 0x7f493c074030 2024-03-04T07:41:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.956+0000 7f493a000640 1 -- 172.21.15.116:0/1826009377 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4930003c40 con 0x7f493c074030 2024-03-04T07:41:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.956+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c1a5dd0 con 0x7f493c074030 2024-03-04T07:41:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.956+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c1a6290 con 0x7f493c074030 2024-03-04T07:41:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f493a000640 1 -- 172.21.15.116:0/1826009377 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4930046020 con 0x7f493c074030 2024-03-04T07:41:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 Processor -- start 2024-03-04T07:41:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 -- start start 2024-03-04T07:41:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5321f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5327040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5329b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f524f090 con 0x55b6f53276b0 2024-03-04T07:41:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f5286d30 con 0x55b6f5321b50 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.959+0000 7f4946e00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f522fa30 con 0x55b6f5322540 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5321f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5321f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48366/0 (socket says 172.21.15.116:48366) 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 -- 172.21.15.116:0/644634955 learned_addr learned my addr 172.21.15.116:0/644634955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4940e00640 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5329b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4941800640 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5327040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 -- 172.21.15.116:0/644634955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 msgr2=0x55b6f5327040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5327040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 -- 172.21.15.116:0/644634955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 msgr2=0x55b6f5329b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5329b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 -- 172.21.15.116:0/644634955 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6f50657d0 con 0x55b6f5321b50 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.960+0000 7f4942200640 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5321f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4934009fd0 tx=0x7f49340118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd8adb4c186980f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f492ea00640 1 -- 172.21.15.116:0/644634955 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934004d20 con 0x55b6f5321b50 2024-03-04T07:41:04.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f492ea00640 1 -- 172.21.15.116:0/644634955 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4934004050 con 0x55b6f5321b50 2024-03-04T07:41:04.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f492ea00640 1 -- 172.21.15.116:0/644634955 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934005260 con 0x55b6f5321b50 2024-03-04T07:41:04.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 -- 172.21.15.116:0/644634955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 msgr2=0x55b6f5321f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5321f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4934009fd0 tx=0x7f49340118d0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f493a000640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49240a1e00 0x7f49240a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 -- 172.21.15.116:0/644634955 shutdown_connections 2024-03-04T07:41:04.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5329b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5327040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 --2- 172.21.15.116:0/644634955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5321f70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 -- 172.21.15.116:0/644634955 >> 172.21.15.116:0/644634955 conn(0x55b6f531d0c0 msgr2=0x55b6f531d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4941800640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49240a1e00 0x7f49240a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.961+0000 7f4946e00740 1 -- 172.21.15.116:0/644634955 shutdown_connections 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.962+0000 7f493a000640 1 -- 172.21.15.116:0/1826009377 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f49300efdf0 con 0x7f493c074030 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.962+0000 7f4946e00740 1 -- 172.21.15.116:0/644634955 wait complete. 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.962+0000 7f4946e00740 1 -- start start 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.962+0000 7f4941800640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49240a1e00 0x7f49240a42c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f49300ed8b0 tx=0x7f4930005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.962+0000 7f4946e00740 1 Processor -- start 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4946e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5321b50 0x55b6f532bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4946e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5322540 0x55b6f532c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4946e00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f53276b0 0x55b6f53334a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4946e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f52250c0 con 0x55b6f5322540 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4946e00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f5282dd0 con 0x55b6f53276b0 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4946e00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f532a660 con 0x55b6f5321b50 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4946e00740 10 client.0 _ll_register_callbacks cb 0x55b6f5242860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4940e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5322540 0x55b6f532c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4940e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5322540 0x55b6f532c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60140/0 (socket says 172.21.15.116:60140) 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4940e00640 1 -- 172.21.15.116:0/4119012365 learned_addr learned my addr 172.21.15.116:0/4119012365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4942200640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5321b50 0x55b6f532bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.963+0000 7f4942200640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f53276b0 0x55b6f53334a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4940e00640 1 -- 172.21.15.116:0/4119012365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5321b50 msgr2=0x55b6f532bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4940e00640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5321b50 0x55b6f532bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4940e00640 1 -- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f53276b0 msgr2=0x55b6f53334a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4940e00640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f53276b0 0x55b6f53334a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4940e00640 1 -- 172.21.15.116:0/4119012365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6f4f9b3f0 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4942200640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5321b50 0x55b6f532bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4940e00640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5322540 0x55b6f532c460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4928012700 tx=0x7f4928012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc08b84170eea03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4923400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4923400640 1 -- 172.21.15.116:0/4119012365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928011c10 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4923400640 1 -- 172.21.15.116:0/4119012365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4928011db0 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4934004050 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.964+0000 7f4923400640 1 -- 172.21.15.116:0/4119012365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f492800f460 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 10 client.15264 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b6f4f5b0f0 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4923400640 1 -- 172.21.15.116:0/4119012365 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4934004050 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4923400640 1 -- 172.21.15.116:0/4119012365 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f492801c620 con 0x55b6f5322540 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 10 client.15264 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:04.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 10 client.15264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:04.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 20 client.15264 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:04.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5337b00 0x55b6f5339f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 4 client.15264 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:04.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6f5294270 con 0x55b6f5337b00 2024-03-04T07:41:04.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.965+0000 7f4942200640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5337b00 0x55b6f5339f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.966+0000 7f4942200640 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5337b00 0x55b6f5339f20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd36c8d47a9e3e53 server_cookie=bd4eef281d35aa95 in_seq=0 out_seq=0 2024-03-04T07:41:04.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.966+0000 7f4923400640 10 client.15264 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4923400640 1 -- 172.21.15.116:0/4119012365 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b6f5294270 con 0x55b6f5337b00 2024-03-04T07:41:04.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4923400640 10 client.15264 handle_command_reply: tid=0 2024-03-04T07:41:04.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 client.15264 shutdown 2024-03-04T07:41:04.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5322540 msgr2=0x55b6f532c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5322540 0x55b6f532c460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4928012700 tx=0x7f4928012bc0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 shutdown_connections 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5337b00 0x55b6f5339f20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f53276b0 0x55b6f53334a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5322540 0x55b6f532c460 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 --2- 172.21.15.116:0/4119012365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5321b50 0x55b6f532bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4923400640 0 client.15264 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 >> 172.21.15.116:0/4119012365 conn(0x55b6f531d0c0 msgr2=0x55b6f531c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 shutdown_connections 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 1 -- 172.21.15.116:0/4119012365 wait complete. 2024-03-04T07:41:04.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.967+0000 7f4946e00740 20 client.15264 trim_cache size 0 max 16384 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 Processor -- start 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 -- start start 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5321b50 0x55b6f5320a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5322540 0x55b6f5333460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f53276b0 0x55b6f533e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1ad070 con 0x55b6f5321b50 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1af650 con 0x55b6f5322540 2024-03-04T07:41:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4946e00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1ad670 con 0x55b6f53276b0 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4942200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5321b50 0x55b6f5320a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4940e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5322540 0x55b6f5333460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4940e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5322540 0x55b6f5333460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48376/0 (socket says 172.21.15.116:48376) 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.985+0000 7f4940e00640 1 -- 172.21.15.116:0/292719698 learned_addr learned my addr 172.21.15.116:0/292719698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4942200640 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f53276b0 0x55b6f533e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4940e00640 1 -- 172.21.15.116:0/292719698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f53276b0 msgr2=0x55b6f533e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4940e00640 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f53276b0 0x55b6f533e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4940e00640 1 -- 172.21.15.116:0/292719698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5321b50 msgr2=0x55b6f5320a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4940e00640 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5321b50 0x55b6f5320a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4940e00640 1 -- 172.21.15.116:0/292719698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6f531de30 con 0x55b6f5322540 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4942200640 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f53276b0 0x55b6f533e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:41:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f4940e00640 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5322540 0x55b6f5333460 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f492801ca90 tx=0x7f492801df50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a02ef032527db5f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.986+0000 7f492e000640 1 -- 172.21.15.116:0/292719698 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928011c10 con 0x55b6f5322540 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f492e000640 1 -- 172.21.15.116:0/292719698 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4928011db0 con 0x55b6f5322540 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f492e000640 1 -- 172.21.15.116:0/292719698 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928005430 con 0x55b6f5322540 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 -- 172.21.15.116:0/292719698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5322540 msgr2=0x55b6f5333460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5322540 0x55b6f5333460 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f492801ca90 tx=0x7f492801df50 comp rx=0 tx=0).stop 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 -- 172.21.15.116:0/292719698 shutdown_connections 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f53276b0 0x55b6f533e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5322540 0x55b6f5333460 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 --2- 172.21.15.116:0/292719698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f5321b50 0x55b6f5320a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 -- 172.21.15.116:0/292719698 >> 172.21.15.116:0/292719698 conn(0x55b6f531d0c0 msgr2=0x55b6f5320320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 -- 172.21.15.116:0/292719698 shutdown_connections 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.987+0000 7f4946e00740 1 -- 172.21.15.116:0/292719698 wait complete. 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 -- start start 2024-03-04T07:41:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 Processor -- start 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5331210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5331750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5332760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1adcd0 con 0x55b6f53276b0 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1aeff0 con 0x55b6f5321b50 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1ae330 con 0x55b6f5322540 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.988+0000 7f4946e00740 10 client.0 _ll_register_callbacks cb 0x7f4928009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4942200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5331210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5331750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5331750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:34530/0 (socket says 172.21.15.116:34530) 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4942200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5331210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48386/0 (socket says 172.21.15.116:48386) 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 -- 172.21.15.116:0/2499820435 learned_addr learned my addr 172.21.15.116:0/2499820435 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4942200640 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5332760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 -- 172.21.15.116:0/2499820435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 msgr2=0x55b6f5331210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5331210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 -- 172.21.15.116:0/2499820435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 msgr2=0x55b6f5332760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5332760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 -- 172.21.15.116:0/2499820435 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6f5332d50 con 0x55b6f5322540 2024-03-04T07:41:04.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4940e00640 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5331750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4928012700 tx=0x7f4928005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49907eb7349d97f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:04.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.989+0000 7f4923400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:41:04.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.990+0000 7f4923400640 1 -- 172.21.15.116:0/2499820435 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492803d060 con 0x55b6f5322540 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.990+0000 7f4923400640 1 -- 172.21.15.116:0/2499820435 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4928018050 con 0x55b6f5322540 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.990+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4928011db0 con 0x55b6f5322540 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.990+0000 7f4923400640 1 -- 172.21.15.116:0/2499820435 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f492801c630 con 0x55b6f5322540 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.990+0000 7f4946e00740 10 client.25354 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.990+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b6f5393970 con 0x55b6f5322540 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4923400640 1 -- 172.21.15.116:0/2499820435 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4928011db0 con 0x55b6f5322540 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4923400640 1 -- 172.21.15.116:0/2499820435 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4928042070 con 0x55b6f5322540 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4946e00740 10 client.25354 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4946e00740 10 client.25354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4946e00740 20 client.25354 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4946e00740 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5394260 0x55b6f5396680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4946e00740 4 client.25354 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b6f532bd10 con 0x55b6f5394260 2024-03-04T07:41:04.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.991+0000 7f4942200640 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5394260 0x55b6f5396680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4942200640 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5394260 0x55b6f5396680 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51eada205b830985 server_cookie=f874fc902f12c443 in_seq=0 out_seq=0 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4923400640 10 client.25354 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4923400640 1 -- 172.21.15.116:0/2499820435 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b6f532bd10 con 0x55b6f5394260 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4923400640 10 client.25354 handle_command_reply: tid=0 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 client.25354 shutdown 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 msgr2=0x55b6f5331750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5331750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4928012700 tx=0x7f4928005f80 comp rx=0 tx=0).stop 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 shutdown_connections 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6f5394260 0x55b6f5396680 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f53276b0 0x55b6f5332760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6f5322540 0x55b6f5331750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4923400640 0 client.25354 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 --2- 172.21.15.116:0/2499820435 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6f5321b50 0x55b6f5331210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 >> 172.21.15.116:0/2499820435 conn(0x55b6f531d0c0 msgr2=0x55b6f5320bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 shutdown_connections 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.992+0000 7f4946e00740 1 -- 172.21.15.116:0/2499820435 wait complete. 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4946e00740 20 client.25354 trim_cache size 0 max 16384 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:41:04.997 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:41:04.998 INFO:teuthology.orchestra.run.smithi116.stdout: "6c911a83-e6da-4e14-8e24-90d1f6e19894": { 2024-03-04T07:41:04.998 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:41:04.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "6c911a83-e6da-4e14-8e24-90d1f6e19894", 2024-03-04T07:41:04.998 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:41:04.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:41:04.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:41:04.998 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49240a1e00 msgr2=0x7f49240a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49240a1e00 0x7f49240a42c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f49300ed8b0 tx=0x7f4930005e50 comp rx=0 tx=0).stop 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c074030 msgr2=0x7f493c19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c074030 0x7f493c19df20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4930000c00 tx=0x7f493002c650 comp rx=0 tx=0).stop 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 shutdown_connections 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f493c10dbb0 0x7f493c1a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49240a1e00 0x7f49240a42c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f493c075610 0x7f493c19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 --2- 172.21.15.116:0/1826009377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f493c074030 0x7f493c19df20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 >> 172.21.15.116:0/1826009377 conn(0x7f493c0fcb70 msgr2=0x7f493c100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 shutdown_connections 2024-03-04T07:41:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:04.993+0000 7f4944200640 1 -- 172.21.15.116:0/1826009377 wait complete. 2024-03-04T07:41:05.012 INFO:tasks.cephfs.filesystem:scrub status for tag:6c911a83-e6da-4e14-8e24-90d1f6e19894 - {'path': '/', 'tag': '6c911a83-e6da-4e14-8e24-90d1f6e19894', 'options': 'recursive,force'} 2024-03-04T07:41:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:04 smithi152 ceph-mon[37154]: pgmap v410: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:05.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:04 smithi133 ceph-mon[36505]: pgmap v410: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:05 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:41:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:05 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:06.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:05 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:41:06.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:05 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:06.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:05 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:41:06.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:05 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:06.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:05 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:41:06.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:05 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:06.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:05 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:41:06.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:05 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:06.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:05 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:41:06.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:05 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:06 smithi116 ceph-mon[32150]: pgmap v411: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:06 smithi152 ceph-mon[37154]: pgmap v411: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:07.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:06 smithi133 ceph-mon[36505]: pgmap v411: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:08 smithi116 ceph-mon[32150]: pgmap v412: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-04T07:41:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:08 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:41:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:08 smithi152 ceph-mon[37154]: pgmap v412: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-04T07:41:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:08 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:41:09.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:08 smithi133 ceph-mon[36505]: pgmap v412: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-04T07:41:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:08 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:41:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:10 smithi116 ceph-mon[32150]: pgmap v413: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:10 smithi152 ceph-mon[37154]: pgmap v413: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:10 smithi133 ceph-mon[36505]: pgmap v413: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 426 B/s wr, 3 op/s 2024-03-04T07:41:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 -- 172.21.15.116:0/942489698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4101ea0 msgr2=0x7efee4104290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 --2- 172.21.15.116:0/942489698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4101ea0 0x7efee4104290 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efed4009900 tx=0x7efed402bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 -- 172.21.15.116:0/942489698 shutdown_connections 2024-03-04T07:41:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 --2- 172.21.15.116:0/942489698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee4107100 0x7efee4109550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 --2- 172.21.15.116:0/942489698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee41047d0 0x7efee4106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 --2- 172.21.15.116:0/942489698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4101ea0 0x7efee4104290 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 -- 172.21.15.116:0/942489698 >> 172.21.15.116:0/942489698 conn(0x7efee40fb8b0 msgr2=0x7efee40fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 -- 172.21.15.116:0/942489698 shutdown_connections 2024-03-04T07:41:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.486+0000 7efee3e00640 1 -- 172.21.15.116:0/942489698 wait complete. 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.487+0000 7efee3e00640 1 Processor -- start 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.487+0000 7efee3e00640 1 -- start start 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.487+0000 7efee3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101ea0 0x7efee41976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.487+0000 7efee3e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee41047d0 0x7efee4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.487+0000 7efee3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4107100 0x7efee419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee410d100 con 0x7efee4101ea0 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee3e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee410cf80 con 0x7efee4107100 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee3e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee410d280 con 0x7efee41047d0 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101ea0 0x7efee41976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101ea0 0x7efee41976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34118/0 (socket says 172.21.15.116:34118) 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 -- 172.21.15.116:0/1360543236 learned_addr learned my addr 172.21.15.116:0/1360543236 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee3400640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4107100 0x7efee419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2000640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee41047d0 0x7efee4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 -- 172.21.15.116:0/1360543236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee41047d0 msgr2=0x7efee4197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee41047d0 0x7efee4197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 -- 172.21.15.116:0/1360543236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4107100 msgr2=0x7efee419ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4107100 0x7efee419ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee2a00640 1 -- 172.21.15.116:0/1360543236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee419f300 con 0x7efee4101ea0 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.488+0000 7efee3400640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4107100 0x7efee419ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.489+0000 7efee2a00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101ea0 0x7efee41976e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efed4000c00 tx=0x7efed402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.489+0000 7efee8a00640 1 -- 172.21.15.116:0/1360543236 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4043020 con 0x7efee4101ea0 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.489+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee419f590 con 0x7efee4101ea0 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.489+0000 7efee8a00640 1 -- 172.21.15.116:0/1360543236 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efed40054e0 con 0x7efee4101ea0 2024-03-04T07:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.489+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee419fb30 con 0x7efee4101ea0 2024-03-04T07:41:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.490+0000 7efee8a00640 1 -- 172.21.15.116:0/1360543236 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4003be0 con 0x7efee4101ea0 2024-03-04T07:41:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.493+0000 7efeecb76740 1 Processor -- start 2024-03-04T07:41:11.494 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.493+0000 7efeecb76740 1 -- start start 2024-03-04T07:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.493+0000 7efeecb76740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f6fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.493+0000 7efee8a00640 1 -- 172.21.15.116:0/1360543236 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7efed4046020 con 0x7efee4101ea0 2024-03-04T07:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.493+0000 7efeecb76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f70400 0x55f3b4f74f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.493+0000 7efeecb76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f77a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.493+0000 7efeecb76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4ed5370 con 0x55f3b4f75570 2024-03-04T07:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efeecb76740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4e7e550 con 0x55f3b4f6fa10 2024-03-04T07:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efeecb76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4ed1410 con 0x55f3b4f70400 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f6fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f6fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48274/0 (socket says 172.21.15.116:48274) 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 -- 172.21.15.116:0/231916316 learned_addr learned my addr 172.21.15.116:0/231916316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee2000640 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f77a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee2a00640 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f70400 0x55f3b4f74f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 -- 172.21.15.116:0/231916316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f70400 msgr2=0x55f3b4f74f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f70400 0x55f3b4f74f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 -- 172.21.15.116:0/231916316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 msgr2=0x55f3b4f77a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f77a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee3400640 1 -- 172.21.15.116:0/231916316 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3b4e923c0 con 0x55f3b4f6fa10 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.494+0000 7efee2000640 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f77a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.495+0000 7efee3400640 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f6fe30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efedc015bf0 tx=0x7efedc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b91f16966b98566 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.495+0000 7efecfe00640 1 -- 172.21.15.116:0/231916316 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc003c30 con 0x55f3b4f6fa10 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.495+0000 7efecfe00640 1 -- 172.21.15.116:0/231916316 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efedc003dd0 con 0x55f3b4f6fa10 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.495+0000 7efecfe00640 1 -- 172.21.15.116:0/231916316 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc004030 con 0x55f3b4f6fa10 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.495+0000 7efee8a00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efec80a1d70 0x7efec80a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.495+0000 7efee8a00640 1 -- 172.21.15.116:0/1360543236 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7efed40efd90 con 0x7efee4101ea0 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.495+0000 7efee2a00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efec80a1d70 0x7efec80a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 -- 172.21.15.116:0/231916316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 msgr2=0x55f3b4f6fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f6fe30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efedc015bf0 tx=0x7efedc00f790 comp rx=0 tx=0).stop 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 -- 172.21.15.116:0/231916316 shutdown_connections 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f77a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f70400 0x55f3b4f74f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 --2- 172.21.15.116:0/231916316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f6fe30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 -- 172.21.15.116:0/231916316 >> 172.21.15.116:0/231916316 conn(0x55f3b4f6af80 msgr2=0x55f3b4f6b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efee2a00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efec80a1d70 0x7efec80a4230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efed40b8f20 tx=0x7efed4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 -- 172.21.15.116:0/231916316 shutdown_connections 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 -- 172.21.15.116:0/231916316 wait complete. 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.496+0000 7efeecb76740 1 -- start start 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 1 Processor -- start 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f79e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f70400 0x55f3b4f7a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.500 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f75570 0x55f3b4f81320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4f78520 con 0x55f3b4f70400 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4e919b0 con 0x55f3b4f6fa10 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3b4f786a0 con 0x55f3b4f75570 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efeecb76740 10 client.0 _ll_register_callbacks cb 0x55f3b4ecd340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efee2000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f70400 0x55f3b4f7a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efee3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f79e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efee2000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f70400 0x55f3b4f7a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34142/0 (socket says 172.21.15.116:34142) 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efee2000640 1 -- 172.21.15.116:0/1922939600 learned_addr learned my addr 172.21.15.116:0/1922939600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efee3400640 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f79e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48278/0 (socket says 172.21.15.116:48278) 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.497+0000 7efee3400640 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f75570 0x55f3b4f81320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efee2000640 1 -- 172.21.15.116:0/1922939600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f75570 msgr2=0x55f3b4f81320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efee2000640 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f75570 0x55f3b4f81320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efee2000640 1 -- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 msgr2=0x55f3b4f79e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efee2000640 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f79e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efee2000640 1 -- 172.21.15.116:0/1922939600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3b4bea210 con 0x55f3b4f70400 2024-03-04T07:41:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efee2000640 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f70400 0x55f3b4f7a350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efed0012700 tx=0x7efed0012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea8a9c06ba163e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efeccc00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:41:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efeccc00640 1 -- 172.21.15.116:0/1922939600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0004050 con 0x55f3b4f70400 2024-03-04T07:41:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efedc003dd0 con 0x55f3b4f70400 2024-03-04T07:41:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.498+0000 7efeccc00640 1 -- 172.21.15.116:0/1922939600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7efed00041f0 con 0x55f3b4f70400 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeccc00640 1 -- 172.21.15.116:0/1922939600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efed00043f0 con 0x55f3b4f70400 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeecb76740 10 client.15282 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f3b4cb6620 con 0x55f3b4f70400 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeccc00640 1 -- 172.21.15.116:0/1922939600 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efedc003dd0 con 0x55f3b4f70400 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeccc00640 1 -- 172.21.15.116:0/1922939600 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efed0003c40 con 0x55f3b4f70400 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeecb76740 10 client.15282 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeecb76740 10 client.15282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeecb76740 20 client.15282 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.499+0000 7efeecb76740 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4f85c40 0x55f3b4f88030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.500+0000 7efeecb76740 4 client.15282 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.500+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3b4ee28b0 con 0x55f3b4f85c40 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.500+0000 7efee3400640 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4f85c40 0x55f3b4f88030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.500+0000 7efee3400640 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4f85c40 0x55f3b4f88030 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba13779cbfd8b671 server_cookie=7fd5275786c132cf in_seq=0 out_seq=0 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.500+0000 7efeccc00640 10 client.15282 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeccc00640 1 -- 172.21.15.116:0/1922939600 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f3b4ee28b0 con 0x55f3b4f85c40 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeccc00640 10 client.15282 handle_command_reply: tid=0 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 client.15282 shutdown 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f70400 msgr2=0x55f3b4f7a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f70400 0x55f3b4f7a350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efed0012700 tx=0x7efed0012bc0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 shutdown_connections 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4f85c40 0x55f3b4f88030 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f75570 0x55f3b4f81320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f70400 0x55f3b4f7a350 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 --2- 172.21.15.116:0/1922939600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f79e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 >> 172.21.15.116:0/1922939600 conn(0x55f3b4f6af80 msgr2=0x55f3b4f6a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeccc00640 0 client.15282 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 shutdown_connections 2024-03-04T07:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.501+0000 7efeecb76740 1 -- 172.21.15.116:0/1922939600 wait complete. 2024-03-04T07:41:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.502+0000 7efeecb76740 20 client.15282 trim_cache size 0 max 16384 2024-03-04T07:41:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.518+0000 7efeecb76740 1 Processor -- start 2024-03-04T07:41:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.518+0000 7efeecb76740 1 -- start start 2024-03-04T07:41:11.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.518+0000 7efeecb76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.518+0000 7efeecb76740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f75570 0x55f3b4f71d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.518+0000 7efeecb76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f664e0 0x55f3b4f812e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.518+0000 7efeecb76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee41a4650 con 0x55f3b4f6fa10 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.518+0000 7efeecb76740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee41a6c30 con 0x55f3b4f75570 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efeecb76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee41a4c50 con 0x55f3b4f664e0 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f75570 0x55f3b4f71d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f75570 0x55f3b4f71d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48290/0 (socket says 172.21.15.116:48290) 2024-03-04T07:41:11.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 -- 172.21.15.116:0/3764049972 learned_addr learned my addr 172.21.15.116:0/3764049972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:11.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 -- 172.21.15.116:0/3764049972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f664e0 msgr2=0x55f3b4f812e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T07:41:11.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 --2- 172.21.15.116:0/3764049972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f664e0 0x55f3b4f812e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 -- 172.21.15.116:0/3764049972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f6fa10 msgr2=0x55f3b4f717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 --2- 172.21.15.116:0/3764049972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 -- 172.21.15.116:0/3764049972 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3b4f66090 con 0x55f3b4f75570 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.519+0000 7efee2000640 1 --2- 172.21.15.116:0/3764049972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f75570 0x55f3b4f71d00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efed0012700 tx=0x7efed0009d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba855424249bcc91 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efecf400640 1 -- 172.21.15.116:0/3764049972 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed002c070 con 0x55f3b4f75570 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efecf400640 1 -- 172.21.15.116:0/3764049972 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efed0010e30 con 0x55f3b4f75570 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efecf400640 1 -- 172.21.15.116:0/3764049972 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed000fa80 con 0x55f3b4f75570 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efeecb76740 1 -- 172.21.15.116:0/3764049972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f75570 msgr2=0x55f3b4f71d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efeecb76740 1 --2- 172.21.15.116:0/3764049972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f75570 0x55f3b4f71d00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efed0012700 tx=0x7efed0009d50 comp rx=0 tx=0).stop 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efeecb76740 1 -- 172.21.15.116:0/3764049972 shutdown_connections 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efeecb76740 1 --2- 172.21.15.116:0/3764049972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f664e0 0x55f3b4f812e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efeecb76740 1 --2- 172.21.15.116:0/3764049972 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f75570 0x55f3b4f71d00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efeecb76740 1 --2- 172.21.15.116:0/3764049972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.520+0000 7efeecb76740 1 -- 172.21.15.116:0/3764049972 >> 172.21.15.116:0/3764049972 conn(0x55f3b4f6af80 msgr2=0x55f3b4f6b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.521+0000 7efeecb76740 1 -- 172.21.15.116:0/3764049972 shutdown_connections 2024-03-04T07:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.521+0000 7efeecb76740 1 -- 172.21.15.116:0/3764049972 wait complete. 2024-03-04T07:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.521+0000 7efeecb76740 1 -- start start 2024-03-04T07:41:11.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.521+0000 7efeecb76740 1 Processor -- start 2024-03-04T07:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efeecb76740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f664e0 0x55f3b4f7f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efeecb76740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f7f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efeecb76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f807c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efeecb76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee41a52b0 con 0x55f3b4f75570 2024-03-04T07:41:11.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efeecb76740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee41a65d0 con 0x55f3b4f664e0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efeecb76740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee41a5910 con 0x55f3b4f6fa10 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efeecb76740 10 client.0 _ll_register_callbacks cb 0x7efed0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f664e0 0x55f3b4f7f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f807c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f664e0 0x55f3b4f7f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:48302/0 (socket says 172.21.15.116:48302) 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 -- 172.21.15.116:0/1110648415 learned_addr learned my addr 172.21.15.116:0/1110648415 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee2000640 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f7f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 -- 172.21.15.116:0/1110648415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f6fa10 msgr2=0x55f3b4f7f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f7f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 -- 172.21.15.116:0/1110648415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 msgr2=0x55f3b4f807c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f807c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.522+0000 7efee3400640 1 -- 172.21.15.116:0/1110648415 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3b4f80db0 con 0x55f3b4f664e0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efee3400640 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f664e0 0x55f3b4f7f270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efedc005c40 tx=0x7efedc017040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aee132ec4a2a7e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeccc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeccc00640 1 -- 172.21.15.116:0/1110648415 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc026040 con 0x55f3b4f664e0 2024-03-04T07:41:11.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efed0010e30 con 0x55f3b4f664e0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeccc00640 1 -- 172.21.15.116:0/1110648415 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7efedc0330b0 con 0x55f3b4f664e0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeccc00640 1 -- 172.21.15.116:0/1110648415 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efedc0332b0 con 0x55f3b4f664e0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeecb76740 10 client.25394 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f3b4fe1d90 con 0x55f3b4f664e0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeccc00640 1 -- 172.21.15.116:0/1110648415 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efed0010e30 con 0x55f3b4f664e0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeccc00640 1 -- 172.21.15.116:0/1110648415 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efedc02c040 con 0x55f3b4f664e0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeecb76740 10 client.25394 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeecb76740 10 client.25394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeecb76740 20 client.25394 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.523+0000 7efeecb76740 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4fe2450 0x55f3b4fe4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.524+0000 7efeecb76740 4 client.25394 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.524+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f3b4edcc00 con 0x55f3b4fe2450 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.524+0000 7efee2000640 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4fe2450 0x55f3b4fe4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.524+0000 7efee2000640 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4fe2450 0x55f3b4fe4870 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=119e641b25031487 server_cookie=467f697f93d561c in_seq=0 out_seq=0 2024-03-04T07:41:11.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.524+0000 7efeccc00640 10 client.25394 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeccc00640 1 -- 172.21.15.116:0/1110648415 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f3b4edcc00 con 0x55f3b4fe2450 2024-03-04T07:41:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeccc00640 10 client.25394 handle_command_reply: tid=0 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 client.25394 shutdown 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f664e0 msgr2=0x55f3b4f7f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f664e0 0x55f3b4f7f270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efedc005c40 tx=0x7efedc017040 comp rx=0 tx=0).stop 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 shutdown_connections 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f3b4fe2450 0x55f3b4fe4870 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3b4f75570 0x55f3b4f807c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f3b4f6fa10 0x55f3b4f7f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeccc00640 0 client.25394 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 --2- 172.21.15.116:0/1110648415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f3b4f664e0 0x55f3b4f7f270 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.531+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 >> 172.21.15.116:0/1110648415 conn(0x55f3b4f6af80 msgr2=0x55f3b4f79df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.532+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 shutdown_connections 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.532+0000 7efeecb76740 1 -- 172.21.15.116:0/1110648415 wait complete. 2024-03-04T07:41:11.533 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.532+0000 7efeecb76740 20 client.25394 trim_cache size 0 max 16384 2024-03-04T07:41:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:41:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:41:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:41:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:41:11.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.532+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efec80a1d70 msgr2=0x7efec80a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.532+0000 7efee3e00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efec80a1d70 0x7efec80a4230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efed40b8f20 tx=0x7efed4009260 comp rx=0 tx=0).stop 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.532+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101ea0 msgr2=0x7efee41976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.532+0000 7efee3e00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101ea0 0x7efee41976e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efed4000c00 tx=0x7efed402c650 comp rx=0 tx=0).stop 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 shutdown_connections 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efee4107100 0x7efee419ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efec80a1d70 0x7efec80a4230 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efee41047d0 0x7efee4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 --2- 172.21.15.116:0/1360543236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4101ea0 0x7efee41976e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:11.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 >> 172.21.15.116:0/1360543236 conn(0x7efee40fb8b0 msgr2=0x7efee4102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 shutdown_connections 2024-03-04T07:41:11.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:11.533+0000 7efee3e00640 1 -- 172.21.15.116:0/1360543236 wait complete. 2024-03-04T07:41:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:12 smithi116 ceph-mon[32150]: pgmap v414: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-04T07:41:13.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:12 smithi152 ceph-mon[37154]: pgmap v414: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-04T07:41:13.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:12 smithi133 ceph-mon[36505]: pgmap v414: 97 pgs: 97 active+clean; 652 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-04T07:41:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:14 smithi116 ceph-mon[32150]: pgmap v415: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-04T07:41:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:14 smithi152 ceph-mon[37154]: pgmap v415: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-04T07:41:15.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:14 smithi133 ceph-mon[36505]: pgmap v415: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-04T07:41:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:16 smithi116 ceph-mon[32150]: pgmap v416: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-04T07:41:17.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:16 smithi152 ceph-mon[37154]: pgmap v416: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-04T07:41:17.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:16 smithi133 ceph-mon[36505]: pgmap v416: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-04T07:41:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:18 smithi116 ceph-mon[32150]: pgmap v417: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-04T07:41:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:18 smithi152 ceph-mon[37154]: pgmap v417: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-04T07:41:19.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:18 smithi133 ceph-mon[36505]: pgmap v417: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-04T07:41:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:20 smithi116 ceph-mon[32150]: pgmap v418: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-04T07:41:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:20 smithi152 ceph-mon[37154]: pgmap v418: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-04T07:41:21.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:20 smithi133 ceph-mon[36505]: pgmap v418: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-04T07:41:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:22 smithi116 ceph-mon[32150]: pgmap v419: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-04T07:41:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:22 smithi152 ceph-mon[37154]: pgmap v419: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-04T07:41:23.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:22 smithi133 ceph-mon[36505]: pgmap v419: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.670+0000 7fa914e00640 1 -- 172.21.15.116:0/508432000 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa910102090 msgr2=0x7fa910104480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.670+0000 7fa914e00640 1 --2- 172.21.15.116:0/508432000 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa910102090 0x7fa910104480 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa900009900 tx=0x7fa90002bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.671+0000 7fa914e00640 1 -- 172.21.15.116:0/508432000 shutdown_connections 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.671+0000 7fa914e00640 1 --2- 172.21.15.116:0/508432000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9101072f0 0x7fa910109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.671+0000 7fa914e00640 1 --2- 172.21.15.116:0/508432000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9101049c0 0x7fa910106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.671+0000 7fa914e00640 1 --2- 172.21.15.116:0/508432000 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa910102090 0x7fa910104480 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.671+0000 7fa914e00640 1 -- 172.21.15.116:0/508432000 >> 172.21.15.116:0/508432000 conn(0x7fa9100fba80 msgr2=0x7fa9100fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.671+0000 7fa914e00640 1 -- 172.21.15.116:0/508432000 shutdown_connections 2024-03-04T07:41:24.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.671+0000 7fa914e00640 1 -- 172.21.15.116:0/508432000 wait complete. 2024-03-04T07:41:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.672+0000 7fa914e00640 1 Processor -- start 2024-03-04T07:41:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.672+0000 7fa914e00640 1 -- start start 2024-03-04T07:41:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa914e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910102090 0x7fa910101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa914e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9101049c0 0x7fa910100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa914e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9101072f0 0x7fa910100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa914e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91010d2f0 con 0x7fa9101049c0 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa914e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91010d170 con 0x7fa9101072f0 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa914e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91010d470 con 0x7fa910102090 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910102090 0x7fa910101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9101049c0 0x7fa910100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9101072f0 0x7fa910100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910102090 0x7fa910101a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53764/0 (socket says 172.21.15.116:53764) 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90de00640 1 -- 172.21.15.116:0/3571382331 learned_addr learned my addr 172.21.15.116:0/3571382331 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90e800640 1 -- 172.21.15.116:0/3571382331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910102090 msgr2=0x7fa910101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90e800640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910102090 0x7fa910101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90e800640 1 -- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9101049c0 msgr2=0x7fa910100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90e800640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9101049c0 0x7fa910100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.673+0000 7fa90e800640 1 -- 172.21.15.116:0/3571382331 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa910100e10 con 0x7fa9101072f0 2024-03-04T07:41:24.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.674+0000 7fa90e800640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9101072f0 0x7fa910100580 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa90800da10 tx=0x7fa90800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:24.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.674+0000 7fa906a00640 1 -- 172.21.15.116:0/3571382331 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90800bd70 con 0x7fa9101072f0 2024-03-04T07:41:24.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.674+0000 7fa906a00640 1 -- 172.21.15.116:0/3571382331 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa908003e50 con 0x7fa9101072f0 2024-03-04T07:41:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.674+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9101a19a0 con 0x7fa9101072f0 2024-03-04T07:41:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.674+0000 7fa906a00640 1 -- 172.21.15.116:0/3571382331 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9080106a0 con 0x7fa9101072f0 2024-03-04T07:41:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.674+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9101a1ef0 con 0x7fa9101072f0 2024-03-04T07:41:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.678+0000 7fa9179ed740 1 Processor -- start 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa906a00640 1 -- 172.21.15.116:0/3571382331 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa908003970 con 0x7fa9101072f0 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa9179ed740 1 -- start start 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a1240 0x5616e66a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa9179ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616e6606370 con 0x5616e66a0850 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa9179ed740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616e65af550 con 0x5616e66a63b0 2024-03-04T07:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa9179ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616e6602410 con 0x5616e66a1240 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa90d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa90d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66a0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40354/0 (socket says 172.21.15.116:40354) 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.679+0000 7fa90d400640 1 -- 172.21.15.116:0/2874913405 learned_addr learned my addr 172.21.15.116:0/2874913405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.680+0000 7fa90de00640 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a1240 0x5616e66a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.680+0000 7fa90de00640 1 -- 172.21.15.116:0/2874913405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 msgr2=0x5616e66a8840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.680+0000 7fa90de00640 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.680+0000 7fa90de00640 1 -- 172.21.15.116:0/2874913405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 msgr2=0x5616e66a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.680+0000 7fa90de00640 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.680+0000 7fa90de00640 1 -- 172.21.15.116:0/2874913405 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616e65c33c0 con 0x5616e66a1240 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.680+0000 7fa90de00640 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a1240 0x5616e66a5d40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa900005ab0 tx=0x7fa9000095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dbbdda95abd884d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.681+0000 7fa906a00640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa8f40a1ea0 0x7fa8f40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.681+0000 7fa8fb400640 1 -- 172.21.15.116:0/2874913405 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900045070 con 0x5616e66a1240 2024-03-04T07:41:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.681+0000 7fa906a00640 1 -- 172.21.15.116:0/3571382331 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa9080ce180 con 0x7fa9101072f0 2024-03-04T07:41:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.681+0000 7fa8fb400640 1 -- 172.21.15.116:0/2874913405 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa900004040 con 0x5616e66a1240 2024-03-04T07:41:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.681+0000 7fa8fb400640 1 -- 172.21.15.116:0/2874913405 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900004ce0 con 0x5616e66a1240 2024-03-04T07:41:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.681+0000 7fa90d400640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa8f40a1ea0 0x7fa8f40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 -- 172.21.15.116:0/2874913405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a1240 msgr2=0x5616e66a5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a1240 0x5616e66a5d40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa900005ab0 tx=0x7fa9000095e0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 -- 172.21.15.116:0/2874913405 shutdown_connections 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a1240 0x5616e66a5d40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2874913405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 -- 172.21.15.116:0/2874913405 >> 172.21.15.116:0/2874913405 conn(0x5616e669bdc0 msgr2=0x5616e669c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 -- 172.21.15.116:0/2874913405 shutdown_connections 2024-03-04T07:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa90d400640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa8f40a1ea0 0x7fa8f40a4360 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x5616e66a1aa0 tx=0x7fa8fc009a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 -- 172.21.15.116:0/2874913405 wait complete. 2024-03-04T07:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.682+0000 7fa9179ed740 1 -- start start 2024-03-04T07:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 1 Processor -- start 2024-03-04T07:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66aab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a1240 0x5616e66ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a63b0 0x5616e66b2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616e66a9300 con 0x5616e66a0850 2024-03-04T07:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616e65c29b0 con 0x5616e66a1240 2024-03-04T07:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616e66a9480 con 0x5616e66a63b0 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.683+0000 7fa9179ed740 10 client.0 _ll_register_callbacks cb 0x5616e65fe340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a63b0 0x5616e66b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a1240 0x5616e66ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a63b0 0x5616e66b2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53784/0 (socket says 172.21.15.116:53784) 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 -- 172.21.15.116:0/764904709 learned_addr learned my addr 172.21.15.116:0/764904709 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90de00640 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66aab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 -- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a1240 msgr2=0x5616e66ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a1240 0x5616e66ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 -- 172.21.15.116:0/764904709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 msgr2=0x5616e66aab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66aab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90e800640 1 -- 172.21.15.116:0/764904709 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616e631b210 con 0x5616e66a63b0 2024-03-04T07:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.684+0000 7fa90d400640 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a1240 0x5616e66ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.685+0000 7fa90e800640 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a63b0 0x5616e66b2090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa90809ae40 tx=0x7fa90809b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cff463362b9018f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.685+0000 7fa8efe00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:41:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.685+0000 7fa8efe00640 1 -- 172.21.15.116:0/764904709 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9080b2020 con 0x5616e66a63b0 2024-03-04T07:41:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.685+0000 7fa8efe00640 1 -- 172.21.15.116:0/764904709 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa908004000 con 0x5616e66a63b0 2024-03-04T07:41:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.685+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa900004040 con 0x5616e66a63b0 2024-03-04T07:41:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.690+0000 7fa8efe00640 1 -- 172.21.15.116:0/764904709 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9080a3650 con 0x5616e66a63b0 2024-03-04T07:41:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.690+0000 7fa9179ed740 10 client.25381 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.690+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5616e63e7620 con 0x5616e66a63b0 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.690+0000 7fa8efe00640 1 -- 172.21.15.116:0/764904709 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa900004040 con 0x5616e66a63b0 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.690+0000 7fa8efe00640 1 -- 172.21.15.116:0/764904709 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa908024060 con 0x5616e66a63b0 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.690+0000 7fa9179ed740 10 client.25381 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.690+0000 7fa9179ed740 10 client.25381 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.691+0000 7fa9179ed740 20 client.25381 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.691+0000 7fa9179ed740 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e66b69f0 0x5616e66b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.691+0000 7fa9179ed740 4 client.25381 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.691+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616e66138b0 con 0x5616e66b69f0 2024-03-04T07:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.691+0000 7fa90de00640 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e66b69f0 0x5616e66b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.691+0000 7fa90de00640 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e66b69f0 0x5616e66b8de0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdb9a12ff5278b63 server_cookie=902648c7c251c138 in_seq=0 out_seq=0 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.691+0000 7fa8efe00640 10 client.25381 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.692+0000 7fa8efe00640 1 -- 172.21.15.116:0/764904709 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5616e66138b0 con 0x5616e66b69f0 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.692+0000 7fa8efe00640 10 client.25381 handle_command_reply: tid=0 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.692+0000 7fa9179ed740 1 client.25381 shutdown 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a63b0 msgr2=0x5616e66b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a63b0 0x5616e66b2090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa90809ae40 tx=0x7fa90809b670 comp rx=0 tx=0).stop 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 shutdown_connections 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e66b69f0 0x5616e66b8de0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a63b0 0x5616e66b2090 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a1240 0x5616e66ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 --2- 172.21.15.116:0/764904709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a0850 0x5616e66aab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa8efe00640 0 client.25381 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 >> 172.21.15.116:0/764904709 conn(0x5616e669bdc0 msgr2=0x5616e669afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 shutdown_connections 2024-03-04T07:41:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 1 -- 172.21.15.116:0/764904709 wait complete. 2024-03-04T07:41:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.693+0000 7fa9179ed740 20 client.25381 trim_cache size 0 max 16384 2024-03-04T07:41:24.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 Processor -- start 2024-03-04T07:41:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 -- start start 2024-03-04T07:41:24.721 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66b2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101a8aa0 con 0x5616e66a1240 2024-03-04T07:41:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101ab080 con 0x5616e66a63b0 2024-03-04T07:41:24.722 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.720+0000 7fa9179ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101a90a0 con 0x5616e66a0850 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66b2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40366/0 (socket says 172.21.15.116:40366) 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90d400640 1 -- 172.21.15.116:0/2660190833 learned_addr learned my addr 172.21.15.116:0/2660190833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90de00640 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90e800640 1 -- 172.21.15.116:0/2660190833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 msgr2=0x5616e66a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90e800640 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66a25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90e800640 1 -- 172.21.15.116:0/2660190833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 msgr2=0x5616e66b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90e800640 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66b2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90e800640 1 -- 172.21.15.116:0/2660190833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616e669cb30 con 0x5616e66a63b0 2024-03-04T07:41:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90de00640 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:41:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.721+0000 7fa90e800640 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a2ae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa90809b070 tx=0x7fa90809f180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1631543ba8958ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.722+0000 7fa8faa00640 1 -- 172.21.15.116:0/2660190833 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908014070 con 0x5616e66a63b0 2024-03-04T07:41:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.722+0000 7fa8faa00640 1 -- 172.21.15.116:0/2660190833 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa908003c60 con 0x5616e66a63b0 2024-03-04T07:41:24.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.722+0000 7fa8faa00640 1 -- 172.21.15.116:0/2660190833 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9080b2050 con 0x5616e66a63b0 2024-03-04T07:41:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.722+0000 7fa9179ed740 1 -- 172.21.15.116:0/2660190833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 msgr2=0x5616e66a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.727 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a2ae0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa90809b070 tx=0x7fa90809f180 comp rx=0 tx=0).stop 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 -- 172.21.15.116:0/2660190833 shutdown_connections 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66a2ae0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 --2- 172.21.15.116:0/2660190833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 -- 172.21.15.116:0/2660190833 >> 172.21.15.116:0/2660190833 conn(0x5616e669bdc0 msgr2=0x5616e669c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 -- 172.21.15.116:0/2660190833 shutdown_connections 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 -- 172.21.15.116:0/2660190833 wait complete. 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 -- start start 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.723+0000 7fa9179ed740 1 Processor -- start 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66af920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66afe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa9179ed740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa9179ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101a9700 con 0x5616e66a1240 2024-03-04T07:41:24.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa9179ed740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101aaa20 con 0x5616e66a63b0 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa9179ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101a9d60 con 0x5616e66a0850 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa9179ed740 10 client.0 _ll_register_callbacks cb 0x7fa908096a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa90de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa90de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53800/0 (socket says 172.21.15.116:53800) 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa90de00640 1 -- 172.21.15.116:0/434699797 learned_addr learned my addr 172.21.15.116:0/434699797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa90d400640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.724+0000 7fa90e800640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa90de00640 1 -- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 msgr2=0x5616e66b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa90de00640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa90de00640 1 -- 172.21.15.116:0/434699797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 msgr2=0x5616e66afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa90de00640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa90de00640 1 -- 172.21.15.116:0/434699797 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616e66b1410 con 0x5616e66a0850 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa90d400640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa90de00640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66af920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa90002c170 tx=0x7fa900005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96b300c1294e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.725+0000 7fa8efe00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.726+0000 7fa8efe00640 1 -- 172.21.15.116:0/434699797 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900045040 con 0x5616e66a0850 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.726+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa908003c60 con 0x5616e66a0850 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.726+0000 7fa8efe00640 1 -- 172.21.15.116:0/434699797 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa90002cb90 con 0x5616e66a0850 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.726+0000 7fa8efe00640 1 -- 172.21.15.116:0/434699797 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9000046c0 con 0x5616e66a0850 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.726+0000 7fa9179ed740 10 client.25384 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.726+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5616e6712400 con 0x5616e66a0850 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.726+0000 7fa8efe00640 1 -- 172.21.15.116:0/434699797 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa908003c60 con 0x5616e66a0850 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa8efe00640 1 -- 172.21.15.116:0/434699797 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa900005440 con 0x5616e66a0850 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa9179ed740 10 client.25384 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa9179ed740 10 client.25384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa9179ed740 20 client.25384 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa9179ed740 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e6712990 0x5616e6714db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa9179ed740 4 client.25384 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:41:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5616e660dc00 con 0x5616e6712990 2024-03-04T07:41:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa90d400640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e6712990 0x5616e6714db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa90d400640 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e6712990 0x5616e6714db0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cda3bc33baa624cf server_cookie=2fc07e7b4e5e767b in_seq=0 out_seq=0 2024-03-04T07:41:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.727+0000 7fa8efe00640 10 client.25384 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa8efe00640 1 -- 172.21.15.116:0/434699797 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5616e660dc00 con 0x5616e6712990 2024-03-04T07:41:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa8efe00640 10 client.25384 handle_command_reply: tid=0 2024-03-04T07:41:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 client.25384 shutdown 2024-03-04T07:41:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 msgr2=0x5616e66af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66af920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa90002c170 tx=0x7fa900005e50 comp rx=0 tx=0).stop 2024-03-04T07:41:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 shutdown_connections 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5616e6712990 0x5616e6714db0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5616e66a63b0 0x5616e66b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616e66a1240 0x5616e66afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 --2- 172.21.15.116:0/434699797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616e66a0850 0x5616e66af920 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa8efe00640 0 client.25384 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 >> 172.21.15.116:0/434699797 conn(0x5616e669bdc0 msgr2=0x5616e66aab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 shutdown_connections 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 1 -- 172.21.15.116:0/434699797 wait complete. 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.731+0000 7fa9179ed740 20 client.25384 trim_cache size 0 max 16384 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:41:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:41:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:41:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:41:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.732+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa8f40a1ea0 msgr2=0x7fa8f40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.732+0000 7fa914e00640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa8f40a1ea0 0x7fa8f40a4360 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x5616e66a1aa0 tx=0x7fa8fc009a30 comp rx=0 tx=0).stop 2024-03-04T07:41:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.732+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9101072f0 msgr2=0x7fa910100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.732+0000 7fa914e00640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9101072f0 0x7fa910100580 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa90800da10 tx=0x7fa90800ded0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 shutdown_connections 2024-03-04T07:41:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9101072f0 0x7fa910100580 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9101049c0 0x7fa910100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa8f40a1ea0 0x7fa8f40a4360 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 --2- 172.21.15.116:0/3571382331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910102090 0x7fa910101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 >> 172.21.15.116:0/3571382331 conn(0x7fa9100fba80 msgr2=0x7fa9101029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 shutdown_connections 2024-03-04T07:41:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:24.733+0000 7fa914e00640 1 -- 172.21.15.116:0/3571382331 wait complete. 2024-03-04T07:41:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:24 smithi116 ceph-mon[32150]: pgmap v420: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-04T07:41:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:24 smithi152 ceph-mon[37154]: pgmap v420: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-04T07:41:25.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:24 smithi133 ceph-mon[36505]: pgmap v420: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-04T07:41:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:26 smithi116 ceph-mon[32150]: pgmap v421: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 341 B/s wr, 2 op/s 2024-03-04T07:41:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:26 smithi152 ceph-mon[37154]: pgmap v421: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 341 B/s wr, 2 op/s 2024-03-04T07:41:27.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:26 smithi133 ceph-mon[36505]: pgmap v421: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 341 B/s wr, 2 op/s 2024-03-04T07:41:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:28 smithi116 ceph-mon[32150]: pgmap v422: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-04T07:41:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:28 smithi152 ceph-mon[37154]: pgmap v422: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-04T07:41:29.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:28 smithi133 ceph-mon[36505]: pgmap v422: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-04T07:41:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:30 smithi116 ceph-mon[32150]: pgmap v423: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:30 smithi152 ceph-mon[37154]: pgmap v423: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:30 smithi133 ceph-mon[36505]: pgmap v423: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:32 smithi116 ceph-mon[32150]: pgmap v424: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:33.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:32 smithi152 ceph-mon[37154]: pgmap v424: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:33.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:32 smithi133 ceph-mon[36505]: pgmap v424: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:34.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:33 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:41:34.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:33 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:41:34.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:33 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:41:35.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:41:35.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:41:35.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:34 smithi133 ceph-mon[36505]: pgmap v425: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:41:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:41:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:34 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.109 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:34 smithi116 ceph-mon[32150]: pgmap v425: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:35.109 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.109 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.109 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:41:35.109 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:41:35.109 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:34 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:34 smithi152 ceph-mon[37154]: pgmap v425: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:41:35.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:41:35.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:34 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:41:35.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 -- 172.21.15.116:0/2687877502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c30103080 msgr2=0x7f1c30109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 --2- 172.21.15.116:0/2687877502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c30103080 0x7f1c30109910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009ca0 tx=0x7f1c2802c100 comp rx=0 tx=0).stop 2024-03-04T07:41:35.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 -- 172.21.15.116:0/2687877502 shutdown_connections 2024-03-04T07:41:35.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 --2- 172.21.15.116:0/2687877502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c30103080 0x7f1c30109910 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 --2- 172.21.15.116:0/2687877502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c301026c0 0x7f1c30102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 --2- 172.21.15.116:0/2687877502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c301014d0 0x7f1c301018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 -- 172.21.15.116:0/2687877502 >> 172.21.15.116:0/2687877502 conn(0x7f1c300fca40 msgr2=0x7f1c300fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 -- 172.21.15.116:0/2687877502 shutdown_connections 2024-03-04T07:41:35.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.296+0000 7f1c35600640 1 -- 172.21.15.116:0/2687877502 wait complete. 2024-03-04T07:41:35.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.298+0000 7f1c35600640 1 Processor -- start 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.298+0000 7f1c35600640 1 -- start start 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.298+0000 7f1c35600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c301014d0 0x7f1c3019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.298+0000 7f1c35600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c301026c0 0x7f1c3019e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.298+0000 7f1c35600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c30103080 0x7f1c301a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c35600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3010af70 con 0x7f1c301026c0 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c35600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3010adf0 con 0x7f1c301014d0 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c35600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3010b0f0 con 0x7f1c30103080 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c301014d0 0x7f1c3019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c34c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c30103080 0x7f1c301a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c34c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c30103080 0x7f1c301a5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:34144/0 (socket says 172.21.15.116:34144) 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c301014d0 0x7f1c3019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37326/0 (socket says 172.21.15.116:37326) 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c34c00640 1 -- 172.21.15.116:0/3404644711 learned_addr learned my addr 172.21.15.116:0/3404644711 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2f400640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c301026c0 0x7f1c3019e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2fe00640 1 -- 172.21.15.116:0/3404644711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c30103080 msgr2=0x7f1c301a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c30103080 0x7f1c301a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2fe00640 1 -- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c301026c0 msgr2=0x7f1c3019e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c301026c0 0x7f1c3019e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2fe00640 1 -- 172.21.15.116:0/3404644711 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c301952c0 con 0x7f1c301014d0 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.299+0000 7f1c2f400640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c301026c0 0x7f1c3019e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.300+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c301014d0 0x7f1c3019dda0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e9c0 tx=0x7f1c2000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.300+0000 7f1c2cc00640 1 -- 172.21.15.116:0/3404644711 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000ccc0 con 0x7f1c301014d0 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.300+0000 7f1c2cc00640 1 -- 172.21.15.116:0/3404644711 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c200035f0 con 0x7f1c301014d0 2024-03-04T07:41:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.300+0000 7f1c2cc00640 1 -- 172.21.15.116:0/3404644711 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000ce60 con 0x7f1c301014d0 2024-03-04T07:41:35.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.301+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c301955b0 con 0x7f1c301014d0 2024-03-04T07:41:35.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.301+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c30195a80 con 0x7f1c301014d0 2024-03-04T07:41:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.306+0000 7f1c399bb740 1 Processor -- start 2024-03-04T07:41:35.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.306+0000 7f1c399bb740 1 -- start start 2024-03-04T07:41:35.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.306+0000 7f1c2cc00640 1 -- 172.21.15.116:0/3404644711 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1c200037a0 con 0x7f1c301014d0 2024-03-04T07:41:35.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.306+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b820ff8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.306+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b820ffd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b820fffd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c399bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b820f25090 con 0x55b820ff8700 2024-03-04T07:41:35.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c399bb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b820f5cd30 con 0x55b820ffd870 2024-03-04T07:41:35.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c399bb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b820f05a30 con 0x55b820ff7d10 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b820fffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b820fffd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37336/0 (socket says 172.21.15.116:37336) 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 -- 172.21.15.116:0/1889592145 learned_addr learned my addr 172.21.15.116:0/1889592145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b820ffd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c34c00640 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b820ff8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 -- 172.21.15.116:0/1889592145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 msgr2=0x55b820ff8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b820ff8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 -- 172.21.15.116:0/1889592145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 msgr2=0x55b820ffd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b820ffd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.307+0000 7f1c2f400640 1 -- 172.21.15.116:0/1889592145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b820d3b7d0 con 0x55b820ffd870 2024-03-04T07:41:35.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.308+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b820ffd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:35.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.308+0000 7f1c2f400640 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b820fffd00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009c60 tx=0x7f1c1c010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e3d31360de18b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.308+0000 7f1c25200640 1 -- 172.21.15.116:0/1889592145 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c005020 con 0x55b820ffd870 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.308+0000 7f1c25200640 1 -- 172.21.15.116:0/1889592145 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c1c0047a0 con 0x55b820ffd870 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.308+0000 7f1c25200640 1 -- 172.21.15.116:0/1889592145 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c012720 con 0x55b820ffd870 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.308+0000 7f1c2cc00640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c140a1df0 0x7f1c140a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.309+0000 7f1c2cc00640 1 -- 172.21.15.116:0/3404644711 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1c200ce4f0 con 0x7f1c301014d0 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.309+0000 7f1c399bb740 1 -- 172.21.15.116:0/1889592145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 msgr2=0x55b820fffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.309+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b820fffd00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009c60 tx=0x7f1c1c010740 comp rx=0 tx=0).stop 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.309+0000 7f1c34c00640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c140a1df0 0x7f1c140a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.310+0000 7f1c399bb740 1 -- 172.21.15.116:0/1889592145 shutdown_connections 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.310+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b820fffd00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.310+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b820ffd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.310+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1889592145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b820ff8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.310+0000 7f1c399bb740 1 -- 172.21.15.116:0/1889592145 >> 172.21.15.116:0/1889592145 conn(0x55b820ff3280 msgr2=0x55b820ff3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.310+0000 7f1c399bb740 1 -- 172.21.15.116:0/1889592145 shutdown_connections 2024-03-04T07:41:35.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.310+0000 7f1c399bb740 1 -- 172.21.15.116:0/1889592145 wait complete. 2024-03-04T07:41:35.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.311+0000 7f1c34c00640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c140a1df0 0x7f1c140a42b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55b820ff8f60 tx=0x7f1c2802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.311+0000 7f1c399bb740 1 -- start start 2024-03-04T07:41:35.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.311+0000 7f1c399bb740 1 Processor -- start 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ffd870 0x55b821009460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c399bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b820efb0c0 con 0x55b820ffd870 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c399bb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b820f58dd0 con 0x55b820ff7d10 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c399bb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821000820 con 0x55b820ff8700 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c399bb740 10 client.0 _ll_register_callbacks cb 0x55b820f18860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c2fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c2f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c2f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:34158/0 (socket says 172.21.15.116:34158) 2024-03-04T07:41:35.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c2f400640 1 -- 172.21.15.116:0/2291247963 learned_addr learned my addr 172.21.15.116:0/2291247963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.312+0000 7f1c34c00640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ffd870 0x55b821009460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c2f400640 1 -- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ff7d10 msgr2=0x55b821001f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c2f400640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c2f400640 1 -- 172.21.15.116:0/2291247963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ffd870 msgr2=0x55b821009460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c2f400640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ffd870 0x55b821009460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c2f400640 1 -- 172.21.15.116:0/2291247963 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b820c713f0 con 0x55b820ff8700 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c34c00640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ffd870 0x55b821009460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c2f400640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821002490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005bc0 tx=0x7f1c1c0051b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dbbb569a6180ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.313+0000 7f1c19600640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:41:35.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.314+0000 7f1c19600640 1 -- 172.21.15.116:0/2291247963 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c004020 con 0x55b820ff8700 2024-03-04T07:41:35.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.314+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c1c0047a0 con 0x55b820ff8700 2024-03-04T07:41:35.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.314+0000 7f1c19600640 1 -- 172.21.15.116:0/2291247963 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1c1c0041c0 con 0x55b820ff8700 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.314+0000 7f1c19600640 1 -- 172.21.15.116:0/2291247963 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c1c0043c0 con 0x55b820ff8700 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.314+0000 7f1c399bb740 10 client.25396 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.314+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b820c310f0 con 0x55b820ff8700 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.314+0000 7f1c19600640 1 -- 172.21.15.116:0/2291247963 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c1c0047a0 con 0x55b820ff8700 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c19600640 1 -- 172.21.15.116:0/2291247963 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1c1c02d020 con 0x55b820ff8700 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c399bb740 10 client.25396 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c399bb740 10 client.25396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c399bb740 20 client.25396 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c399bb740 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b82100d760 0x55b82100fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c399bb740 4 client.25396 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b820f6a270 con 0x55b82100d760 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.315+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b82100d760 0x55b82100fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.316+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b82100d760 0x55b82100fb50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8b04d6a9a96aaeb server_cookie=8e3b4211eff2aca4 in_seq=0 out_seq=0 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.316+0000 7f1c19600640 10 client.25396 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.316+0000 7f1c19600640 1 -- 172.21.15.116:0/2291247963 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b820f6a270 con 0x55b82100d760 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.316+0000 7f1c19600640 10 client.25396 handle_command_reply: tid=0 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.316+0000 7f1c399bb740 1 client.25396 shutdown 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 msgr2=0x55b821002490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821002490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005bc0 tx=0x7f1c1c0051b0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 shutdown_connections 2024-03-04T07:41:35.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b82100d760 0x55b82100fb50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ffd870 0x55b821009460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 --2- 172.21.15.116:0/2291247963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 >> 172.21.15.116:0/2291247963 conn(0x55b820ff3280 msgr2=0x55b820ff24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c19600640 0 client.25396 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 shutdown_connections 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 1 -- 172.21.15.116:0/2291247963 wait complete. 2024-03-04T07:41:35.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.317+0000 7f1c399bb740 20 client.25396 trim_cache size 0 max 16384 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.350+0000 7f1c399bb740 1 Processor -- start 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c399bb740 1 -- start start 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821009420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c399bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3019c700 con 0x55b820ff8700 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c399bb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c301b7bd0 con 0x55b820ffd870 2024-03-04T07:41:35.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c399bb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3019cd00 con 0x55b820ff7d10 2024-03-04T07:41:35.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c2f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c34c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821009420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c2f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42882/0 (socket says 172.21.15.116:42882) 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c34c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821009420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37354/0 (socket says 172.21.15.116:37354) 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.351+0000 7f1c2f400640 1 -- 172.21.15.116:0/4027537735 learned_addr learned my addr 172.21.15.116:0/4027537735 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.352+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.352+0000 7f1c34c00640 1 -- 172.21.15.116:0/4027537735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 msgr2=0x55b821001f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.352+0000 7f1c34c00640 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.352+0000 7f1c34c00640 1 -- 172.21.15.116:0/4027537735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 msgr2=0x55b821002490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.352+0000 7f1c34c00640 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.352+0000 7f1c34c00640 1 -- 172.21.15.116:0/4027537735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b820ff94d0 con 0x55b820ffd870 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.352+0000 7f1c34c00640 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821009420 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006010 tx=0x7f1c28040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f9de6d3d0c48d8c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.353+0000 7f1c1be00640 1 -- 172.21.15.116:0/4027537735 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c28039040 con 0x55b820ffd870 2024-03-04T07:41:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.353+0000 7f1c1be00640 1 -- 172.21.15.116:0/4027537735 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c28052070 con 0x55b820ffd870 2024-03-04T07:41:35.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.353+0000 7f1c1be00640 1 -- 172.21.15.116:0/4027537735 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c280406c0 con 0x55b820ffd870 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 -- 172.21.15.116:0/4027537735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 msgr2=0x55b821009420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821009420 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006010 tx=0x7f1c28040000 comp rx=0 tx=0).stop 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 -- 172.21.15.116:0/4027537735 shutdown_connections 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821009420 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff8700 0x55b821002490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 --2- 172.21.15.116:0/4027537735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff7d10 0x55b821001f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 -- 172.21.15.116:0/4027537735 >> 172.21.15.116:0/4027537735 conn(0x55b820ff3280 msgr2=0x55b821001570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 -- 172.21.15.116:0/4027537735 shutdown_connections 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 -- 172.21.15.116:0/4027537735 wait complete. 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.354+0000 7f1c399bb740 1 -- start start 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 1 Processor -- start 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff7d10 0x55b821006c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821007140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821008150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3019d360 con 0x55b820ff7d10 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c301b7570 con 0x55b820ffd870 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c301b68e0 con 0x55b820ff8700 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.355+0000 7f1c399bb740 10 client.0 _ll_register_callbacks cb 0x7f1c1c004c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821008150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821008150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37356/0 (socket says 172.21.15.116:37356) 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 -- 172.21.15.116:0/1375492035 learned_addr learned my addr 172.21.15.116:0/1375492035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff7d10 0x55b821006c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c2f400640 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 -- 172.21.15.116:0/1375492035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 msgr2=0x55b821007140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821007140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 -- 172.21.15.116:0/1375492035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff7d10 msgr2=0x55b821006c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff7d10 0x55b821006c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 -- 172.21.15.116:0/1375492035 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b821008740 con 0x55b820ffd870 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c34c00640 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821008150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1c2802e010 tx=0x7f1c28009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63431c1f08cbac06 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.356+0000 7f1c19600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.357+0000 7f1c19600640 1 -- 172.21.15.116:0/1375492035 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2805c070 con 0x55b820ffd870 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.357+0000 7f1c19600640 1 -- 172.21.15.116:0/1375492035 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1c28003db0 con 0x55b820ffd870 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.357+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c28052070 con 0x55b820ffd870 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.357+0000 7f1c19600640 1 -- 172.21.15.116:0/1375492035 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c28066050 con 0x55b820ffd870 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 10 client.25427 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b821068fe0 con 0x55b820ffd870 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c19600640 1 -- 172.21.15.116:0/1375492035 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c28052070 con 0x55b820ffd870 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c19600640 1 -- 172.21.15.116:0/1375492035 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1c28040a10 con 0x55b820ffd870 2024-03-04T07:41:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 10 client.25427 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 10 client.25427 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 20 client.25427 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b8210698f0 0x55b82106bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 4 client.25427 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:41:35.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.358+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b820f645c0 con 0x55b8210698f0 2024-03-04T07:41:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.360+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b8210698f0 0x55b82106bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c2fe00640 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b8210698f0 0x55b82106bd90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b46454a37d974e server_cookie=b2c423420abfbba3 in_seq=0 out_seq=0 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c19600640 10 client.25427 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c19600640 1 -- 172.21.15.116:0/1375492035 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b820f645c0 con 0x55b8210698f0 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c19600640 10 client.25427 handle_command_reply: tid=0 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c399bb740 1 client.25427 shutdown 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 msgr2=0x55b821008150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821008150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1c2802e010 tx=0x7f1c28009880 comp rx=0 tx=0).stop 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 shutdown_connections 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.362+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b8210698f0 0x55b82106bd90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b820ffd870 0x55b821008150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b820ff8700 0x55b821007140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c399bb740 1 --2- 172.21.15.116:0/1375492035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b820ff7d10 0x55b821006c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 >> 172.21.15.116:0/1375492035 conn(0x55b820ff3280 msgr2=0x55b820ff64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c19600640 0 client.25427 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 shutdown_connections 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c399bb740 1 -- 172.21.15.116:0/1375492035 wait complete. 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.363+0000 7f1c399bb740 20 client.25427 trim_cache size 0 max 16384 2024-03-04T07:41:35.365 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:41:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:41:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:41:35.366 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:41:35.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c140a1df0 msgr2=0x7f1c140a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c140a1df0 0x7f1c140a42b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55b820ff8f60 tx=0x7f1c2802e040 comp rx=0 tx=0).stop 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c301014d0 msgr2=0x7f1c3019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c301014d0 0x7f1c3019dda0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e9c0 tx=0x7f1c2000ee80 comp rx=0 tx=0).stop 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 shutdown_connections 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c140a1df0 0x7f1c140a42b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c30103080 0x7f1c301a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c301026c0 0x7f1c3019e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 --2- 172.21.15.116:0/3404644711 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c301014d0 0x7f1c3019dda0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 >> 172.21.15.116:0/3404644711 conn(0x7f1c300fca40 msgr2=0x7f1c301059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 shutdown_connections 2024-03-04T07:41:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.364+0000 7f1c35600640 1 -- 172.21.15.116:0/3404644711 wait complete. 2024-03-04T07:41:35.389 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:41:35.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:41:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 -- 172.21.15.116:0/3003788071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f4075590 msgr2=0x7f88f40739e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 --2- 172.21.15.116:0/3003788071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f4075590 0x7f88f40739e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009900 tx=0x7f88e802bcd0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 -- 172.21.15.116:0/3003788071 shutdown_connections 2024-03-04T07:41:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 --2- 172.21.15.116:0/3003788071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88f41092e0 0x7f88f410b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 --2- 172.21.15.116:0/3003788071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 0x7f88f4074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 --2- 172.21.15.116:0/3003788071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f4075590 0x7f88f40739e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 -- 172.21.15.116:0/3003788071 >> 172.21.15.116:0/3003788071 conn(0x7f88f40fc9a0 msgr2=0x7f88f40fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 -- 172.21.15.116:0/3003788071 shutdown_connections 2024-03-04T07:41:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.660+0000 7f88f9e00640 1 -- 172.21.15.116:0/3003788071 wait complete. 2024-03-04T07:41:35.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.661+0000 7f88f9e00640 1 Processor -- start 2024-03-04T07:41:35.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.661+0000 7f88f9e00640 1 -- start start 2024-03-04T07:41:35.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.661+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 0x7f88f4199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.661+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88f4075590 0x7f88f4199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f41092e0 0x7f88f41a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f9e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f410f310 con 0x7f88f4073fb0 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f9e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f410f190 con 0x7f88f41092e0 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f9e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f410f490 con 0x7f88f4075590 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 0x7f88f4199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f3e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88f4075590 0x7f88f4199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f41092e0 0x7f88f41a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 0x7f88f4199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42908/0 (socket says 172.21.15.116:42908) 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 -- 172.21.15.116:0/2370536737 learned_addr learned my addr 172.21.15.116:0/2370536737 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 -- 172.21.15.116:0/2370536737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88f4075590 msgr2=0x7f88f4199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88f4075590 0x7f88f4199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 -- 172.21.15.116:0/2370536737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f41092e0 msgr2=0x7f88f41a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f41092e0 0x7f88f41a0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.662+0000 7f88f8a00640 1 -- 172.21.15.116:0/2370536737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f41a1540 con 0x7f88f4073fb0 2024-03-04T07:41:35.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.663+0000 7f88f8a00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 0x7f88f4199920 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f88e8000c00 tx=0x7f88e8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.663+0000 7f88f1600640 1 -- 172.21.15.116:0/2370536737 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8043020 con 0x7f88f4073fb0 2024-03-04T07:41:35.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.663+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f41a17d0 con 0x7f88f4073fb0 2024-03-04T07:41:35.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.663+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f41a1d70 con 0x7f88f4073fb0 2024-03-04T07:41:35.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.663+0000 7f88f1600640 1 -- 172.21.15.116:0/2370536737 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88e80054e0 con 0x7f88f4073fb0 2024-03-04T07:41:35.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.663+0000 7f88f1600640 1 -- 172.21.15.116:0/2370536737 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8003cd0 con 0x7f88f4073fb0 2024-03-04T07:41:35.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.666+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f41080a0 con 0x7f88f4073fb0 2024-03-04T07:41:35.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.667+0000 7f88f1600640 1 -- 172.21.15.116:0/2370536737 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f88e8046020 con 0x7f88f4073fb0 2024-03-04T07:41:35.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.668+0000 7f88f1600640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f88dc0a1d80 0x7f88dc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:35.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.669+0000 7f88f1600640 1 -- 172.21.15.116:0/2370536737 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f88e80ef930 con 0x7f88f4073fb0 2024-03-04T07:41:35.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.669+0000 7f88f3e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f88dc0a1d80 0x7f88dc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:35.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.670+0000 7f88f3e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f88dc0a1d80 0x7f88dc0a4240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88f419ae30 tx=0x7f88e000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:35.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.678+0000 7f88f1600640 1 -- 172.21.15.116:0/2370536737 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f88e804d030 con 0x7f88f4073fb0 2024-03-04T07:41:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.910+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88f40621d0 con 0x7f88f4073fb0 2024-03-04T07:41:35.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.910+0000 7f88f1600640 1 -- 172.21.15.116:0/2370536737 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f88e80bcd60 con 0x7f88f4073fb0 2024-03-04T07:41:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:41:35.912 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:41:35.913 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:41:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f88dc0a1d80 msgr2=0x7f88dc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f88dc0a1d80 0x7f88dc0a4240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88f419ae30 tx=0x7f88e000b390 comp rx=0 tx=0).stop 2024-03-04T07:41:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 msgr2=0x7f88f4199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 0x7f88f4199920 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f88e8000c00 tx=0x7f88e8005dd0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 shutdown_connections 2024-03-04T07:41:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f88f41092e0 0x7f88f41a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f88dc0a1d80 0x7f88dc0a4240 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88f4075590 0x7f88f4199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 --2- 172.21.15.116:0/2370536737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4073fb0 0x7f88f4199920 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 >> 172.21.15.116:0/2370536737 conn(0x7f88f40fc9a0 msgr2=0x7f88f4105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 shutdown_connections 2024-03-04T07:41:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:35.913+0000 7f88f9e00640 1 -- 172.21.15.116:0/2370536737 wait complete. 2024-03-04T07:41:35.932 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:41:35.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:41:36.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 -- 172.21.15.116:0/2669204371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c104860 msgr2=0x7f978c106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 --2- 172.21.15.116:0/2669204371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c104860 0x7f978c106c50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9778009900 tx=0x7f977802bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:36.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 -- 172.21.15.116:0/2669204371 shutdown_connections 2024-03-04T07:41:36.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 --2- 172.21.15.116:0/2669204371 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f978c107190 0x7f978c1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 --2- 172.21.15.116:0/2669204371 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c104860 0x7f978c106c50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 --2- 172.21.15.116:0/2669204371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978c101f30 0x7f978c104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 -- 172.21.15.116:0/2669204371 >> 172.21.15.116:0/2669204371 conn(0x7f978c0fb940 msgr2=0x7f978c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:36.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 -- 172.21.15.116:0/2669204371 shutdown_connections 2024-03-04T07:41:36.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.120+0000 7f9792c00640 1 -- 172.21.15.116:0/2669204371 wait complete. 2024-03-04T07:41:36.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 Processor -- start 2024-03-04T07:41:36.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 -- start start 2024-03-04T07:41:36.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c101f30 0x7f978c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978c104860 0x7f978c0ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f978c107190 0x7f978c100420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c10d140 con 0x7f978c104860 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c10cfc0 con 0x7f978c101f30 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.121+0000 7f9792c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c10d2c0 con 0x7f978c107190 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c101f30 0x7f978c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c101f30 0x7f978c1018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44292/0 (socket says 172.21.15.116:44292) 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 -- 172.21.15.116:0/3145223346 learned_addr learned my addr 172.21.15.116:0/3145223346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9790e00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978c104860 0x7f978c0ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9792200640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f978c107190 0x7f978c100420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 -- 172.21.15.116:0/3145223346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f978c107190 msgr2=0x7f978c100420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f978c107190 0x7f978c100420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 -- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978c104860 msgr2=0x7f978c0ffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978c104860 0x7f978c0ffee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9791800640 1 -- 172.21.15.116:0/3145223346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c100cb0 con 0x7f978c101f30 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.122+0000 7f9792200640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f978c107190 0x7f978c100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.123+0000 7f9791800640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c101f30 0x7f978c1018a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f978000e9f0 tx=0x7f978000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.123+0000 7f978a000640 1 -- 172.21.15.116:0/3145223346 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978000cd80 con 0x7f978c101f30 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.123+0000 7f978a000640 1 -- 172.21.15.116:0/3145223346 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9780010910 con 0x7f978c101f30 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.123+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c1a18e0 con 0x7f978c101f30 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.123+0000 7f978a000640 1 -- 172.21.15.116:0/3145223346 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780018990 con 0x7f978c101f30 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.123+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c1a1e50 con 0x7f978c101f30 2024-03-04T07:41:36.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.126+0000 7f978a000640 1 -- 172.21.15.116:0/3145223346 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9780018c30 con 0x7f978c101f30 2024-03-04T07:41:36.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.127+0000 7f978a000640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97740a1e00 0x7f97740a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.127+0000 7f978a000640 1 -- 172.21.15.116:0/3145223346 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f97800ce130 con 0x7f978c101f30 2024-03-04T07:41:36.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 Processor -- start 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 -- start start 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c3caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c41b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c421e0 0x55e380c44670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e380b98d60 con 0x55e380c3c680 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e380b41cf0 con 0x55e380c421e0 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9796fd8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e380b94e00 con 0x55e380c3d070 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.128+0000 7f9790e00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97740a1e00 0x7f97740a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9791800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c3caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9791800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c3caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38984/0 (socket says 172.21.15.116:38984) 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9791800640 1 -- 172.21.15.116:0/2729988350 learned_addr learned my addr 172.21.15.116:0/2729988350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9792200640 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c421e0 0x55e380c44670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9790e00640 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c41b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9792200640 1 -- 172.21.15.116:0/2729988350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 msgr2=0x55e380c41b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9792200640 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c41b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9792200640 1 -- 172.21.15.116:0/2729988350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 msgr2=0x55e380c3caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9792200640 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c3caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9792200640 1 -- 172.21.15.116:0/2729988350 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3808ad3f0 con 0x55e380c421e0 2024-03-04T07:41:36.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.129+0000 7f9790e00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97740a1e00 0x7f97740a42c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9778000c00 tx=0x7f9778005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:36.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.130+0000 7f9792200640 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c421e0 0x55e380c44670 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9784009f90 tx=0x7f9784010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5597e996fa73290b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:36.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.130+0000 7f977ea00640 1 -- 172.21.15.116:0/2729988350 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784011700 con 0x55e380c421e0 2024-03-04T07:41:36.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.130+0000 7f977ea00640 1 -- 172.21.15.116:0/2729988350 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9784011d80 con 0x55e380c421e0 2024-03-04T07:41:36.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.130+0000 7f977ea00640 1 -- 172.21.15.116:0/2729988350 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978401a380 con 0x55e380c421e0 2024-03-04T07:41:36.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 -- 172.21.15.116:0/2729988350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c421e0 msgr2=0x55e380c44670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c421e0 0x55e380c44670 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9784009f90 tx=0x7f9784010860 comp rx=0 tx=0).stop 2024-03-04T07:41:36.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 -- 172.21.15.116:0/2729988350 shutdown_connections 2024-03-04T07:41:36.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c421e0 0x55e380c44670 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c41b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2729988350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c3caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 -- 172.21.15.116:0/2729988350 >> 172.21.15.116:0/2729988350 conn(0x55e380c37bf0 msgr2=0x55e380c37ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 -- 172.21.15.116:0/2729988350 shutdown_connections 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.131+0000 7f9796fd8740 1 -- 172.21.15.116:0/2729988350 wait complete. 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.132+0000 7f9796fd8740 1 -- start start 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.132+0000 7f9796fd8740 1 Processor -- start 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c469c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3d070 0x55e380c46f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9796fd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e380c45190 con 0x55e380c3c680 2024-03-04T07:41:36.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9796fd8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e380b37220 con 0x55e380c3d070 2024-03-04T07:41:36.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9796fd8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e380c45310 con 0x55e380c421e0 2024-03-04T07:41:36.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9796fd8740 10 client.0 _ll_register_callbacks cb 0x55e3808acbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:36.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9790e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3d070 0x55e380c46f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9790e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3d070 0x55e380c46f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44306/0 (socket says 172.21.15.116:44306) 2024-03-04T07:41:36.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9790e00640 1 -- 172.21.15.116:0/1433521487 learned_addr learned my addr 172.21.15.116:0/1433521487 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:36.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9791800640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.133+0000 7f9792200640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9790e00640 1 -- 172.21.15.116:0/1433521487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 msgr2=0x55e380c4df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9790e00640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9790e00640 1 -- 172.21.15.116:0/1433521487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 msgr2=0x55e380c469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9790e00640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9790e00640 1 -- 172.21.15.116:0/1433521487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3809777d0 con 0x55e380c3d070 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9791800640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9790e00640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3d070 0x55e380c46f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f977802c7b0 tx=0x7f977802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1422f6bbb65ddca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9773400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9773400640 1 -- 172.21.15.116:0/1433521487 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977804e070 con 0x55e380c3d070 2024-03-04T07:41:36.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9773400640 1 -- 172.21.15.116:0/1433521487 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9778005510 con 0x55e380c3d070 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.134+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9784011d80 con 0x55e380c3d070 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.135+0000 7f9773400640 1 -- 172.21.15.116:0/1433521487 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97780046c0 con 0x55e380c3d070 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.135+0000 7f9796fd8740 10 client.25445 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.135+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e380c51320 con 0x55e380c3d070 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.135+0000 7f9773400640 1 -- 172.21.15.116:0/1433521487 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9784011d80 con 0x55e380c3d070 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9773400640 1 -- 172.21.15.116:0/1433521487 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9778003ae0 con 0x55e380c3d070 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9796fd8740 10 client.25445 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9796fd8740 10 client.25445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9796fd8740 20 client.25445 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380c52970 0x55e380c54dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9796fd8740 4 client.25445 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e380ba62a0 con 0x55e380c52970 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.136+0000 7f9791800640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380c52970 0x55e380c54dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.137+0000 7f9773400640 10 client.25445 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.137+0000 7f9791800640 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380c52970 0x55e380c54dc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a57e18e95faf94 server_cookie=ce128e9610575d3a in_seq=0 out_seq=0 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.137+0000 7f9773400640 1 -- 172.21.15.116:0/1433521487 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e380ba62a0 con 0x55e380c52970 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.137+0000 7f9773400640 10 client.25445 handle_command_reply: tid=0 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 client.25445 shutdown 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3d070 msgr2=0x55e380c46f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3d070 0x55e380c46f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f977802c7b0 tx=0x7f977802bc30 comp rx=0 tx=0).stop 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 shutdown_connections 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380c52970 0x55e380c54dc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3d070 0x55e380c46f00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1433521487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3c680 0x55e380c469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 >> 172.21.15.116:0/1433521487 conn(0x55e380c37bf0 msgr2=0x55e380c2e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9773400640 0 client.25445 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 shutdown_connections 2024-03-04T07:41:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 1 -- 172.21.15.116:0/1433521487 wait complete. 2024-03-04T07:41:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.138+0000 7f9796fd8740 20 client.25445 trim_cache size 0 max 16384 2024-03-04T07:41:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.151+0000 7f9796fd8740 1 Processor -- start 2024-03-04T07:41:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9796fd8740 1 -- start start 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c3e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c4df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c421e0 0x55e380c3e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9796fd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a4710 con 0x55e380c421e0 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9796fd8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a6cf0 con 0x55e380c3c680 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9796fd8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a4d10 con 0x55e380c3d070 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9792200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c421e0 0x55e380c3e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.152+0000 7f9790e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c4df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c4df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:59630/0 (socket says 172.21.15.116:59630) 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 -- 172.21.15.116:0/1126368496 learned_addr learned my addr 172.21.15.116:0/1126368496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9791800640 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c3e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 -- 172.21.15.116:0/1126368496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 msgr2=0x55e380c3e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c3e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 -- 172.21.15.116:0/1126368496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c421e0 msgr2=0x55e380c3e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c421e0 0x55e380c3e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 -- 172.21.15.116:0/1126368496 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e380c41670 con 0x55e380c3d070 2024-03-04T07:41:36.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9791800640 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c3e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:36.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9792200640 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c421e0 0x55e380c3e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.153+0000 7f9790e00640 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c4df00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9778003ea0 tx=0x7f977802cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c4102488ef779be server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.154+0000 7f977e000640 1 -- 172.21.15.116:0/1126368496 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977804e070 con 0x55e380c3d070 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.154+0000 7f977e000640 1 -- 172.21.15.116:0/1126368496 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9778003bd0 con 0x55e380c3d070 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.154+0000 7f977e000640 1 -- 172.21.15.116:0/1126368496 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9778004e90 con 0x55e380c3d070 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 -- 172.21.15.116:0/1126368496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 msgr2=0x55e380c4df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c4df00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9778003ea0 tx=0x7f977802cdb0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 -- 172.21.15.116:0/1126368496 shutdown_connections 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c421e0 0x55e380c3e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c3d070 0x55e380c4df00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 --2- 172.21.15.116:0/1126368496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c3e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 -- 172.21.15.116:0/1126368496 >> 172.21.15.116:0/1126368496 conn(0x55e380c37bf0 msgr2=0x55e380c37fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 -- 172.21.15.116:0/1126368496 shutdown_connections 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 -- 172.21.15.116:0/1126368496 wait complete. 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 -- start start 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.155+0000 7f9796fd8740 1 Processor -- start 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c4bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3d070 0x55e380c4c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9796fd8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9796fd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a5370 con 0x55e380c3d070 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9796fd8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a6690 con 0x55e380c3c680 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9796fd8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a59d0 con 0x55e380c421e0 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9796fd8740 10 client.0 _ll_register_callbacks cb 0x7f9778043050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9791800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c4bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9790e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3d070 0x55e380c4c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9791800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c4bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44312/0 (socket says 172.21.15.116:44312) 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9791800640 1 -- 172.21.15.116:0/2284061527 learned_addr learned my addr 172.21.15.116:0/2284061527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.156+0000 7f9792200640 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9791800640 1 -- 172.21.15.116:0/2284061527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 msgr2=0x55e380c4d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9791800640 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9791800640 1 -- 172.21.15.116:0/2284061527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3d070 msgr2=0x55e380c4c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9791800640 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3d070 0x55e380c4c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9791800640 1 -- 172.21.15.116:0/2284061527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e380c4d5f0 con 0x55e380c3c680 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9790e00640 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3d070 0x55e380c4c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9791800640 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c4bb50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f978009a7a0 tx=0x7f978009ae20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43dd1032bd31d24e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9773400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9773400640 1 -- 172.21.15.116:0/2284061527 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800b9200 con 0x55e380c3c680 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9773400640 1 -- 172.21.15.116:0/2284061527 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9780010ac0 con 0x55e380c3c680 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.157+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9778003bd0 con 0x55e380c3c680 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9773400640 1 -- 172.21.15.116:0/2284061527 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9780097490 con 0x55e380c3c680 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 10 client.25451 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e380cae770 con 0x55e380c3c680 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9773400640 1 -- 172.21.15.116:0/2284061527 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9778003bd0 con 0x55e380c3c680 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9773400640 1 -- 172.21.15.116:0/2284061527 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f978009a270 con 0x55e380c3c680 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 10 client.25451 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 10 client.25451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 20 client.25451 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380caf0f0 0x55e380cb1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 4 client.25451 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:41:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.158+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e380ba05f0 con 0x55e380caf0f0 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.159+0000 7f9790e00640 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380caf0f0 0x55e380cb1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.159+0000 7f9773400640 10 client.25451 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.159+0000 7f9790e00640 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380caf0f0 0x55e380cb1510 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=833ad2c508ddcc8 server_cookie=1f7b1e281a05d327 in_seq=0 out_seq=0 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9773400640 1 -- 172.21.15.116:0/2284061527 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e380ba05f0 con 0x55e380caf0f0 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9773400640 10 client.25451 handle_command_reply: tid=0 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 client.25451 shutdown 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 msgr2=0x55e380c4bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c4bb50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f978009a7a0 tx=0x7f978009ae20 comp rx=0 tx=0).stop 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 shutdown_connections 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e380caf0f0 0x55e380cb1510 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e380c421e0 0x55e380c4d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e380c3d070 0x55e380c4c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 --2- 172.21.15.116:0/2284061527 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e380c3c680 0x55e380c4bb50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 >> 172.21.15.116:0/2284061527 conn(0x55e380c37bf0 msgr2=0x55e380c3e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9773400640 0 client.25451 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 shutdown_connections 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 1 -- 172.21.15.116:0/2284061527 wait complete. 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.160+0000 7f9796fd8740 20 client.25451 trim_cache size 0 max 16384 2024-03-04T07:41:36.166 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:41:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97740a1e00 msgr2=0x7f97740a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97740a1e00 0x7f97740a42c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9778000c00 tx=0x7f9778005d50 comp rx=0 tx=0).stop 2024-03-04T07:41:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c101f30 msgr2=0x7f978c1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c101f30 0x7f978c1018a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f978000e9f0 tx=0x7f978000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 shutdown_connections 2024-03-04T07:41:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97740a1e00 0x7f97740a42c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f978c107190 0x7f978c100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978c104860 0x7f978c0ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 --2- 172.21.15.116:0/3145223346 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f978c101f30 0x7f978c1018a0 secure :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f978000e9f0 tx=0x7f978000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:41:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 >> 172.21.15.116:0/3145223346 conn(0x7f978c0fb940 msgr2=0x7f978c102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 shutdown_connections 2024-03-04T07:41:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:36.161+0000 7f9792c00640 1 -- 172.21.15.116:0/3145223346 wait complete. 2024-03-04T07:41:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:36 smithi133 ceph-mon[36505]: pgmap v426: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:36 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2370536737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:41:37.179 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:41:37.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:41:37.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:41:37.250 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:36 smithi116 ceph-mon[32150]: pgmap v426: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:37.250 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:36 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2370536737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:41:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:36 smithi152 ceph-mon[37154]: pgmap v426: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:36 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2370536737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:41:37.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.369+0000 7fa15c600640 1 -- 172.21.15.116:0/1275697973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 msgr2=0x7fa154104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.369+0000 7fa15c600640 1 --2- 172.21.15.116:0/1275697973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 0x7fa154104c40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa140009860 tx=0x7fa14002bb70 comp rx=0 tx=0).stop 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.369+0000 7fa15c600640 1 -- 172.21.15.116:0/1275697973 shutdown_connections 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.369+0000 7fa15c600640 1 --2- 172.21.15.116:0/1275697973 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa154105180 0x7fa154105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.369+0000 7fa15c600640 1 --2- 172.21.15.116:0/1275697973 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 0x7fa154104c40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.369+0000 7fa15c600640 1 --2- 172.21.15.116:0/1275697973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540ffe60 0x7fa154102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.369+0000 7fa15c600640 1 -- 172.21.15.116:0/1275697973 >> 172.21.15.116:0/1275697973 conn(0x7fa1540fba60 msgr2=0x7fa1540fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.370+0000 7fa15c600640 1 -- 172.21.15.116:0/1275697973 shutdown_connections 2024-03-04T07:41:37.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.370+0000 7fa15c600640 1 -- 172.21.15.116:0/1275697973 wait complete. 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 Processor -- start 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 -- start start 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540ffe60 0x7fa15419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 0x7fa15419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa154105180 0x7fa1541a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154113840 con 0x7fa1540ffe60 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541136c0 con 0x7fa154105180 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa15c600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541139c0 con 0x7fa1541027d0 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa159c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540ffe60 0x7fa15419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa159200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 0x7fa15419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.371+0000 7fa159c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540ffe60 0x7fa15419df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39034/0 (socket says 172.21.15.116:39034) 2024-03-04T07:41:37.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa159c00640 1 -- 172.21.15.116:0/112976773 learned_addr learned my addr 172.21.15.116:0/112976773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa15a600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa154105180 0x7fa1541a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa15a600640 1 -- 172.21.15.116:0/112976773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 msgr2=0x7fa15419e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa15a600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 0x7fa15419e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa15a600640 1 -- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540ffe60 msgr2=0x7fa15419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa15a600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540ffe60 0x7fa15419df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa15a600640 1 -- 172.21.15.116:0/112976773 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1541a5b70 con 0x7fa154105180 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa159200640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 0x7fa15419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.372+0000 7fa15a600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa154105180 0x7fa1541a5500 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00ec50 tx=0x7fa14c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.373+0000 7fa152000640 1 -- 172.21.15.116:0/112976773 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c003b70 con 0x7fa154105180 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.373+0000 7fa152000640 1 -- 172.21.15.116:0/112976773 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa14c003d10 con 0x7fa154105180 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.373+0000 7fa152000640 1 -- 172.21.15.116:0/112976773 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c010890 con 0x7fa154105180 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.373+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1541a5e60 con 0x7fa154105180 2024-03-04T07:41:37.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.373+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1541a62d0 con 0x7fa154105180 2024-03-04T07:41:37.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.376+0000 7fa15f3a8740 1 Processor -- start 2024-03-04T07:41:37.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.377+0000 7fa152000640 1 -- 172.21.15.116:0/112976773 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa14c003fc0 con 0x7fa154105180 2024-03-04T07:41:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.378+0000 7fa15f3a8740 1 -- start start 2024-03-04T07:41:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.378+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5d0d0 0x55cff8e5d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.378+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.378+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e62c30 0x55cff8e650c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.378+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8d70e20 con 0x55cff8e5dac0 2024-03-04T07:41:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.378+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8dc2a20 con 0x55cff8e62c30 2024-03-04T07:41:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.378+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8d6bb00 con 0x55cff8e5d0d0 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e62c30 0x55cff8e650c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e62c30 0x55cff8e650c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44338/0 (socket says 172.21.15.116:44338) 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 -- 172.21.15.116:0/1052199402 learned_addr learned my addr 172.21.15.116:0/1052199402 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159c00640 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5d0d0 0x55cff8e5d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 -- 172.21.15.116:0/1052199402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5d0d0 msgr2=0x55cff8e5d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5d0d0 0x55cff8e5d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 -- 172.21.15.116:0/1052199402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 msgr2=0x55cff8e625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e625c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159200640 1 -- 172.21.15.116:0/1052199402 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cff8d80e10 con 0x55cff8e62c30 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.379+0000 7fa159c00640 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e625c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.380+0000 7fa152000640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa13c0a1e00 0x7fa13c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.380+0000 7fa159200640 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e62c30 0x55cff8e650c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa1400057c0 tx=0x7fa140035f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df6ed530cd2528d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.380+0000 7fa152000640 1 -- 172.21.15.116:0/112976773 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa14c0ce290 con 0x7fa154105180 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.380+0000 7fa159c00640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa13c0a1e00 0x7fa13c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.380+0000 7fa146a00640 1 -- 172.21.15.116:0/1052199402 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140049070 con 0x55cff8e62c30 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.380+0000 7fa146a00640 1 -- 172.21.15.116:0/1052199402 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa14002c510 con 0x55cff8e62c30 2024-03-04T07:41:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.380+0000 7fa146a00640 1 -- 172.21.15.116:0/1052199402 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140039040 con 0x55cff8e62c30 2024-03-04T07:41:37.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.381+0000 7fa15f3a8740 1 -- 172.21.15.116:0/1052199402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e62c30 msgr2=0x55cff8e650c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.381+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e62c30 0x55cff8e650c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa1400057c0 tx=0x7fa140035f70 comp rx=0 tx=0).stop 2024-03-04T07:41:37.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 -- 172.21.15.116:0/1052199402 shutdown_connections 2024-03-04T07:41:37.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e62c30 0x55cff8e650c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e625c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/1052199402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5d0d0 0x55cff8e5d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 -- 172.21.15.116:0/1052199402 >> 172.21.15.116:0/1052199402 conn(0x55cff8e58640 msgr2=0x55cff8e58a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 -- 172.21.15.116:0/1052199402 shutdown_connections 2024-03-04T07:41:37.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 -- 172.21.15.116:0/1052199402 wait complete. 2024-03-04T07:41:37.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa159c00640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa13c0a1e00 0x7fa13c0a42c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55cff8e595b0 tx=0x7fa148009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.382+0000 7fa15f3a8740 1 -- start start 2024-03-04T07:41:37.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 1 Processor -- start 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e674f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5dac0 0x55cff8e67a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e62c30 0x55cff8e6ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8d7ef40 con 0x55cff8e62c30 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8dbeac0 con 0x55cff8e5d0d0 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8e65b80 con 0x55cff8e5dac0 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.383+0000 7fa15f3a8740 10 client.0 _ll_register_callbacks cb 0x55cff8dbdaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e674f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44354/0 (socket says 172.21.15.116:44354) 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 -- 172.21.15.116:0/3972786229 learned_addr learned my addr 172.21.15.116:0/3972786229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159200640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5dac0 0x55cff8e67a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa15a600640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e62c30 0x55cff8e6ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 -- 172.21.15.116:0/3972786229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5dac0 msgr2=0x55cff8e67a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5dac0 0x55cff8e67a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 -- 172.21.15.116:0/3972786229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e62c30 msgr2=0x55cff8e6ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e62c30 0x55cff8e6ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa159c00640 1 -- 172.21.15.116:0/3972786229 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cff8b9f7e0 con 0x55cff8e5d0d0 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.384+0000 7fa15a600640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e62c30 0x55cff8e6ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa159c00640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e674f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa148004600 tx=0x7fa148004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51b09b8b7ca6b337 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa13b400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa13b400640 1 -- 172.21.15.116:0/3972786229 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14801e740 con 0x55cff8e5d0d0 2024-03-04T07:41:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa13b400640 1 -- 172.21.15.116:0/3972786229 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa14801e8e0 con 0x55cff8e5d0d0 2024-03-04T07:41:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa14002c510 con 0x55cff8e5d0d0 2024-03-04T07:41:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa13b400640 1 -- 172.21.15.116:0/3972786229 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa14801eae0 con 0x55cff8e5d0d0 2024-03-04T07:41:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa15f3a8740 10 client.25463 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.385+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cff8bfff70 con 0x55cff8e5d0d0 2024-03-04T07:41:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa13b400640 1 -- 172.21.15.116:0/3972786229 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa14002c510 con 0x55cff8e5d0d0 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa13b400640 1 -- 172.21.15.116:0/3972786229 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa148027070 con 0x55cff8e5d0d0 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa15f3a8740 10 client.25463 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa15f3a8740 10 client.25463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa15f3a8740 20 client.25463 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8e73310 0x55cff8e757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa15f3a8740 4 client.25463 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cff8dcff60 con 0x55cff8e73310 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.386+0000 7fa159200640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8e73310 0x55cff8e757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.387+0000 7fa159200640 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8e73310 0x55cff8e757b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eea0c0d53d28837a server_cookie=6eb89fe47b31df5e in_seq=0 out_seq=0 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.387+0000 7fa13b400640 10 client.25463 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.387+0000 7fa13b400640 1 -- 172.21.15.116:0/3972786229 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cff8dcff60 con 0x55cff8e73310 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.387+0000 7fa13b400640 10 client.25463 handle_command_reply: tid=0 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 client.25463 shutdown 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 msgr2=0x55cff8e674f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e674f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa148004600 tx=0x7fa148004730 comp rx=0 tx=0).stop 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 shutdown_connections 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8e73310 0x55cff8e757b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e62c30 0x55cff8e6ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e5dac0 0x55cff8e67a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3972786229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e674f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 >> 172.21.15.116:0/3972786229 conn(0x55cff8e58640 msgr2=0x55cff8e59890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa13b400640 0 client.25463 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 shutdown_connections 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3972786229 wait complete. 2024-03-04T07:41:37.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.388+0000 7fa15f3a8740 20 client.25463 trim_cache size 0 max 16384 2024-03-04T07:41:37.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.405+0000 7fa15f3a8740 1 Processor -- start 2024-03-04T07:41:37.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.405+0000 7fa15f3a8740 1 -- start start 2024-03-04T07:41:37.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e67810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e6ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541ad130 con 0x55cff8e5dac0 2024-03-04T07:41:37.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541af710 con 0x55cff8e5d0d0 2024-03-04T07:41:37.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541ad730 con 0x55cff8e62c30 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e6ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e67810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e67810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44362/0 (socket says 172.21.15.116:44362) 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 -- 172.21.15.116:0/3238989869 learned_addr learned my addr 172.21.15.116:0/3238989869 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa15a600640 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 -- 172.21.15.116:0/3238989869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 msgr2=0x55cff8e571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 -- 172.21.15.116:0/3238989869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 msgr2=0x55cff8e6ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e6ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.406+0000 7fa159c00640 1 -- 172.21.15.116:0/3238989869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cff8a97170 con 0x55cff8e5d0d0 2024-03-04T07:41:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.407+0000 7fa159c00640 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e67810 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa148009a80 tx=0x7fa148004540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27a40779653e0195 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.407+0000 7fa146000640 1 -- 172.21.15.116:0/3238989869 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14801ec80 con 0x55cff8e5d0d0 2024-03-04T07:41:37.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.407+0000 7fa146000640 1 -- 172.21.15.116:0/3238989869 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa148038070 con 0x55cff8e5d0d0 2024-03-04T07:41:37.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.407+0000 7fa146000640 1 -- 172.21.15.116:0/3238989869 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148004b10 con 0x55cff8e5d0d0 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3238989869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 msgr2=0x55cff8e67810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e67810 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa148009a80 tx=0x7fa148004540 comp rx=0 tx=0).stop 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3238989869 shutdown_connections 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5dac0 0x55cff8e6ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3238989869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5d0d0 0x55cff8e67810 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3238989869 >> 172.21.15.116:0/3238989869 conn(0x55cff8e58640 msgr2=0x55cff8e67100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3238989869 shutdown_connections 2024-03-04T07:41:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3238989869 wait complete. 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.408+0000 7fa15f3a8740 1 -- start start 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 1 Processor -- start 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5d0d0 0x55cff8e6c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5dac0 0x55cff8e6cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e6da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541add90 con 0x55cff8e5d0d0 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541af0b0 con 0x55cff8e5dac0 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541ae3f0 con 0x55cff8e62c30 2024-03-04T07:41:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15f3a8740 10 client.0 _ll_register_callbacks cb 0x7fa148009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15a600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e6da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15a600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e6da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:59684/0 (socket says 172.21.15.116:59684) 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa15a600640 1 -- 172.21.15.116:0/3285160748 learned_addr learned my addr 172.21.15.116:0/3285160748 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.409+0000 7fa159200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5dac0 0x55cff8e6cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa159c00640 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5d0d0 0x55cff8e6c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa15a600640 1 -- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5dac0 msgr2=0x55cff8e6cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa15a600640 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5dac0 0x55cff8e6cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa15a600640 1 -- 172.21.15.116:0/3285160748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5d0d0 msgr2=0x55cff8e6c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa15a600640 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5d0d0 0x55cff8e6c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa15a600640 1 -- 172.21.15.116:0/3285160748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cff8e6e050 con 0x55cff8e62c30 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa159200640 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5dac0 0x55cff8e6cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa15a600640 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e6da60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa14c005fd0 tx=0x7fa14c009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d55196f4e9d86d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.410+0000 7fa13b400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa13b400640 1 -- 172.21.15.116:0/3285160748 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c0a34d0 con 0x55cff8e62c30 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa148038070 con 0x55cff8e62c30 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa13b400640 1 -- 172.21.15.116:0/3285160748 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa14c0b9270 con 0x55cff8e62c30 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa13b400640 1 -- 172.21.15.116:0/3285160748 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa14c09f110 con 0x55cff8e62c30 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa15f3a8740 10 client.25444 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55cff8ecefa0 con 0x55cff8e62c30 2024-03-04T07:41:37.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa13b400640 1 -- 172.21.15.116:0/3285160748 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa148038070 con 0x55cff8e62c30 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.411+0000 7fa13b400640 1 -- 172.21.15.116:0/3285160748 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa14c09d040 con 0x55cff8e62c30 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.412+0000 7fa15f3a8740 10 client.25444 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.412+0000 7fa15f3a8740 10 client.25444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.412+0000 7fa15f3a8740 20 client.25444 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.412+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8ecf970 0x55cff8ed1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.412+0000 7fa15f3a8740 4 client.25444 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.412+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cff8dca2b0 con 0x55cff8ecf970 2024-03-04T07:41:37.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.412+0000 7fa159c00640 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8ecf970 0x55cff8ed1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.413+0000 7fa13b400640 10 client.25444 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.413+0000 7fa159c00640 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8ecf970 0x55cff8ed1dd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5960c7545c38883 server_cookie=e6436aa38f123c17 in_seq=0 out_seq=0 2024-03-04T07:41:37.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa13b400640 1 -- 172.21.15.116:0/3285160748 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cff8dca2b0 con 0x55cff8ecf970 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa13b400640 10 client.25444 handle_command_reply: tid=0 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 client.25444 shutdown 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 msgr2=0x55cff8e6da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e6da60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa14c005fd0 tx=0x7fa14c009820 comp rx=0 tx=0).stop 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 shutdown_connections 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55cff8ecf970 0x55cff8ed1dd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cff8e62c30 0x55cff8e6da60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55cff8e5dac0 0x55cff8e6cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 --2- 172.21.15.116:0/3285160748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cff8e5d0d0 0x55cff8e6c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 >> 172.21.15.116:0/3285160748 conn(0x55cff8e58640 msgr2=0x55cff8e571d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.414+0000 7fa13b400640 0 client.25444 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.415+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 shutdown_connections 2024-03-04T07:41:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.415+0000 7fa15f3a8740 1 -- 172.21.15.116:0/3285160748 wait complete. 2024-03-04T07:41:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.415+0000 7fa15f3a8740 20 client.25444 trim_cache size 0 max 16384 2024-03-04T07:41:37.418 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:41:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:41:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "adc7a048-937f-4daa-b643-1ddfe1b24058", 2024-03-04T07:41:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:41:37.418 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:41:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa13c0a1e00 msgr2=0x7fa13c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa13c0a1e00 0x7fa13c0a42c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55cff8e595b0 tx=0x7fa148009260 comp rx=0 tx=0).stop 2024-03-04T07:41:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa154105180 msgr2=0x7fa1541a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa154105180 0x7fa1541a5500 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00ec50 tx=0x7fa14c00c6a0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 shutdown_connections 2024-03-04T07:41:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa154105180 0x7fa1541a5500 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa13c0a1e00 0x7fa13c0a42c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1541027d0 0x7fa15419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 --2- 172.21.15.116:0/112976773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540ffe60 0x7fa15419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 >> 172.21.15.116:0/112976773 conn(0x7fa1540fba60 msgr2=0x7fa1541008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 shutdown_connections 2024-03-04T07:41:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.416+0000 7fa15c600640 1 -- 172.21.15.116:0/112976773 wait complete. 2024-03-04T07:41:37.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:41:37.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:41:37.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.632+0000 7fe2e1800640 1 -- 172.21.15.116:0/1433095154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 msgr2=0x7fe2dc103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.632+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1433095154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 0x7fe2dc103c30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009860 tx=0x7fe2cc02bb70 comp rx=0 tx=0).stop 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.632+0000 7fe2e1800640 1 -- 172.21.15.116:0/1433095154 shutdown_connections 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.632+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1433095154 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc1053c0 0x7fe2dc10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.632+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1433095154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2dc104a00 0x7fe2dc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.632+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1433095154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 0x7fe2dc103c30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.632+0000 7fe2e1800640 1 -- 172.21.15.116:0/1433095154 >> 172.21.15.116:0/1433095154 conn(0x7fe2dc0fed80 msgr2=0x7fe2dc1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.633+0000 7fe2e1800640 1 -- 172.21.15.116:0/1433095154 shutdown_connections 2024-03-04T07:41:37.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.633+0000 7fe2e1800640 1 -- 172.21.15.116:0/1433095154 wait complete. 2024-03-04T07:41:37.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.633+0000 7fe2e1800640 1 Processor -- start 2024-03-04T07:41:37.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2e1800640 1 -- start start 2024-03-04T07:41:37.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2e1800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 0x7fe2dc199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2e1800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc104a00 0x7fe2dc19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2e1800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2dc1053c0 0x7fe2dc1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2e1800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc10f460 con 0x7fe2dc1053c0 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2e1800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc10f2e0 con 0x7fe2dc104a00 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2e1800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc10f5e0 con 0x7fe2dc103810 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2da000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc104a00 0x7fe2dc19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2da000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc104a00 0x7fe2dc19a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44382/0 (socket says 172.21.15.116:44382) 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2da000640 1 -- 172.21.15.116:0/1755943351 learned_addr learned my addr 172.21.15.116:0/1755943351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.634+0000 7fe2daa00640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 0x7fe2dc199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2da000640 1 -- 172.21.15.116:0/1755943351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 msgr2=0x7fe2dc199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2da000640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 0x7fe2dc199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2da000640 1 -- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2dc1053c0 msgr2=0x7fe2dc1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2da000640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2dc1053c0 0x7fe2dc1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2da000640 1 -- 172.21.15.116:0/1755943351 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc1a1750 con 0x7fe2dc104a00 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2da000640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc104a00 0x7fe2dc19a070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800e9c0 tx=0x7fe2c800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2d3400640 1 -- 172.21.15.116:0/1755943351 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c800cde0 con 0x7fe2dc104a00 2024-03-04T07:41:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2d3400640 1 -- 172.21.15.116:0/1755943351 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2c80035f0 con 0x7fe2dc104a00 2024-03-04T07:41:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2d3400640 1 -- 172.21.15.116:0/1755943351 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8010690 con 0x7fe2dc104a00 2024-03-04T07:41:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc1a1a40 con 0x7fe2dc104a00 2024-03-04T07:41:37.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.635+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc1a1f40 con 0x7fe2dc104a00 2024-03-04T07:41:37.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.639+0000 7fe2d3400640 1 -- 172.21.15.116:0/1755943351 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe2c80037a0 con 0x7fe2dc104a00 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.640+0000 7fe2e447d740 1 Processor -- start 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.640+0000 7fe2e447d740 1 -- start start 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2d3400640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe2c00a1e00 0x7fe2c00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2d3400640 1 -- 172.21.15.116:0/1755943351 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe2c80ce190 con 0x7fe2dc104a00 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x5596282f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f6540 0x5596282fb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x5596282fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2e447d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559628223090 con 0x5596282f5b50 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2e447d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55962825ad30 con 0x5596282fb6b0 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.641+0000 7fe2e447d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559628203a30 con 0x5596282f6540 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2db400640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe2c00a1e00 0x7fe2c00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x5596282f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x5596282f5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39090/0 (socket says 172.21.15.116:39090) 2024-03-04T07:41:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 -- 172.21.15.116:0/3429391804 learned_addr learned my addr 172.21.15.116:0/3429391804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2db400640 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x5596282fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2da000640 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f6540 0x5596282fb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 -- 172.21.15.116:0/3429391804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f6540 msgr2=0x5596282fb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f6540 0x5596282fb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 -- 172.21.15.116:0/3429391804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 msgr2=0x5596282fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x5596282fdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.642+0000 7fe2daa00640 1 -- 172.21.15.116:0/3429391804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596280397d0 con 0x5596282f5b50 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.643+0000 7fe2db400640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe2c00a1e00 0x7fe2c00a42c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009fd0 tx=0x7fe2d40098f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.643+0000 7fe2daa00640 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x5596282f5f70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc000c00 tx=0x7fe2cc005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80dd0414592816b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.643+0000 7fe2c7e00640 1 -- 172.21.15.116:0/3429391804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc039040 con 0x5596282f5b50 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.643+0000 7fe2c7e00640 1 -- 172.21.15.116:0/3429391804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2cc043070 con 0x5596282f5b50 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 -- 172.21.15.116:0/3429391804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 msgr2=0x5596282f5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x5596282f5f70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc000c00 tx=0x7fe2cc005da0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 -- 172.21.15.116:0/3429391804 shutdown_connections 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x5596282fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f6540 0x5596282fb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3429391804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x5596282f5f70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 -- 172.21.15.116:0/3429391804 >> 172.21.15.116:0/3429391804 conn(0x5596282f10c0 msgr2=0x5596282f14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.644+0000 7fe2e447d740 1 -- 172.21.15.116:0/3429391804 shutdown_connections 2024-03-04T07:41:37.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.645+0000 7fe2e447d740 1 -- 172.21.15.116:0/3429391804 wait complete. 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.645+0000 7fe2e447d740 1 -- start start 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.645+0000 7fe2e447d740 1 Processor -- start 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.646+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f5b50 0x559628308700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.646+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x559628308c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.646+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x55962830fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.646+0000 7fe2e447d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596281f90c0 con 0x5596282f6540 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.646+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f5b50 0x559628308700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.646+0000 7fe2e447d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559628256dd0 con 0x5596282fb6b0 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.646+0000 7fe2e447d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596282fe6a0 con 0x5596282f5b50 2024-03-04T07:41:37.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f5b50 0x559628308700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:59720/0 (socket says 172.21.15.116:59720) 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 -- 172.21.15.116:0/3565566771 learned_addr learned my addr 172.21.15.116:0/3565566771 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2e447d740 10 client.0 _ll_register_callbacks cb 0x559628216860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2da000640 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x559628308c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2db400640 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x55962830fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 -- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 msgr2=0x55962830fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x55962830fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 -- 172.21.15.116:0/3565566771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 msgr2=0x559628308c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x559628308c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 -- 172.21.15.116:0/3565566771 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559627f6f3f0 con 0x5596282f5b50 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2daa00640 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f5b50 0x559628308700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc040040 tx=0x7fe2cc0058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3213c6171815be50 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.647+0000 7fe2c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.648+0000 7fe2c4c00640 1 -- 172.21.15.116:0/3565566771 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc039040 con 0x5596282f5b50 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.648+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2cc043070 con 0x5596282f5b50 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.648+0000 7fe2c4c00640 1 -- 172.21.15.116:0/3565566771 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe2cc034db0 con 0x5596282f5b50 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.648+0000 7fe2c4c00640 1 -- 172.21.15.116:0/3565566771 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2cc034440 con 0x5596282f5b50 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.648+0000 7fe2e447d740 10 client.25456 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.648+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559627f2f0f0 con 0x5596282f5b50 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2c4c00640 1 -- 172.21.15.116:0/3565566771 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2cc043070 con 0x5596282f5b50 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2c4c00640 1 -- 172.21.15.116:0/3565566771 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe2cc0048f0 con 0x5596282f5b50 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2e447d740 10 client.25456 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:37.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2e447d740 10 client.25456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2e447d740 20 client.25456 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628313d50 0x559628316140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2e447d740 4 client.25456 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559628268270 con 0x559628313d50 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.649+0000 7fe2da000640 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628313d50 0x559628316140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.650+0000 7fe2da000640 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628313d50 0x559628316140 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa3ec88176c40cc2 server_cookie=879af08ffbe59044 in_seq=0 out_seq=0 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.650+0000 7fe2c4c00640 10 client.25456 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.650+0000 7fe2c4c00640 1 -- 172.21.15.116:0/3565566771 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559628268270 con 0x559628313d50 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.650+0000 7fe2c4c00640 10 client.25456 handle_command_reply: tid=0 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.650+0000 7fe2e447d740 1 client.25456 shutdown 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f5b50 msgr2=0x559628308700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f5b50 0x559628308700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc040040 tx=0x7fe2cc0058b0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 shutdown_connections 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628313d50 0x559628316140 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282fb6b0 0x55962830fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x559628308c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 --2- 172.21.15.116:0/3565566771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282f5b50 0x559628308700 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2c4c00640 0 client.25456 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 >> 172.21.15.116:0/3565566771 conn(0x5596282f10c0 msgr2=0x5596282f0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 shutdown_connections 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 1 -- 172.21.15.116:0/3565566771 wait complete. 2024-03-04T07:41:37.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.651+0000 7fe2e447d740 20 client.25456 trim_cache size 0 max 16384 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 Processor -- start 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 -- start start 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f5b50 0x5596282f4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x55962830fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x559628307d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1a48f0 con 0x5596282f6540 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1a6eb0 con 0x5596282f5b50 2024-03-04T07:41:37.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.668+0000 7fe2e447d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1a4ef0 con 0x5596282fb6b0 2024-03-04T07:41:37.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f5b50 0x5596282f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f5b50 0x5596282f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44410/0 (socket says 172.21.15.116:44410) 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 -- 172.21.15.116:0/1545488369 learned_addr learned my addr 172.21.15.116:0/1545488369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2db400640 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x559628307d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2da000640 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x55962830fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 -- 172.21.15.116:0/1545488369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 msgr2=0x559628307d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x559628307d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 -- 172.21.15.116:0/1545488369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 msgr2=0x55962830fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x55962830fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 -- 172.21.15.116:0/1545488369 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596282f6920 con 0x5596282f5b50 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2db400640 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x559628307d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:41:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.669+0000 7fe2daa00640 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f5b50 0x5596282f4990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009780 tx=0x7fe2cc02c1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a132d8aba0d229 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.670+0000 7fe2c7400640 1 -- 172.21.15.116:0/1545488369 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc039070 con 0x5596282f5b50 2024-03-04T07:41:37.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.670+0000 7fe2c7400640 1 -- 172.21.15.116:0/1545488369 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2cc043070 con 0x5596282f5b50 2024-03-04T07:41:37.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.670+0000 7fe2c7400640 1 -- 172.21.15.116:0/1545488369 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc02c510 con 0x5596282f5b50 2024-03-04T07:41:37.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.670+0000 7fe2e447d740 1 -- 172.21.15.116:0/1545488369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f5b50 msgr2=0x5596282f4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.670+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f5b50 0x5596282f4990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009780 tx=0x7fe2cc02c1f0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 -- 172.21.15.116:0/1545488369 shutdown_connections 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x559628307d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f6540 0x55962830fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1545488369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f5b50 0x5596282f4990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 -- 172.21.15.116:0/1545488369 >> 172.21.15.116:0/1545488369 conn(0x5596282f10c0 msgr2=0x5596282f0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 -- 172.21.15.116:0/1545488369 shutdown_connections 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 -- 172.21.15.116:0/1545488369 wait complete. 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 -- start start 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.671+0000 7fe2e447d740 1 Processor -- start 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x55962830e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f6540 0x55962830e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2e447d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x55962830ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2e447d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1a5530 con 0x5596282f5b50 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2e447d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1a6850 con 0x5596282f6540 2024-03-04T07:41:37.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2e447d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc1a5b90 con 0x5596282fb6b0 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2e447d740 10 client.0 _ll_register_callbacks cb 0x7fe2c8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x55962830e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2daa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x55962830e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39112/0 (socket says 172.21.15.116:39112) 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2daa00640 1 -- 172.21.15.116:0/1306026048 learned_addr learned my addr 172.21.15.116:0/1306026048 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2da000640 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f6540 0x55962830e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.672+0000 7fe2db400640 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x55962830ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2da000640 1 -- 172.21.15.116:0/1306026048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 msgr2=0x55962830ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2da000640 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x55962830ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2da000640 1 -- 172.21.15.116:0/1306026048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 msgr2=0x55962830e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2da000640 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x55962830e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2da000640 1 -- 172.21.15.116:0/1306026048 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55962836f4e0 con 0x5596282f6540 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2da000640 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f6540 0x55962830e750 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000c00 tx=0x7fe2c809b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ee96e0642c93dde server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2c4c00640 1 -- 172.21.15.116:0/1306026048 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c80a3660 con 0x5596282f6540 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2c4c00640 1 -- 172.21.15.116:0/1306026048 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe2c80991c0 con 0x5596282f6540 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2cc043070 con 0x5596282f6540 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2c4c00640 1 -- 172.21.15.116:0/1306026048 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2c8003b10 con 0x5596282f6540 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2e447d740 10 client.25487 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.673+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559628370740 con 0x5596282f6540 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2c4c00640 1 -- 172.21.15.116:0/1306026048 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2cc043070 con 0x5596282f6540 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2c4c00640 1 -- 172.21.15.116:0/1306026048 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe2c809a3a0 con 0x5596282f6540 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2e447d740 10 client.25487 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2e447d740 10 client.25487 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:37.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2e447d740 20 client.25487 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:37.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628371050 0x559628373470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2e447d740 4 client.25487 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:41:37.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5596282625c0 con 0x559628371050 2024-03-04T07:41:37.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.674+0000 7fe2daa00640 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628371050 0x559628373470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2daa00640 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628371050 0x559628373470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=189eeff9f2bea72c server_cookie=fbe0521118da820f in_seq=0 out_seq=0 2024-03-04T07:41:37.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2c4c00640 10 client.25487 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2c4c00640 1 -- 172.21.15.116:0/1306026048 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5596282625c0 con 0x559628371050 2024-03-04T07:41:37.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2c4c00640 10 client.25487 handle_command_reply: tid=0 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 client.25487 shutdown 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f6540 msgr2=0x55962830e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f6540 0x55962830e750 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000c00 tx=0x7fe2c809b680 comp rx=0 tx=0).stop 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 shutdown_connections 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559628371050 0x559628373470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5596282fb6b0 0x55962830ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5596282f6540 0x55962830e750 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 --2- 172.21.15.116:0/1306026048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596282f5b50 0x55962830e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 >> 172.21.15.116:0/1306026048 conn(0x5596282f10c0 msgr2=0x55962831a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2c4c00640 0 client.25487 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 shutdown_connections 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.675+0000 7fe2e447d740 1 -- 172.21.15.116:0/1306026048 wait complete. 2024-03-04T07:41:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e447d740 20 client.25487 trim_cache size 0 max 16384 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: "adc7a048-937f-4daa-b643-1ddfe1b24058": { 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "adc7a048-937f-4daa-b643-1ddfe1b24058", 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:41:37.681 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:41:37.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe2c00a1e00 msgr2=0x7fe2c00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe2c00a1e00 0x7fe2c00a42c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009fd0 tx=0x7fe2d40098f0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc104a00 msgr2=0x7fe2dc19a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc104a00 0x7fe2dc19a070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800e9c0 tx=0x7fe2c800ee80 comp rx=0 tx=0).stop 2024-03-04T07:41:37.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 shutdown_connections 2024-03-04T07:41:37.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2dc1053c0 0x7fe2dc1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2dc104a00 0x7fe2dc19a070 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe2c00a1e00 0x7fe2c00a42c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 --2- 172.21.15.116:0/1755943351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2dc103810 0x7fe2dc199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.676+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 >> 172.21.15.116:0/1755943351 conn(0x7fe2dc0fed80 msgr2=0x7fe2dc107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.677+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 shutdown_connections 2024-03-04T07:41:37.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:41:37.677+0000 7fe2e1800640 1 -- 172.21.15.116:0/1755943351 wait complete. 2024-03-04T07:41:37.694 INFO:tasks.cephfs.filesystem:scrub status for tag:adc7a048-937f-4daa-b643-1ddfe1b24058 - {'path': '/', 'tag': 'adc7a048-937f-4daa-b643-1ddfe1b24058', 'options': 'recursive,force'} 2024-03-04T07:41:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.881+0000 7fe835400640 1 -- 172.21.15.116:0/1483461744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830075610 msgr2=0x7fe830073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.881+0000 7fe835400640 1 --2- 172.21.15.116:0/1483461744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830075610 0x7fe830073a60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe820009930 tx=0x7fe82002bc20 comp rx=0 tx=0).stop 2024-03-04T07:41:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.881+0000 7fe835400640 1 -- 172.21.15.116:0/1483461744 shutdown_connections 2024-03-04T07:41:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.881+0000 7fe835400640 1 --2- 172.21.15.116:0/1483461744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe83010da10 0x7fe83010fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.881+0000 7fe835400640 1 --2- 172.21.15.116:0/1483461744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe830074030 0x7fe8300744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.881+0000 7fe835400640 1 --2- 172.21.15.116:0/1483461744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830075610 0x7fe830073a60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.881+0000 7fe835400640 1 -- 172.21.15.116:0/1483461744 >> 172.21.15.116:0/1483461744 conn(0x7fe8300fc9b0 msgr2=0x7fe8300fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.882+0000 7fe835400640 1 -- 172.21.15.116:0/1483461744 shutdown_connections 2024-03-04T07:41:37.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.882+0000 7fe835400640 1 -- 172.21.15.116:0/1483461744 wait complete. 2024-03-04T07:41:37.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.882+0000 7fe835400640 1 Processor -- start 2024-03-04T07:41:37.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe835400640 1 -- start start 2024-03-04T07:41:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe835400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830074030 0x7fe83019dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe835400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe830075610 0x7fe83019e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe835400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe83010da10 0x7fe8301a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe835400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830113700 con 0x7fe830075610 2024-03-04T07:41:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe835400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830113580 con 0x7fe830074030 2024-03-04T07:41:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe835400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830113880 con 0x7fe83010da10 2024-03-04T07:41:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830074030 0x7fe83019dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830074030 0x7fe83019dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44424/0 (socket says 172.21.15.116:44424) 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe82fe00640 1 -- 172.21.15.116:0/2785455920 learned_addr learned my addr 172.21.15.116:0/2785455920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe82f400640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe830075610 0x7fe83019e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.883+0000 7fe834a00640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe83010da10 0x7fe8301a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82fe00640 1 -- 172.21.15.116:0/2785455920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe83010da10 msgr2=0x7fe8301a52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe83010da10 0x7fe8301a52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82fe00640 1 -- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe830075610 msgr2=0x7fe83019e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe830075610 0x7fe83019e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82fe00640 1 -- 172.21.15.116:0/2785455920 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8301a5950 con 0x7fe830074030 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830074030 0x7fe83019dd40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe820000c00 tx=0x7fe82002c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82cc00640 1 -- 172.21.15.116:0/2785455920 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820043020 con 0x7fe830074030 2024-03-04T07:41:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82cc00640 1 -- 172.21.15.116:0/2785455920 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8200054e0 con 0x7fe830074030 2024-03-04T07:41:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe82cc00640 1 -- 172.21.15.116:0/2785455920 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820003c10 con 0x7fe830074030 2024-03-04T07:41:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8301a5be0 con 0x7fe830074030 2024-03-04T07:41:37.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.884+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8301a60e0 con 0x7fe830074030 2024-03-04T07:41:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.888+0000 7fe82cc00640 1 -- 172.21.15.116:0/2785455920 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe820046020 con 0x7fe830074030 2024-03-04T07:41:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 Processor -- start 2024-03-04T07:41:37.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 -- start start 2024-03-04T07:41:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc43803e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43804400 0x55bc43808f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43809570 0x55bc4380ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc43769370 con 0x55bc43804400 2024-03-04T07:41:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc43712550 con 0x55bc43809570 2024-03-04T07:41:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.889+0000 7fe8397ba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc43765410 con 0x55bc43803a10 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43809570 0x55bc4380ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43809570 0x55bc4380ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44430/0 (socket says 172.21.15.116:44430) 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 -- 172.21.15.116:0/2091351785 learned_addr learned my addr 172.21.15.116:0/2091351785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43804400 0x55bc43808f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe834a00640 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc43803e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 -- 172.21.15.116:0/2091351785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 msgr2=0x55bc43803e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc43803e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 -- 172.21.15.116:0/2091351785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43804400 msgr2=0x55bc43808f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43804400 0x55bc43808f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82f400640 1 -- 172.21.15.116:0/2091351785 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc437263c0 con 0x55bc43809570 2024-03-04T07:41:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.890+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43804400 0x55bc43808f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.891+0000 7fe82cc00640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe8140a1e70 0x7fe8140a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.891+0000 7fe82cc00640 1 -- 172.21.15.116:0/2785455920 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe8200eff40 con 0x7fe830074030 2024-03-04T07:41:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.891+0000 7fe834a00640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe8140a1e70 0x7fe8140a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.891+0000 7fe82f400640 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43809570 0x55bc4380ba00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009c80 tx=0x7fe81c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d443d7f61336b9f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.892+0000 7fe834a00640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe8140a1e70 0x7fe8140a4330 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55bc43804c60 tx=0x7fe828009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.892+0000 7fe825200640 1 -- 172.21.15.116:0/2091351785 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c012450 con 0x55bc43809570 2024-03-04T07:41:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.892+0000 7fe825200640 1 -- 172.21.15.116:0/2091351785 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81c012ad0 con 0x55bc43809570 2024-03-04T07:41:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.892+0000 7fe825200640 1 -- 172.21.15.116:0/2091351785 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c00fc00 con 0x55bc43809570 2024-03-04T07:41:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.892+0000 7fe8397ba740 1 -- 172.21.15.116:0/2091351785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43809570 msgr2=0x55bc4380ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.892+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43809570 0x55bc4380ba00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009c80 tx=0x7fe81c0117b0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 -- 172.21.15.116:0/2091351785 shutdown_connections 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43809570 0x55bc4380ba00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43804400 0x55bc43808f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2091351785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc43803e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 -- 172.21.15.116:0/2091351785 >> 172.21.15.116:0/2091351785 conn(0x55bc437fef80 msgr2=0x55bc437ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 -- 172.21.15.116:0/2091351785 shutdown_connections 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 -- 172.21.15.116:0/2091351785 wait complete. 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 -- start start 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.893+0000 7fe8397ba740 1 Processor -- start 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc4380dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc4380e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43809570 0x55bc43815320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe8397ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc4380c520 con 0x55bc43809570 2024-03-04T07:41:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe8397ba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc437259b0 con 0x55bc43804400 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe8397ba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc4380c6a0 con 0x55bc43803a10 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe8397ba740 10 client.0 _ll_register_callbacks cb 0x55bc43761340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe82f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc4380e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe82f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc4380e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44438/0 (socket says 172.21.15.116:44438) 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe82f400640 1 -- 172.21.15.116:0/3757255317 learned_addr learned my addr 172.21.15.116:0/3757255317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe834a00640 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43809570 0x55bc43815320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.894+0000 7fe82fe00640 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc4380dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe82f400640 1 -- 172.21.15.116:0/3757255317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 msgr2=0x55bc4380dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe82f400640 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc4380dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe82f400640 1 -- 172.21.15.116:0/3757255317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43809570 msgr2=0x55bc43815320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe82f400640 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43809570 0x55bc43815320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe82f400640 1 -- 172.21.15.116:0/3757255317 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc4347e210 con 0x55bc43804400 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe82f400640 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc4380e2e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00f480 tx=0x7fe81c009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2925403bf4a3f2d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe819600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe819600640 1 -- 172.21.15.116:0/3757255317 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c028020 con 0x55bc43804400 2024-03-04T07:41:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe81c012ad0 con 0x55bc43804400 2024-03-04T07:41:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.895+0000 7fe819600640 1 -- 172.21.15.116:0/3757255317 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe81c012650 con 0x55bc43804400 2024-03-04T07:41:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe819600640 1 -- 172.21.15.116:0/3757255317 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe81c00f4f0 con 0x55bc43804400 2024-03-04T07:41:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 10 client.25499 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bc4354a620 con 0x55bc43804400 2024-03-04T07:41:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe819600640 1 -- 172.21.15.116:0/3757255317 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81c012ad0 con 0x55bc43804400 2024-03-04T07:41:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe819600640 1 -- 172.21.15.116:0/3757255317 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe81c003680 con 0x55bc43804400 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 10 client.25499 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 10 client.25499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 20 client.25499 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43819920 0x55bc4381bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 4 client.25499 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc437768b0 con 0x55bc43819920 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.896+0000 7fe82fe00640 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43819920 0x55bc4381bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.897+0000 7fe82fe00640 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43819920 0x55bc4381bd10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55b3bfbcc1148224 server_cookie=850fb5e300a010a6 in_seq=0 out_seq=0 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.897+0000 7fe819600640 10 client.25499 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.897+0000 7fe819600640 1 -- 172.21.15.116:0/3757255317 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bc437768b0 con 0x55bc43819920 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.897+0000 7fe819600640 10 client.25499 handle_command_reply: tid=0 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.897+0000 7fe8397ba740 1 client.25499 shutdown 2024-03-04T07:41:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 msgr2=0x55bc4380e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc4380e2e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00f480 tx=0x7fe81c009fa0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 shutdown_connections 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43819920 0x55bc4381bd10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43809570 0x55bc43815320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc4380e2e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe819600640 0 client.25499 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 --2- 172.21.15.116:0/3757255317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43803a10 0x55bc4380dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 >> 172.21.15.116:0/3757255317 conn(0x55bc437fef80 msgr2=0x55bc437fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 shutdown_connections 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 1 -- 172.21.15.116:0/3757255317 wait complete. 2024-03-04T07:41:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.898+0000 7fe8397ba740 20 client.25499 trim_cache size 0 max 16384 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.912+0000 7fe8397ba740 1 Processor -- start 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.912+0000 7fe8397ba740 1 -- start start 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc437fef80 0x55bc438021e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc438152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43804400 0x55bc43802720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe8397ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301acf20 con 0x55bc43803a10 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe8397ba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301af500 con 0x55bc437fef80 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe8397ba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301ad520 con 0x55bc43804400 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc437fef80 0x55bc438021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc437fef80 0x55bc438021e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44454/0 (socket says 172.21.15.116:44454) 2024-03-04T07:41:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 -- 172.21.15.116:0/2370759108 learned_addr learned my addr 172.21.15.116:0/2370759108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82f400640 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc438152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe834a00640 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43804400 0x55bc43802720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 -- 172.21.15.116:0/2370759108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43804400 msgr2=0x55bc43802720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43804400 0x55bc43802720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 -- 172.21.15.116:0/2370759108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 msgr2=0x55bc438152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc438152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.913+0000 7fe82fe00640 1 -- 172.21.15.116:0/2370759108 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc43804c90 con 0x55bc437fef80 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.914+0000 7fe834a00640 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43804400 0x55bc43802720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.914+0000 7fe82fe00640 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc437fef80 0x55bc438021e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe8200bc930 tx=0x7fe8200ed8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baf45c0d15c0515d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.914+0000 7fe81be00640 1 -- 172.21.15.116:0/2370759108 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200035f0 con 0x55bc437fef80 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.914+0000 7fe81be00640 1 -- 172.21.15.116:0/2370759108 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe820004250 con 0x55bc437fef80 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.914+0000 7fe81be00640 1 -- 172.21.15.116:0/2370759108 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200c53d0 con 0x55bc437fef80 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 -- 172.21.15.116:0/2370759108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc437fef80 msgr2=0x55bc438021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc437fef80 0x55bc438021e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe8200bc930 tx=0x7fe8200ed8d0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 -- 172.21.15.116:0/2370759108 shutdown_connections 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc43804400 0x55bc43802720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc438152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 --2- 172.21.15.116:0/2370759108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc437fef80 0x55bc438021e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 -- 172.21.15.116:0/2370759108 >> 172.21.15.116:0/2370759108 conn(0x55bc4381c080 msgr2=0x55bc43805720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 -- 172.21.15.116:0/2370759108 shutdown_connections 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 -- 172.21.15.116:0/2370759108 wait complete. 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 -- start start 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.915+0000 7fe8397ba740 1 Processor -- start 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc437fef80 0x55bc43813000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc43813540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe8397ba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc43814500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe8397ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301adb80 con 0x55bc43803a10 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe8397ba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301aeea0 con 0x55bc43804400 2024-03-04T07:41:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe8397ba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301ae1e0 con 0x55bc437fef80 2024-03-04T07:41:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe8397ba740 10 client.0 _ll_register_callbacks cb 0x7fe8280099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc437fef80 0x55bc43813000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe82f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc43813540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe834a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc43814500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe82f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc43813540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39160/0 (socket says 172.21.15.116:39160) 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.916+0000 7fe82f400640 1 -- 172.21.15.116:0/816760219 learned_addr learned my addr 172.21.15.116:0/816760219 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe82f400640 1 -- 172.21.15.116:0/816760219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc437fef80 msgr2=0x55bc43813000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe82f400640 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc437fef80 0x55bc43813000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe82f400640 1 -- 172.21.15.116:0/816760219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 msgr2=0x55bc43814500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe82f400640 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc43814500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe82f400640 1 -- 172.21.15.116:0/816760219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc43814af0 con 0x55bc43803a10 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe82fe00640 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc437fef80 0x55bc43813000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe82f400640 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc43813540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe81c000c00 tx=0x7fe81c009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b7888713ec49f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe819600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe819600640 1 -- 172.21.15.116:0/816760219 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c028040 con 0x55bc43803a10 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe819600640 1 -- 172.21.15.116:0/816760219 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe81c005040 con 0x55bc43803a10 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.917+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc43770c00 con 0x55bc43803a10 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe819600640 1 -- 172.21.15.116:0/816760219 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe81c0037f0 con 0x55bc43803a10 2024-03-04T07:41:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 10 client.15393 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bc43875620 con 0x55bc43803a10 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe819600640 1 -- 172.21.15.116:0/816760219 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bc43770c00 con 0x55bc43803a10 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe819600640 1 -- 172.21.15.116:0/816760219 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe81c041030 con 0x55bc43803a10 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 10 client.15393 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 10 client.15393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 20 client.15393 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43809570 0x55bc43877f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 4 client.15393 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.918+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bc436ef5d0 con 0x55bc43809570 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.919+0000 7fe82fe00640 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43809570 0x55bc43877f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.919+0000 7fe82fe00640 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43809570 0x55bc43877f70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14c3d7b3b7fdccfc server_cookie=b4d846ef72d3d66c in_seq=0 out_seq=0 2024-03-04T07:41:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.919+0000 7fe819600640 10 client.15393 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.922+0000 7fe819600640 1 -- 172.21.15.116:0/816760219 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bc436ef5d0 con 0x55bc43809570 2024-03-04T07:41:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.922+0000 7fe819600640 10 client.15393 handle_command_reply: tid=0 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.922+0000 7fe8397ba740 1 client.15393 shutdown 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 msgr2=0x55bc43813540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc43813540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe81c000c00 tx=0x7fe81c009660 comp rx=0 tx=0).stop 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 shutdown_connections 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bc43809570 0x55bc43877f70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bc43804400 0x55bc43814500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc43803a10 0x55bc43813540 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 --2- 172.21.15.116:0/816760219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bc437fef80 0x55bc43813000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 >> 172.21.15.116:0/816760219 conn(0x55bc4381c080 msgr2=0x55bc438021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe819600640 0 client.15393 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 shutdown_connections 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 1 -- 172.21.15.116:0/816760219 wait complete. 2024-03-04T07:41:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.923+0000 7fe8397ba740 20 client.15393 trim_cache size 0 max 16384 2024-03-04T07:41:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:41:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:41:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:41:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe8140a1e70 msgr2=0x7fe8140a4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe8140a1e70 0x7fe8140a4330 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55bc43804c60 tx=0x7fe828009500 comp rx=0 tx=0).stop 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830074030 msgr2=0x7fe83019dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830074030 0x7fe83019dd40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe820000c00 tx=0x7fe82002c630 comp rx=0 tx=0).stop 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 shutdown_connections 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe8140a1e70 0x7fe8140a4330 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe83010da10 0x7fe8301a52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe830075610 0x7fe83019e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 --2- 172.21.15.116:0/2785455920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe830074030 0x7fe83019dd40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 >> 172.21.15.116:0/2785455920 conn(0x7fe8300fc9b0 msgr2=0x7fe830100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 shutdown_connections 2024-03-04T07:41:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:37.924+0000 7fe835400640 1 -- 172.21.15.116:0/2785455920 wait complete. 2024-03-04T07:41:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:37 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:41:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:37 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:37 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:41:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:37 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:37 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:41:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:37 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:41:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:37 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:37 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:41:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:37 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:37 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:41:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:37 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:41:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:37 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:37 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:41:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:37 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:41:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:37 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:41:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:38 smithi133 ceph-mon[36505]: pgmap v427: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:38 smithi116 ceph-mon[32150]: pgmap v427: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:38 smithi152 ceph-mon[37154]: pgmap v427: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-04T07:41:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:40 smithi133 ceph-mon[36505]: pgmap v428: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-04T07:41:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:40 smithi116 ceph-mon[32150]: pgmap v428: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-04T07:41:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:40 smithi152 ceph-mon[37154]: pgmap v428: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-03-04T07:41:43.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:42 smithi133 ceph-mon[36505]: pgmap v429: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-04T07:41:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:42 smithi116 ceph-mon[32150]: pgmap v429: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-04T07:41:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:42 smithi152 ceph-mon[37154]: pgmap v429: 97 pgs: 97 active+clean; 663 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-04T07:41:45.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:44 smithi133 ceph-mon[36505]: pgmap v430: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:44 smithi116 ceph-mon[32150]: pgmap v430: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:44 smithi152 ceph-mon[37154]: pgmap v430: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:46 smithi133 ceph-mon[36505]: pgmap v431: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:46 smithi116 ceph-mon[32150]: pgmap v431: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:46 smithi152 ceph-mon[37154]: pgmap v431: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:48 smithi133 ceph-mon[36505]: pgmap v432: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:48 smithi116 ceph-mon[32150]: pgmap v432: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:48 smithi152 ceph-mon[37154]: pgmap v432: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 -- 172.21.15.116:0/3550978191 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c1053e0 msgr2=0x7fb14c10bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 --2- 172.21.15.116:0/3550978191 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c1053e0 0x7fb14c10bc70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb144009be0 tx=0x7fb14402bef0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 -- 172.21.15.116:0/3550978191 shutdown_connections 2024-03-04T07:41:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 --2- 172.21.15.116:0/3550978191 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c1053e0 0x7fb14c10bc70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 --2- 172.21.15.116:0/3550978191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb14c104a20 0x7fb14c104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 --2- 172.21.15.116:0/3550978191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb14c103830 0x7fb14c103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 -- 172.21.15.116:0/3550978191 >> 172.21.15.116:0/3550978191 conn(0x7fb14c0fedc0 msgr2=0x7fb14c1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 -- 172.21.15.116:0/3550978191 shutdown_connections 2024-03-04T07:41:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.057+0000 7fb150c00640 1 -- 172.21.15.116:0/3550978191 wait complete. 2024-03-04T07:41:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 Processor -- start 2024-03-04T07:41:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 -- start start 2024-03-04T07:41:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb14c103830 0x7fb14c199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c104a20 0x7fb14c199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb14c1053e0 0x7fb14c1a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c10f4b0 con 0x7fb14c1053e0 2024-03-04T07:41:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c10f330 con 0x7fb14c104a20 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.058+0000 7fb150c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c10f630 con 0x7fb14c103830 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c104a20 0x7fb14c199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c104a20 0x7fb14c199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58228/0 (socket says 172.21.15.116:58228) 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 -- 172.21.15.116:0/1683651686 learned_addr learned my addr 172.21.15.116:0/1683651686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb14a800640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb14c1053e0 0x7fb14c1a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149e00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb14c103830 0x7fb14c199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 -- 172.21.15.116:0/1683651686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb14c103830 msgr2=0x7fb14c199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb14c103830 0x7fb14c199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 -- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb14c1053e0 msgr2=0x7fb14c1a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb14c1053e0 0x7fb14c1a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 -- 172.21.15.116:0/1683651686 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c1a1660 con 0x7fb14c104a20 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb149400640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c104a20 0x7fb14c199f80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb13800e9f0 tx=0x7fb13800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.059+0000 7fb142a00640 1 -- 172.21.15.116:0/1683651686 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13800cd80 con 0x7fb14c104a20 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.060+0000 7fb142a00640 1 -- 172.21.15.116:0/1683651686 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb138005500 con 0x7fb14c104a20 2024-03-04T07:41:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.060+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14c1a1950 con 0x7fb14c104a20 2024-03-04T07:41:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.060+0000 7fb142a00640 1 -- 172.21.15.116:0/1683651686 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138010690 con 0x7fb14c104a20 2024-03-04T07:41:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.060+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14c1a1ea0 con 0x7fb14c104a20 2024-03-04T07:41:51.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.062+0000 7fb142a00640 1 -- 172.21.15.116:0/1683651686 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb1380046c0 con 0x7fb14c104a20 2024-03-04T07:41:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 Processor -- start 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb142a00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1300a1ea0 0x7fb1300a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 -- start start 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb142a00640 1 -- 172.21.15.116:0/1683651686 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb1380ce200 con 0x7fb14c104a20 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d5426f240 0x557d54273d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb14a800640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1300a1ea0 0x7fb1300a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d542743b0 0x557d54276840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d541d4370 con 0x557d542743b0 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d5417d550 con 0x557d5426e850 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb15381d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d541d0410 con 0x557d5426f240 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb149e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb149e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58238/0 (socket says 172.21.15.116:58238) 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb149e00640 1 -- 172.21.15.116:0/106454928 learned_addr learned my addr 172.21.15.116:0/106454928 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb14a800640 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d542743b0 0x557d54276840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.064+0000 7fb149400640 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d5426f240 0x557d54273d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb149e00640 1 -- 172.21.15.116:0/106454928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d5426f240 msgr2=0x557d54273d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb149e00640 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d5426f240 0x557d54273d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb149e00640 1 -- 172.21.15.116:0/106454928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d542743b0 msgr2=0x557d54276840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb149e00640 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d542743b0 0x557d54276840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb149e00640 1 -- 172.21.15.116:0/106454928 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d541913c0 con 0x557d5426e850 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb149e00640 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426ec70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009cf0 tx=0x7fb13c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa7430f2bac691a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb14a800640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1300a1ea0 0x7fb1300a4360 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb144005f00 tx=0x7fb1440098d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb137000640 1 -- 172.21.15.116:0/106454928 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c012450 con 0x557d5426e850 2024-03-04T07:41:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb137000640 1 -- 172.21.15.116:0/106454928 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb13c012ad0 con 0x557d5426e850 2024-03-04T07:41:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.065+0000 7fb137000640 1 -- 172.21.15.116:0/106454928 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c00fb20 con 0x557d5426e850 2024-03-04T07:41:51.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 -- 172.21.15.116:0/106454928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 msgr2=0x557d5426ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426ec70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009cf0 tx=0x7fb13c0117b0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 -- 172.21.15.116:0/106454928 shutdown_connections 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d542743b0 0x557d54276840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d5426f240 0x557d54273d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 --2- 172.21.15.116:0/106454928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426ec70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 -- 172.21.15.116:0/106454928 >> 172.21.15.116:0/106454928 conn(0x557d54269dc0 msgr2=0x557d5426a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 -- 172.21.15.116:0/106454928 shutdown_connections 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.066+0000 7fb15381d740 1 -- 172.21.15.116:0/106454928 wait complete. 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.067+0000 7fb15381d740 1 -- start start 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.067+0000 7fb15381d740 1 Processor -- start 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb15381d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426e850 0x557d54278aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb15381d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426f240 0x557d54278fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb15381d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d54280020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb15381d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d54277360 con 0x557d5426e850 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb15381d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d541909b0 con 0x557d5426f240 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb15381d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d542774e0 con 0x557d542743b0 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb15381d740 10 client.0 _ll_register_callbacks cb 0x557d541cc340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426f240 0x557d54278fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426e850 0x557d54278aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426f240 0x557d54278fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58248/0 (socket says 172.21.15.116:58248) 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 -- 172.21.15.116:0/3515553425 learned_addr learned my addr 172.21.15.116:0/3515553425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb14a800640 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d54280020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 -- 172.21.15.116:0/3515553425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 msgr2=0x557d54280020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d54280020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 -- 172.21.15.116:0/3515553425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426e850 msgr2=0x557d54278aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426e850 0x557d54278aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.068+0000 7fb149400640 1 -- 172.21.15.116:0/3515553425 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d53ee9210 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb149400640 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426f240 0x557d54278fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb13809ae80 tx=0x7fb13809b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce3697c00b728183 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb12b400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb12b400640 1 -- 172.21.15.116:0/3515553425 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1380a3690 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb12b400640 1 -- 172.21.15.116:0/3515553425 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb1380a3200 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb13c012ad0 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb12b400640 1 -- 172.21.15.116:0/3515553425 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1380a3400 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb15381d740 10 client.25517 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557d53fb5620 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb12b400640 1 -- 172.21.15.116:0/3515553425 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb13c012ad0 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb12b400640 1 -- 172.21.15.116:0/3515553425 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb138099f40 con 0x557d5426f240 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb15381d740 10 client.25517 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb15381d740 10 client.25517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb15381d740 20 client.25517 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:51.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.069+0000 7fb15381d740 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542848a0 0x557d54286cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.070+0000 7fb15381d740 4 client.25517 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:41:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.070+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d541e18b0 con 0x557d542848a0 2024-03-04T07:41:51.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.070+0000 7fb149e00640 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542848a0 0x557d54286cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.070+0000 7fb149e00640 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542848a0 0x557d54286cd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=873c6d19834dec22 server_cookie=abb5f504befcde1f in_seq=0 out_seq=0 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.070+0000 7fb12b400640 10 client.25517 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb12b400640 1 -- 172.21.15.116:0/3515553425 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557d541e18b0 con 0x557d542848a0 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb12b400640 10 client.25517 handle_command_reply: tid=0 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 client.25517 shutdown 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426f240 msgr2=0x557d54278fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426f240 0x557d54278fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb13809ae80 tx=0x7fb13809b730 comp rx=0 tx=0).stop 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 shutdown_connections 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542848a0 0x557d54286cd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d54280020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426f240 0x557d54278fe0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb12b400640 0 client.25517 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 --2- 172.21.15.116:0/3515553425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426e850 0x557d54278aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 >> 172.21.15.116:0/3515553425 conn(0x557d54269dc0 msgr2=0x557d54268fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 shutdown_connections 2024-03-04T07:41:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 1 -- 172.21.15.116:0/3515553425 wait complete. 2024-03-04T07:41:51.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.071+0000 7fb15381d740 20 client.25517 trim_cache size 0 max 16384 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 Processor -- start 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 -- start start 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5426d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1a8c80 con 0x557d5426f240 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1ab260 con 0x557d5426e850 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb15381d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1a9280 con 0x557d542743b0 2024-03-04T07:41:51.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb149e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.086+0000 7fb149e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58262/0 (socket says 172.21.15.116:58262) 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149e00640 1 -- 172.21.15.116:0/2859019974 learned_addr learned my addr 172.21.15.116:0/2859019974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149400640 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb14a800640 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5426d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149e00640 1 -- 172.21.15.116:0/2859019974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 msgr2=0x557d5426d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149e00640 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5426d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149e00640 1 -- 172.21.15.116:0/2859019974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 msgr2=0x557d5427ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149e00640 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149e00640 1 -- 172.21.15.116:0/2859019974 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d5426f620 con 0x557d5426e850 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb149e00640 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426d020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009c40 tx=0x7fb13c011da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fedcbd1a209710b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb136600640 1 -- 172.21.15.116:0/2859019974 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c036070 con 0x557d5426e850 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb136600640 1 -- 172.21.15.116:0/2859019974 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb13c012930 con 0x557d5426e850 2024-03-04T07:41:51.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.087+0000 7fb136600640 1 -- 172.21.15.116:0/2859019974 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c005320 con 0x557d5426e850 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 -- 172.21.15.116:0/2859019974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 msgr2=0x557d5426d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426d020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009c40 tx=0x7fb13c011da0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 -- 172.21.15.116:0/2859019974 shutdown_connections 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5426d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 --2- 172.21.15.116:0/2859019974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5426d020 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 -- 172.21.15.116:0/2859019974 >> 172.21.15.116:0/2859019974 conn(0x557d54269dc0 msgr2=0x557d54278aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.088+0000 7fb15381d740 1 -- 172.21.15.116:0/2859019974 shutdown_connections 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.089+0000 7fb15381d740 1 -- 172.21.15.116:0/2859019974 wait complete. 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.089+0000 7fb15381d740 1 -- start start 2024-03-04T07:41:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.089+0000 7fb15381d740 1 Processor -- start 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb15381d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5427dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb15381d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb15381d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5427f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb15381d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1a98e0 con 0x557d5426f240 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb15381d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1aac00 con 0x557d5426e850 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb15381d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c1a9f40 con 0x557d542743b0 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb15381d740 10 client.0 _ll_register_callbacks cb 0x7fb13809acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5427dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5427dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58270/0 (socket says 172.21.15.116:58270) 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 -- 172.21.15.116:0/2373173177 learned_addr learned my addr 172.21.15.116:0/2373173177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149400640 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb14a800640 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5427f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 -- 172.21.15.116:0/2373173177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 msgr2=0x557d5427f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5427f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 -- 172.21.15.116:0/2373173177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 msgr2=0x557d5427e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 -- 172.21.15.116:0/2373173177 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d5427f6d0 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb149e00640 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5427dc00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb13c011e30 tx=0x7fb13c009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc0ef04f014db885 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.090+0000 7fb12b400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb12b400640 1 -- 172.21.15.116:0/2373173177 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c036070 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb12b400640 1 -- 172.21.15.116:0/2373173177 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb13c012450 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb13c012930 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb12b400640 1 -- 172.21.15.116:0/2373173177 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb13c042050 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb15381d740 10 client.25523 fetch_fsmap learned FSMap version 19 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557d542e0630 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb12b400640 1 -- 172.21.15.116:0/2373173177 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb13c012930 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.091+0000 7fb12b400640 1 -- 172.21.15.116:0/2373173177 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb13c003a80 con 0x557d5426e850 2024-03-04T07:41:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb15381d740 10 client.25523 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb15381d740 10 client.25523 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb15381d740 20 client.25523 populate_metadata read hostname 'smithi116' 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb15381d740 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542e0d60 0x557d542e3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb15381d740 4 client.25523 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557d541dbc00 con 0x557d542e0d60 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb149400640 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542e0d60 0x557d542e3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb149400640 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542e0d60 0x557d542e3180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83a59503c920b37 server_cookie=68b0c7fcb971b81e in_seq=0 out_seq=0 2024-03-04T07:41:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.092+0000 7fb12b400640 10 client.25523 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.097+0000 7fb12b400640 1 -- 172.21.15.116:0/2373173177 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557d541dbc00 con 0x557d542e0d60 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.097+0000 7fb12b400640 10 client.25523 handle_command_reply: tid=0 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.097+0000 7fb15381d740 1 client.25523 shutdown 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 msgr2=0x557d5427dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5427dc00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb13c011e30 tx=0x7fb13c009600 comp rx=0 tx=0).stop 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 shutdown_connections 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557d542e0d60 0x557d542e3180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557d542743b0 0x557d5427f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5426f240 0x557d5427e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb12b400640 0 client.25523 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 --2- 172.21.15.116:0/2373173177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557d5426e850 0x557d5427dc00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 >> 172.21.15.116:0/2373173177 conn(0x557d54269dc0 msgr2=0x557d54278a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 shutdown_connections 2024-03-04T07:41:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 1 -- 172.21.15.116:0/2373173177 wait complete. 2024-03-04T07:41:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.098+0000 7fb15381d740 20 client.25523 trim_cache size 0 max 16384 2024-03-04T07:41:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:41:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:41:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:41:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1300a1ea0 msgr2=0x7fb1300a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1300a1ea0 0x7fb1300a4360 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb144005f00 tx=0x7fb1440098d0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c104a20 msgr2=0x7fb14c199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c104a20 0x7fb14c199f80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb13800e9f0 tx=0x7fb13800edf0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 shutdown_connections 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb14c1053e0 0x7fb14c1a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb14c104a20 0x7fb14c199f80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1300a1ea0 0x7fb1300a4360 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 --2- 172.21.15.116:0/1683651686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb14c103830 0x7fb14c199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:41:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 >> 172.21.15.116:0/1683651686 conn(0x7fb14c0fedc0 msgr2=0x7fb14c107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:41:51.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 shutdown_connections 2024-03-04T07:41:51.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:41:51.099+0000 7fb150c00640 1 -- 172.21.15.116:0/1683651686 wait complete. 2024-03-04T07:41:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:50 smithi116 ceph-mon[32150]: pgmap v433: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:50 smithi152 ceph-mon[37154]: pgmap v433: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:51.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:50 smithi133 ceph-mon[36505]: pgmap v433: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:52 smithi116 ceph-mon[32150]: pgmap v434: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:52 smithi152 ceph-mon[37154]: pgmap v434: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:53.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:52 smithi133 ceph-mon[36505]: pgmap v434: 97 pgs: 97 active+clean; 664 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-04T07:41:53.671 INFO:tasks.workunit.client.0.smithi116.stderr:Updating files: 63% (7849/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-04T07:41:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'. 2024-03-04T07:41:54.241 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T07:41:54.242 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-03-04T07:41:54.243 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T07:41:54.243 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-04T07:41:54.243 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-04T07:41:54.243 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at 1bc250c7129 mds/quiesce-db: incorporate review comments 2024-03-04T07:41:54.246 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-04T07:41:54.351 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-04T07:41:54.358 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-04T07:41:54.358 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-04T07:41:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:54 smithi116 ceph-mon[32150]: pgmap v435: 97 pgs: 97 active+clean; 665 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-04T07:41:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:54 smithi152 ceph-mon[37154]: pgmap v435: 97 pgs: 97 active+clean; 665 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-04T07:41:55.313 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-04T07:41:55.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:54 smithi133 ceph-mon[36505]: pgmap v435: 97 pgs: 97 active+clean; 665 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-04T07:41:55.430 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-04T07:41:55.519 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-04T07:41:55.524 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-04T07:41:55.524 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-04T07:41:55.605 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-04T07:41:55.614 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:41:55.614 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-04T07:41:55.664 INFO:tasks.workunit:Running workunits matching fs/test_o_trunc.sh on client.0... 2024-03-04T07:41:55.666 INFO:tasks.workunit:Running workunit fs/test_o_trunc.sh... 2024-03-04T07:41:55.666 DEBUG:teuthology.orchestra.run.smithi116:workunit test fs/test_o_trunc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-03-04T07:41:55.781 INFO:tasks.workunit.client.0.smithi116.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-03-04T07:41:55.784 INFO:tasks.workunit.client.0.smithi116.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs 2024-03-04T07:41:55.784 INFO:tasks.workunit.client.0.smithi116.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc trunc.foo 600 2024-03-04T07:41:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:56 smithi116 ceph-mon[32150]: pgmap v436: 97 pgs: 97 active+clean; 665 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-03-04T07:41:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:56 smithi152 ceph-mon[37154]: pgmap v436: 97 pgs: 97 active+clean; 665 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-03-04T07:41:57.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:56 smithi133 ceph-mon[36505]: pgmap v436: 97 pgs: 97 active+clean; 665 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-03-04T07:41:58.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:41:57 smithi116 ceph-mon[32150]: pgmap v437: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-04T07:41:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:41:57 smithi152 ceph-mon[37154]: pgmap v437: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-04T07:41:58.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:41:57 smithi133 ceph-mon[36505]: pgmap v437: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-04T07:42:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:00 smithi116 ceph-mon[32150]: pgmap v438: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-04T07:42:01.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:00 smithi152 ceph-mon[37154]: pgmap v438: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-04T07:42:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:00 smithi133 ceph-mon[36505]: pgmap v438: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-04T07:42:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:02 smithi116 ceph-mon[32150]: pgmap v439: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-03-04T07:42:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:02 smithi152 ceph-mon[37154]: pgmap v439: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-03-04T07:42:03.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:02 smithi133 ceph-mon[36505]: pgmap v439: 97 pgs: 97 active+clean; 685 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.237+0000 7f1635200640 1 -- 172.21.15.116:0/540797675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16301007d0 msgr2=0x7f1630100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.237+0000 7f1635200640 1 --2- 172.21.15.116:0/540797675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16301007d0 0x7f1630100c30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f161c009900 tx=0x7f161c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.238+0000 7f1635200640 1 -- 172.21.15.116:0/540797675 shutdown_connections 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.238+0000 7f1635200640 1 --2- 172.21.15.116:0/540797675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630101170 0x7f16301015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.238+0000 7f1635200640 1 --2- 172.21.15.116:0/540797675 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16301007d0 0x7f1630100c30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.238+0000 7f1635200640 1 --2- 172.21.15.116:0/540797675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16300ffe70 0x7f1630100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.238+0000 7f1635200640 1 -- 172.21.15.116:0/540797675 >> 172.21.15.116:0/540797675 conn(0x7f16300fba90 msgr2=0x7f16300fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.238+0000 7f1635200640 1 -- 172.21.15.116:0/540797675 shutdown_connections 2024-03-04T07:42:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.238+0000 7f1635200640 1 -- 172.21.15.116:0/540797675 wait complete. 2024-03-04T07:42:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 Processor -- start 2024-03-04T07:42:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 -- start start 2024-03-04T07:42:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16300ffe70 0x7f1630199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16301007d0 0x7f1630199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630101170 0x7f16301a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163010f440 con 0x7f1630101170 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163010f2c0 con 0x7f16300ffe70 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.239+0000 7f1635200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163010f5c0 con 0x7f16301007d0 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16300ffe70 0x7f1630199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16300ffe70 0x7f1630199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55738/0 (socket says 172.21.15.116:55738) 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 -- 172.21.15.116:0/1369730951 learned_addr learned my addr 172.21.15.116:0/1369730951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630101170 0x7f16301a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162d400640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16301007d0 0x7f1630199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 -- 172.21.15.116:0/1369730951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16301007d0 msgr2=0x7f1630199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16301007d0 0x7f1630199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 -- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630101170 msgr2=0x7f16301a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630101170 0x7f16301a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 -- 172.21.15.116:0/1369730951 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16301a1640 con 0x7f16300ffe70 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.240+0000 7f162de00640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16300ffe70 0x7f1630199a20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f162000d950 tx=0x7f162000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.241+0000 7f1626a00640 1 -- 172.21.15.116:0/1369730951 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620017070 con 0x7f16300ffe70 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.241+0000 7f1626a00640 1 -- 172.21.15.116:0/1369730951 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1620009c30 con 0x7f16300ffe70 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.241+0000 7f1626a00640 1 -- 172.21.15.116:0/1369730951 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162000f6b0 con 0x7f16300ffe70 2024-03-04T07:42:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.241+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16301a1930 con 0x7f16300ffe70 2024-03-04T07:42:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.241+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16301a1ea0 con 0x7f16300ffe70 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.244+0000 7f1637f85740 1 Processor -- start 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.244+0000 7f1626a00640 1 -- 172.21.15.116:0/1369730951 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f16200109d0 con 0x7f16300ffe70 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.244+0000 7f1637f85740 1 -- start start 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f1637f85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f1637f85740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eac550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f1637f85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eaf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f1637f85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623c3e04370 con 0x5623c3eacbc0 2024-03-04T07:42:04.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f1637f85740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623c3dad550 con 0x5623c3ea7a50 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f1637f85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623c3e00410 con 0x5623c3ea7060 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eaf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:49556/0 (socket says 172.21.15.116:49556) 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 -- 172.21.15.116:0/1198486764 learned_addr learned my addr 172.21.15.116:0/1198486764 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 -- 172.21.15.116:0/1198486764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 msgr2=0x5623c3eac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 --2- 172.21.15.116:0/1198486764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eac550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 -- 172.21.15.116:0/1198486764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 msgr2=0x5623c3eaf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 --2- 172.21.15.116:0/1198486764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eaf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.245+0000 7f162e800640 1 -- 172.21.15.116:0/1198486764 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623c3dc13c0 con 0x5623c3ea7060 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.246+0000 7f162e800640 1 --2- 172.21.15.116:0/1198486764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea7480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16280039f0 tx=0x7f1628003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b82b775d7c1177 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.246+0000 7f1626a00640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f16140a1ea0 0x7f16140a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.246+0000 7f161b400640 1 -- 172.21.15.116:0/1198486764 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162801a470 con 0x5623c3ea7060 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.246+0000 7f161b400640 1 -- 172.21.15.116:0/1198486764 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1628015070 con 0x5623c3ea7060 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.246+0000 7f161b400640 1 -- 172.21.15.116:0/1198486764 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628010940 con 0x5623c3ea7060 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.246+0000 7f1626a00640 1 -- 172.21.15.116:0/1369730951 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f16200cd8e0 con 0x7f16300ffe70 2024-03-04T07:42:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.246+0000 7f162de00640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f16140a1ea0 0x7f16140a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 -- 172.21.15.116:0/1198486764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 msgr2=0x5623c3ea7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 --2- 172.21.15.116:0/1198486764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea7480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16280039f0 tx=0x7f1628003eb0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f162de00640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f16140a1ea0 0x7f16140a4360 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f16200a2840 tx=0x7f16200d3530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 -- 172.21.15.116:0/1198486764 shutdown_connections 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 --2- 172.21.15.116:0/1198486764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eaf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 --2- 172.21.15.116:0/1198486764 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 --2- 172.21.15.116:0/1198486764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea7480 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 -- 172.21.15.116:0/1198486764 >> 172.21.15.116:0/1198486764 conn(0x5623c3ea25d0 msgr2=0x5623c3ea29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 -- 172.21.15.116:0/1198486764 shutdown_connections 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.247+0000 7f1637f85740 1 -- 172.21.15.116:0/1198486764 wait complete. 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 -- start start 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 Processor -- start 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3eb1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eb8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623c3eafb70 con 0x5623c3eacbc0 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623c3dc09b0 con 0x5623c3ea7a50 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623c3eafcf0 con 0x5623c3ea7060 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f1637f85740 10 client.0 _ll_register_callbacks cb 0x5623c3dfc340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.248+0000 7f162d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55746/0 (socket says 172.21.15.116:55746) 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 -- 172.21.15.116:0/1292698758 learned_addr learned my addr 172.21.15.116:0/1292698758 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162e800640 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3eb1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162e800640 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eb8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 -- 172.21.15.116:0/1292698758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 msgr2=0x5623c3eb1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3eb1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 -- 172.21.15.116:0/1292698758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 msgr2=0x5623c3eb8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eb8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 -- 172.21.15.116:0/1292698758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623c3b19210 con 0x5623c3ea7a50 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f162d400640 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb19a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f161c009900 tx=0x7f161c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d80c7296523f911b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f160fe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f160fe00640 1 -- 172.21.15.116:0/1292698758 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c005d50 con 0x5623c3ea7a50 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1628015070 con 0x5623c3ea7a50 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.249+0000 7f160fe00640 1 -- 172.21.15.116:0/1292698758 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f161c005230 con 0x5623c3ea7a50 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f160fe00640 1 -- 172.21.15.116:0/1292698758 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f161c005470 con 0x5623c3ea7a50 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 10 client.25535 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5623c3be5620 con 0x5623c3ea7a50 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f160fe00640 1 -- 172.21.15.116:0/1292698758 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1628015070 con 0x5623c3ea7a50 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f160fe00640 1 -- 172.21.15.116:0/1292698758 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f161c003b50 con 0x5623c3ea7a50 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 10 client.25535 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 10 client.25535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 20 client.25535 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3ebd1a0 0x5623c3ebf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 4 client.25535 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.250+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5623c3e118b0 con 0x5623c3ebd1a0 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.251+0000 7f162e800640 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3ebd1a0 0x5623c3ebf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.251+0000 7f162e800640 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3ebd1a0 0x5623c3ebf590 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3ec603c57953aad server_cookie=2a9cab427ced37bb in_seq=0 out_seq=0 2024-03-04T07:42:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.251+0000 7f160fe00640 10 client.25535 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f160fe00640 1 -- 172.21.15.116:0/1292698758 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5623c3e118b0 con 0x5623c3ebd1a0 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f160fe00640 10 client.25535 handle_command_reply: tid=0 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 client.25535 shutdown 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 msgr2=0x5623c3eb19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb19a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f161c009900 tx=0x7f161c004270 comp rx=0 tx=0).stop 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 shutdown_connections 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3ebd1a0 0x5623c3ebf590 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3eb8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb19a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 --2- 172.21.15.116:0/1292698758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3eb1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f160fe00640 0 client.25535 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.252+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 >> 172.21.15.116:0/1292698758 conn(0x5623c3ea25d0 msgr2=0x5623c3e98fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.253+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 shutdown_connections 2024-03-04T07:42:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.253+0000 7f1637f85740 1 -- 172.21.15.116:0/1292698758 wait complete. 2024-03-04T07:42:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.253+0000 7f1637f85740 20 client.25535 trim_cache size 0 max 16384 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.267+0000 7f1637f85740 1 Processor -- start 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.267+0000 7f1637f85740 1 -- start start 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f1637f85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f1637f85740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f1637f85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3ea5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f1637f85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301a4800 con 0x5623c3eacbc0 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f1637f85740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301a6de0 con 0x5623c3ea7a50 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f1637f85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301a4e00 con 0x5623c3ea7060 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55748/0 (socket says 172.21.15.116:55748) 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 -- 172.21.15.116:0/2064235965 learned_addr learned my addr 172.21.15.116:0/2064235965 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162e800640 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3ea5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162e800640 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 -- 172.21.15.116:0/2064235965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 msgr2=0x5623c3ea5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 -- 172.21.15.116:0/2064235965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 msgr2=0x5623c3ea5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3ea5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162d400640 1 -- 172.21.15.116:0/2064235965 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623c3ea82e0 con 0x5623c3ea7a50 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.268+0000 7f162e800640 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f162d400640 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb88c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f161c000c00 tx=0x7f161c009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44453f682b77d8b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f161aa00640 1 -- 172.21.15.116:0/2064235965 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c005d50 con 0x5623c3ea7a50 2024-03-04T07:42:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f161aa00640 1 -- 172.21.15.116:0/2064235965 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f161c005610 con 0x5623c3ea7a50 2024-03-04T07:42:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f161aa00640 1 -- 172.21.15.116:0/2064235965 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c0048d0 con 0x5623c3ea7a50 2024-03-04T07:42:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f1637f85740 1 -- 172.21.15.116:0/2064235965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 msgr2=0x5623c3eb88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f1637f85740 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb88c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f161c000c00 tx=0x7f161c009650 comp rx=0 tx=0).stop 2024-03-04T07:42:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f1637f85740 1 -- 172.21.15.116:0/2064235965 shutdown_connections 2024-03-04T07:42:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f1637f85740 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3eacbc0 0x5623c3ea5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f1637f85740 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb88c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.269+0000 7f1637f85740 1 --2- 172.21.15.116:0/2064235965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3ea7060 0x5623c3ea5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.270+0000 7f1637f85740 1 -- 172.21.15.116:0/2064235965 >> 172.21.15.116:0/2064235965 conn(0x5623c3ea25d0 msgr2=0x5623c3ea8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.270+0000 7f1637f85740 1 -- 172.21.15.116:0/2064235965 shutdown_connections 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.270+0000 7f1637f85740 1 -- 172.21.15.116:0/2064235965 wait complete. 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.270+0000 7f1637f85740 1 -- start start 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.270+0000 7f1637f85740 1 Processor -- start 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.270+0000 7f1637f85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3ea7060 0x5623c3eb65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f1637f85740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f1637f85740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3eacbc0 0x5623c3eb7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f1637f85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301a5460 con 0x5623c3ea7060 2024-03-04T07:42:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f1637f85740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301a6780 con 0x5623c3ea7a50 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f1637f85740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301a5ac0 con 0x5623c3eacbc0 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f1637f85740 10 client.0 _ll_register_callbacks cb 0x7f161c02c5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3ea7060 0x5623c3eb65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3ea7060 0x5623c3eb65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43794/0 (socket says 172.21.15.116:43794) 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162e800640 1 -- 172.21.15.116:0/329819918 learned_addr learned my addr 172.21.15.116:0/329819918 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162e800640 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3eacbc0 0x5623c3eb7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162d400640 1 -- 172.21.15.116:0/329819918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3eacbc0 msgr2=0x5623c3eb7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162d400640 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3eacbc0 0x5623c3eb7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162d400640 1 -- 172.21.15.116:0/329819918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3ea7060 msgr2=0x5623c3eb65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162d400640 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3ea7060 0x5623c3eb65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162d400640 1 -- 172.21.15.116:0/329819918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623c3eb8090 con 0x5623c3ea7a50 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.271+0000 7f162e800640 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3eacbc0 0x5623c3eb7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f162d400640 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb6ae0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5623c3ea35f0 tx=0x7f161c0465e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1889c99b9a2942a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f160fe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f160fe00640 1 -- 172.21.15.116:0/329819918 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c05f070 con 0x5623c3ea7a50 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f160fe00640 1 -- 172.21.15.116:0/329819918 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f161c0048d0 con 0x5623c3ea7a50 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f161c005610 con 0x5623c3ea7a50 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f160fe00640 1 -- 172.21.15.116:0/329819918 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f161c004250 con 0x5623c3ea7a50 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f1637f85740 10 client.25541 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5623c3f18f70 con 0x5623c3ea7a50 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f160fe00640 1 -- 172.21.15.116:0/329819918 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f161c005610 con 0x5623c3ea7a50 2024-03-04T07:42:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.272+0000 7f160fe00640 1 -- 172.21.15.116:0/329819918 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f161c062020 con 0x5623c3ea7a50 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f1637f85740 10 client.25541 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f1637f85740 10 client.25541 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f1637f85740 20 client.25541 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f1637f85740 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3f19800 0x5623c3f1bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f1637f85740 4 client.25541 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5623c3e0bc00 con 0x5623c3f19800 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f162e800640 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3f19800 0x5623c3f1bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f162e800640 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3f19800 0x5623c3f1bc20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c71e1d83f7ed9fa server_cookie=3f51ac3018b95bc5 in_seq=0 out_seq=0 2024-03-04T07:42:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.273+0000 7f160fe00640 10 client.25541 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f160fe00640 1 -- 172.21.15.116:0/329819918 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5623c3e0bc00 con 0x5623c3f19800 2024-03-04T07:42:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f160fe00640 10 client.25541 handle_command_reply: tid=0 2024-03-04T07:42:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 client.25541 shutdown 2024-03-04T07:42:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 msgr2=0x5623c3eb6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb6ae0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5623c3ea35f0 tx=0x7f161c0465e0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 shutdown_connections 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5623c3f19800 0x5623c3f1bc20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623c3eacbc0 0x5623c3eb7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5623c3ea7a50 0x5623c3eb6ae0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 --2- 172.21.15.116:0/329819918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5623c3ea7060 0x5623c3eb65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f160fe00640 0 client.25541 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 >> 172.21.15.116:0/329819918 conn(0x5623c3ea25d0 msgr2=0x5623c3ea5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 shutdown_connections 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 1 -- 172.21.15.116:0/329819918 wait complete. 2024-03-04T07:42:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.284+0000 7f1637f85740 20 client.25541 trim_cache size 0 max 16384 2024-03-04T07:42:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:42:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:42:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:42:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f16140a1ea0 msgr2=0x7f16140a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f16140a1ea0 0x7f16140a4360 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f16200a2840 tx=0x7f16200d3530 comp rx=0 tx=0).stop 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16300ffe70 msgr2=0x7f1630199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16300ffe70 0x7f1630199a20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f162000d950 tx=0x7f162000de10 comp rx=0 tx=0).stop 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 shutdown_connections 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630101170 0x7f16301a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f16140a1ea0 0x7f16140a4360 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16301007d0 0x7f1630199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 --2- 172.21.15.116:0/1369730951 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16300ffe70 0x7f1630199a20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 >> 172.21.15.116:0/1369730951 conn(0x7f16300fba90 msgr2=0x7f16300fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 shutdown_connections 2024-03-04T07:42:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:04.285+0000 7f1635200640 1 -- 172.21.15.116:0/1369730951 wait complete. 2024-03-04T07:42:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:04 smithi116 ceph-mon[32150]: pgmap v440: 97 pgs: 97 active+clean; 740 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-04T07:42:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:04 smithi152 ceph-mon[37154]: pgmap v440: 97 pgs: 97 active+clean; 740 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-04T07:42:05.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:04 smithi133 ceph-mon[36505]: pgmap v440: 97 pgs: 97 active+clean; 740 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-04T07:42:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:06 smithi116 ceph-mon[32150]: pgmap v441: 97 pgs: 97 active+clean; 740 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-04T07:42:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:06 smithi152 ceph-mon[37154]: pgmap v441: 97 pgs: 97 active+clean; 740 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-04T07:42:07.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:06 smithi133 ceph-mon[36505]: pgmap v441: 97 pgs: 97 active+clean; 740 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-04T07:42:07.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:42:07.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 -- 172.21.15.116:0/1956972925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 msgr2=0x7fa988103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 --2- 172.21.15.116:0/1956972925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 0x7fa988103c10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa978009900 tx=0x7fa97802bc40 comp rx=0 tx=0).stop 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 -- 172.21.15.116:0/1956972925 shutdown_connections 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 --2- 172.21.15.116:0/1956972925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881053a0 0x7fa98810bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 --2- 172.21.15.116:0/1956972925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9881049e0 0x7fa988104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 --2- 172.21.15.116:0/1956972925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 0x7fa988103c10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 -- 172.21.15.116:0/1956972925 >> 172.21.15.116:0/1956972925 conn(0x7fa9880fed80 msgr2=0x7fa9881011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 -- 172.21.15.116:0/1956972925 shutdown_connections 2024-03-04T07:42:07.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.922+0000 7fa98fa00640 1 -- 172.21.15.116:0/1956972925 wait complete. 2024-03-04T07:42:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 Processor -- start 2024-03-04T07:42:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 -- start start 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 0x7fa988199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881049e0 0x7fa98819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9881053a0 0x7fa9881a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98810f4b0 con 0x7fa9881037f0 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98810f330 con 0x7fa9881049e0 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98fa00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98810f630 con 0x7fa9881053a0 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa97fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881049e0 0x7fa98819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa98d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 0x7fa988199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa97fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881049e0 0x7fa98819a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55036/0 (socket says 172.21.15.116:55036) 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.923+0000 7fa97fe00640 1 -- 172.21.15.116:0/325817790 learned_addr learned my addr 172.21.15.116:0/325817790 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa98da00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9881053a0 0x7fa9881a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa97fe00640 1 -- 172.21.15.116:0/325817790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9881053a0 msgr2=0x7fa9881a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa97fe00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9881053a0 0x7fa9881a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa97fe00640 1 -- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 msgr2=0x7fa988199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa97fe00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 0x7fa988199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa97fe00640 1 -- 172.21.15.116:0/325817790 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9881a1710 con 0x7fa9881049e0 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa98da00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9881053a0 0x7fa9881a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa97fe00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881049e0 0x7fa98819a030 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa97000e9f0 tx=0x7fa97000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa986000640 1 -- 172.21.15.116:0/325817790 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97000cd80 con 0x7fa9881049e0 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9881a1a00 con 0x7fa9881049e0 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa986000640 1 -- 172.21.15.116:0/325817790 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa970005500 con 0x7fa9881049e0 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.924+0000 7fa986000640 1 -- 172.21.15.116:0/325817790 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970010690 con 0x7fa9881049e0 2024-03-04T07:42:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.925+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9881a1f00 con 0x7fa9881049e0 2024-03-04T07:42:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.927+0000 7fa986000640 1 -- 172.21.15.116:0/325817790 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa9700046c0 con 0x7fa9881049e0 2024-03-04T07:42:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 Processor -- start 2024-03-04T07:42:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 -- start start 2024-03-04T07:42:07.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cb540 0x55f0001d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001d2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0000f8090 con 0x55f0001cab50 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f00012fd30 con 0x55f0001d06b0 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.928+0000 7fa9927da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0000d8a30 con 0x55f0001cb540 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa98da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001d2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa98d000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cb540 0x55f0001d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001d2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55044/0 (socket says 172.21.15.116:55044) 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 -- 172.21.15.116:0/1758808309 learned_addr learned my addr 172.21.15.116:0/1758808309 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa986000640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa96c0a1d80 0x7fa96c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 -- 172.21.15.116:0/1758808309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cb540 msgr2=0x55f0001d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 --2- 172.21.15.116:0/1758808309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cb540 0x55f0001d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 -- 172.21.15.116:0/1758808309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 msgr2=0x55f0001caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 --2- 172.21.15.116:0/1758808309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa97fe00640 1 -- 172.21.15.116:0/1758808309 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55effff0e7d0 con 0x55f0001d06b0 2024-03-04T07:42:07.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa986000640 1 -- 172.21.15.116:0/325817790 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa9700ce290 con 0x7fa9881049e0 2024-03-04T07:42:07.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.929+0000 7fa98da00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa96c0a1d80 0x7fa96c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.930+0000 7fa97fe00640 1 --2- 172.21.15.116:0/1758808309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001d2b40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa970050d40 tx=0x7fa97005b580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5a0ab55b821505b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:07.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.930+0000 7fa97e000640 1 -- 172.21.15.116:0/1758808309 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97009ad50 con 0x55f0001d06b0 2024-03-04T07:42:07.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.930+0000 7fa97e000640 1 -- 172.21.15.116:0/1758808309 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa970053660 con 0x55f0001d06b0 2024-03-04T07:42:07.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.930+0000 7fa97e000640 1 -- 172.21.15.116:0/1758808309 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970053460 con 0x55f0001d06b0 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.930+0000 7fa9927da740 1 -- 172.21.15.116:0/1758808309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 msgr2=0x55f0001d2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.930+0000 7fa98da00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa96c0a1d80 0x7fa96c0a4240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55f0001cbda0 tx=0x7fa980004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.930+0000 7fa9927da740 1 --2- 172.21.15.116:0/1758808309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001d2b40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa970050d40 tx=0x7fa97005b580 comp rx=0 tx=0).stop 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 -- 172.21.15.116:0/1758808309 shutdown_connections 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 --2- 172.21.15.116:0/1758808309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001d2b40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 --2- 172.21.15.116:0/1758808309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cb540 0x55f0001d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 --2- 172.21.15.116:0/1758808309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 -- 172.21.15.116:0/1758808309 >> 172.21.15.116:0/1758808309 conn(0x55f0001c60c0 msgr2=0x55f0001c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 -- 172.21.15.116:0/1758808309 shutdown_connections 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 -- 172.21.15.116:0/1758808309 wait complete. 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 -- start start 2024-03-04T07:42:07.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.931+0000 7fa9927da740 1 Processor -- start 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa9927da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001d4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa9927da740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cb540 0x55f0001d5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa9927da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001dc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa9927da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0000ce0c0 con 0x55f0001cab50 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa9927da740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f00012bdd0 con 0x55f0001cb540 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa9927da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0001d3600 con 0x55f0001d06b0 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa9927da740 10 client.0 _ll_register_callbacks cb 0x55f0000eb860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa98d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001d4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa98d000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001d4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60808/0 (socket says 172.21.15.116:60808) 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa98d000640 1 -- 172.21.15.116:0/2648125867 learned_addr learned my addr 172.21.15.116:0/2648125867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa97fe00640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cb540 0x55f0001d5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa98da00640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001dc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa97fe00640 1 -- 172.21.15.116:0/2648125867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 msgr2=0x55f0001dc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa97fe00640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001dc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa97fe00640 1 -- 172.21.15.116:0/2648125867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 msgr2=0x55f0001d4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa97fe00640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001d4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.932+0000 7fa97fe00640 1 -- 172.21.15.116:0/2648125867 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efffe443f0 con 0x55f0001cb540 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa98d000640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001d4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa97fe00640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cb540 0x55f0001d5330 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa970085000 tx=0x7fa97009b100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb5161bdca8e4ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa976a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa976a00640 1 -- 172.21.15.116:0/2648125867 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97001c1c0 con 0x55f0001cb540 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa976a00640 1 -- 172.21.15.116:0/2648125867 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa970099280 con 0x55f0001cb540 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f00013d270 con 0x55f0001cb540 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa976a00640 1 -- 172.21.15.116:0/2648125867 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa97009a8e0 con 0x55f0001cb540 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa9927da740 10 client.25553 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55efffe040f0 con 0x55f0001cb540 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.933+0000 7fa976a00640 1 -- 172.21.15.116:0/2648125867 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f00013d270 con 0x55f0001cb540 2024-03-04T07:42:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa976a00640 1 -- 172.21.15.116:0/2648125867 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa970096cd0 con 0x55f0001cb540 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa9927da740 10 client.25553 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa9927da740 10 client.25553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa9927da740 20 client.25553 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa9927da740 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f0001e04d0 0x55f0001e28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa9927da740 4 client.25553 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f0001375c0 con 0x55f0001e04d0 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.934+0000 7fa98d000640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f0001e04d0 0x55f0001e28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.935+0000 7fa98d000640 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f0001e04d0 0x55f0001e28c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e43de2c1a507611 server_cookie=e326d5590869f1dd in_seq=0 out_seq=0 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.935+0000 7fa976a00640 10 client.25553 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.935+0000 7fa976a00640 1 -- 172.21.15.116:0/2648125867 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f0001375c0 con 0x55f0001e04d0 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.935+0000 7fa976a00640 10 client.25553 handle_command_reply: tid=0 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.935+0000 7fa9927da740 1 client.25553 shutdown 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cb540 msgr2=0x55f0001d5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cb540 0x55f0001d5330 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa970085000 tx=0x7fa97009b100 comp rx=0 tx=0).stop 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 shutdown_connections 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f0001e04d0 0x55f0001e28c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001dc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cb540 0x55f0001d5330 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 --2- 172.21.15.116:0/2648125867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cab50 0x55f0001d4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa976a00640 0 client.25553 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 >> 172.21.15.116:0/2648125867 conn(0x55f0001c60c0 msgr2=0x55f0001c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 shutdown_connections 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 1 -- 172.21.15.116:0/2648125867 wait complete. 2024-03-04T07:42:07.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.936+0000 7fa9927da740 20 client.25553 trim_cache size 0 max 16384 2024-03-04T07:42:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 Processor -- start 2024-03-04T07:42:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 -- start start 2024-03-04T07:42:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cab50 0x55f0001cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001dc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881a8c00 con 0x55f0001cb540 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881ab1e0 con 0x55f0001cab50 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.953+0000 7fa9927da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881a9200 con 0x55f0001d06b0 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cab50 0x55f0001cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa97fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001dc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cab50 0x55f0001cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55060/0 (socket says 172.21.15.116:55060) 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 -- 172.21.15.116:0/1413687324 learned_addr learned my addr 172.21.15.116:0/1413687324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:07.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98da00640 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 -- 172.21.15.116:0/1413687324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 msgr2=0x55f0001c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 -- 172.21.15.116:0/1413687324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 msgr2=0x55f0001dc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001dc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 -- 172.21.15.116:0/1413687324 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0001c0770 con 0x55f0001cab50 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa97fe00640 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001dc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa98d000640 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cab50 0x55f0001cbdd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa978009900 tx=0x7fa97802cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9374ed8ff55fbe6c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.954+0000 7fa97d600640 1 -- 172.21.15.116:0/1413687324 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978004240 con 0x55f0001cab50 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa97d600640 1 -- 172.21.15.116:0/1413687324 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9780043e0 con 0x55f0001cab50 2024-03-04T07:42:07.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa97d600640 1 -- 172.21.15.116:0/1413687324 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780038e0 con 0x55f0001cab50 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 -- 172.21.15.116:0/1413687324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cab50 msgr2=0x55f0001cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cab50 0x55f0001cbdd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa978009900 tx=0x7fa97802cf90 comp rx=0 tx=0).stop 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 -- 172.21.15.116:0/1413687324 shutdown_connections 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001d06b0 0x55f0001c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001dc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 --2- 172.21.15.116:0/1413687324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001cab50 0x55f0001cbdd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 -- 172.21.15.116:0/1413687324 >> 172.21.15.116:0/1413687324 conn(0x55f0001c60c0 msgr2=0x55f0001d4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:07.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 -- 172.21.15.116:0/1413687324 shutdown_connections 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.955+0000 7fa9927da740 1 -- 172.21.15.116:0/1413687324 wait complete. 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 -- start start 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 Processor -- start 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cab50 0x55f0001da330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001da870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001db830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881a9860 con 0x55f0001cb540 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881aab80 con 0x55f0001d06b0 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9881a9ec0 con 0x55f0001cab50 2024-03-04T07:42:07.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa9927da740 10 client.0 _ll_register_callbacks cb 0x7fa970097220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.956+0000 7fa98da00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001db830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55062/0 (socket says 172.21.15.116:55062) 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 -- 172.21.15.116:0/3435919070 learned_addr learned my addr 172.21.15.116:0/3435919070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa97fe00640 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001da870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98d000640 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cab50 0x55f0001da330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 -- 172.21.15.116:0/3435919070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cab50 msgr2=0x55f0001da330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cab50 0x55f0001da330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 -- 172.21.15.116:0/3435919070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 msgr2=0x55f0001da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001da870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 -- 172.21.15.116:0/3435919070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0001dbe20 con 0x55f0001d06b0 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa98da00640 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001db830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa980010ea0 tx=0x7fa980004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c63547ea8f4df7d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa976a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:07.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa976a00640 1 -- 172.21.15.116:0/3435919070 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98001e440 con 0x55f0001d06b0 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.957+0000 7fa976a00640 1 -- 172.21.15.116:0/3435919070 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa98001e5e0 con 0x55f0001d06b0 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9780043e0 con 0x55f0001d06b0 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa976a00640 1 -- 172.21.15.116:0/3435919070 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa98001e7e0 con 0x55f0001d06b0 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 10 client.25559 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f00023c640 con 0x55f0001d06b0 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa976a00640 1 -- 172.21.15.116:0/3435919070 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9780043e0 con 0x55f0001d06b0 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa976a00640 1 -- 172.21.15.116:0/3435919070 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa98002a900 con 0x55f0001d06b0 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 10 client.25559 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 10 client.25559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 20 client.25559 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f00023cff0 0x55f00023f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 4 client.25559 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:42:07.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f0001cc4b0 con 0x55f00023cff0 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.958+0000 7fa98d000640 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f00023cff0 0x55f00023f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa976a00640 10 client.25559 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa98d000640 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f00023cff0 0x55f00023f410 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a069d2278523a7a7 server_cookie=576ef6470e1e7cf in_seq=0 out_seq=0 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa976a00640 1 -- 172.21.15.116:0/3435919070 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f0001cc4b0 con 0x55f00023cff0 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa976a00640 10 client.25559 handle_command_reply: tid=0 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 client.25559 shutdown 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 msgr2=0x55f0001db830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001db830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa980010ea0 tx=0x7fa980004ab0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 shutdown_connections 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f00023cff0 0x55f00023f410 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f0001d06b0 0x55f0001db830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0001cb540 0x55f0001da870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 --2- 172.21.15.116:0/3435919070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f0001cab50 0x55f0001da330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa976a00640 0 client.25559 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 >> 172.21.15.116:0/3435919070 conn(0x55f0001c60c0 msgr2=0x55f0001c9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 shutdown_connections 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.959+0000 7fa9927da740 1 -- 172.21.15.116:0/3435919070 wait complete. 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa9927da740 20 client.25559 trim_cache size 0 max 16384 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:42:07.966 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa96c0a1d80 msgr2=0x7fa96c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa96c0a1d80 0x7fa96c0a4240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55f0001cbda0 tx=0x7fa980004a40 comp rx=0 tx=0).stop 2024-03-04T07:42:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881049e0 msgr2=0x7fa98819a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881049e0 0x7fa98819a030 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa97000e9f0 tx=0x7fa97000edf0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 shutdown_connections 2024-03-04T07:42:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa96c0a1d80 0x7fa96c0a4240 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9881053a0 0x7fa9881a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9881049e0 0x7fa98819a030 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 --2- 172.21.15.116:0/325817790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9881037f0 0x7fa988199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 >> 172.21.15.116:0/325817790 conn(0x7fa9880fed80 msgr2=0x7fa988107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 shutdown_connections 2024-03-04T07:42:07.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:07.960+0000 7fa98fa00640 1 -- 172.21.15.116:0/325817790 wait complete. 2024-03-04T07:42:07.977 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:42:07.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 -- 172.21.15.116:0/1428949782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 msgr2=0x7f1afc1090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 --2- 172.21.15.116:0/1428949782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 0x7f1afc1090a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009900 tx=0x7f1ae802bc30 comp rx=0 tx=0).stop 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 -- 172.21.15.116:0/1428949782 shutdown_connections 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 --2- 172.21.15.116:0/1428949782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc1095e0 0x7f1afc10fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 --2- 172.21.15.116:0/1428949782 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 0x7f1afc1090a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 --2- 172.21.15.116:0/1428949782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1afc107a30 0x7f1afc107e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 -- 172.21.15.116:0/1428949782 >> 172.21.15.116:0/1428949782 conn(0x7f1afc0769e0 msgr2=0x7f1afc078e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 -- 172.21.15.116:0/1428949782 shutdown_connections 2024-03-04T07:42:08.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.178+0000 7f1b02a00640 1 -- 172.21.15.116:0/1428949782 wait complete. 2024-03-04T07:42:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 Processor -- start 2024-03-04T07:42:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 -- start start 2024-03-04T07:42:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc107a30 0x7f1afc19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 0x7f1afc19e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1afc1095e0 0x7f1afc1a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc113700 con 0x7f1afc107a30 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc113580 con 0x7f1afc108c20 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b02a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc113880 con 0x7f1afc1095e0 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b01600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc107a30 0x7f1afc19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b00c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 0x7f1afc19e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b01600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc107a30 0x7f1afc19dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60856/0 (socket says 172.21.15.116:60856) 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.179+0000 7f1b01600640 1 -- 172.21.15.116:0/2347253885 learned_addr learned my addr 172.21.15.116:0/2347253885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b02000640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1afc1095e0 0x7f1afc1a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b00c00640 1 -- 172.21.15.116:0/2347253885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1afc1095e0 msgr2=0x7f1afc1a52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b00c00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1afc1095e0 0x7f1afc1a52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b00c00640 1 -- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc107a30 msgr2=0x7f1afc19dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b00c00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc107a30 0x7f1afc19dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b00c00640 1 -- 172.21.15.116:0/2347253885 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc1a5940 con 0x7f1afc108c20 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b01600640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc107a30 0x7f1afc19dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1b00c00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 0x7f1afc19e270 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8036040 tx=0x7f1ae802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.180+0000 7f1afa000640 1 -- 172.21.15.116:0/2347253885 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8043020 con 0x7f1afc108c20 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.181+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc1a5bd0 con 0x7f1afc108c20 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.181+0000 7f1afa000640 1 -- 172.21.15.116:0/2347253885 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ae80054e0 con 0x7f1afc108c20 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.181+0000 7f1afa000640 1 -- 172.21.15.116:0/2347253885 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8003c30 con 0x7f1afc108c20 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.181+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc1a60f0 con 0x7f1afc108c20 2024-03-04T07:42:08.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.182+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc10ed50 con 0x7f1afc108c20 2024-03-04T07:42:08.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.184+0000 7f1afa000640 1 -- 172.21.15.116:0/2347253885 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1ae8046020 con 0x7f1afc108c20 2024-03-04T07:42:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.185+0000 7f1afa000640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ae40a1d80 0x7f1ae40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.185+0000 7f1afa000640 1 -- 172.21.15.116:0/2347253885 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1ae80effe0 con 0x7f1afc108c20 2024-03-04T07:42:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.185+0000 7f1b01600640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ae40a1d80 0x7f1ae40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.186+0000 7f1b01600640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ae40a1d80 0x7f1ae40a4240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1afc19f200 tx=0x7f1af0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.188+0000 7f1afa000640 1 -- 172.21.15.116:0/2347253885 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1ae804d020 con 0x7f1afc108c20 2024-03-04T07:42:08.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.427+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1afc02cff0 con 0x7f1afc108c20 2024-03-04T07:42:08.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.429+0000 7f1afa000640 1 -- 172.21.15.116:0/2347253885 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f1ae80bd410 con 0x7f1afc108c20 2024-03-04T07:42:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:42:08.429 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:42:08.429 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:42:08.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.431+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ae40a1d80 msgr2=0x7f1ae40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.431+0000 7f1b02a00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ae40a1d80 0x7f1ae40a4240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1afc19f200 tx=0x7f1af0005f90 comp rx=0 tx=0).stop 2024-03-04T07:42:08.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.431+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 msgr2=0x7f1afc19e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.431+0000 7f1b02a00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 0x7f1afc19e270 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8036040 tx=0x7f1ae802c650 comp rx=0 tx=0).stop 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 shutdown_connections 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1ae40a1d80 0x7f1ae40a4240 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1afc1095e0 0x7f1afc1a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1afc108c20 0x7f1afc19e270 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 --2- 172.21.15.116:0/2347253885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1afc107a30 0x7f1afc19dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 >> 172.21.15.116:0/2347253885 conn(0x7f1afc0769e0 msgr2=0x7f1afc10bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 shutdown_connections 2024-03-04T07:42:08.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.432+0000 7f1b02a00640 1 -- 172.21.15.116:0/2347253885 wait complete. 2024-03-04T07:42:08.451 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:42:08.452 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:42:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 -- 172.21.15.116:0/3458434962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 msgr2=0x7fd960104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 --2- 172.21.15.116:0/3458434962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960104440 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd950009900 tx=0x7fd95002bc50 comp rx=0 tx=0).stop 2024-03-04T07:42:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 -- 172.21.15.116:0/3458434962 shutdown_connections 2024-03-04T07:42:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 --2- 172.21.15.116:0/3458434962 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9601072b0 0x7fd960109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 --2- 172.21.15.116:0/3458434962 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd960104980 0x7fd960106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 --2- 172.21.15.116:0/3458434962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960104440 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 -- 172.21.15.116:0/3458434962 >> 172.21.15.116:0/3458434962 conn(0x7fd9600fba60 msgr2=0x7fd9600fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 -- 172.21.15.116:0/3458434962 shutdown_connections 2024-03-04T07:42:08.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.661+0000 7fd966800640 1 -- 172.21.15.116:0/3458434962 wait complete. 2024-03-04T07:42:08.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.662+0000 7fd966800640 1 Processor -- start 2024-03-04T07:42:08.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.662+0000 7fd966800640 1 -- start start 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd966800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd966800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd960104980 0x7fd96019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd966800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9601072b0 0x7fd9601a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd966800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96010d2b0 con 0x7fd960102050 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd966800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96010d130 con 0x7fd9601072b0 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd966800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96010d430 con 0x7fd960104980 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60884/0 (socket says 172.21.15.116:60884) 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 -- 172.21.15.116:0/2521366970 learned_addr learned my addr 172.21.15.116:0/2521366970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:08.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95fe00640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9601072b0 0x7fd9601a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd956a00640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd960104980 0x7fd96019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 -- 172.21.15.116:0/2521366970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd960104980 msgr2=0x7fd96019a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd960104980 0x7fd96019a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 -- 172.21.15.116:0/2521366970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9601072b0 msgr2=0x7fd9601a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9601072b0 0x7fd9601a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95f400640 1 -- 172.21.15.116:0/2521366970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9601a17a0 con 0x7fd960102050 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.663+0000 7fd95fe00640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9601072b0 0x7fd9601a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.664+0000 7fd95f400640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960199b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd95002c260 tx=0x7fd95002c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.664+0000 7fd95cc00640 1 -- 172.21.15.116:0/2521366970 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd950041070 con 0x7fd960102050 2024-03-04T07:42:08.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.664+0000 7fd95cc00640 1 -- 172.21.15.116:0/2521366970 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd950004450 con 0x7fd960102050 2024-03-04T07:42:08.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.664+0000 7fd95cc00640 1 -- 172.21.15.116:0/2521366970 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd950034440 con 0x7fd960102050 2024-03-04T07:42:08.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.664+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9601a1a30 con 0x7fd960102050 2024-03-04T07:42:08.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.664+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9601a1f30 con 0x7fd960102050 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.667+0000 7fd9695ae740 1 Processor -- start 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.667+0000 7fd95cc00640 1 -- 172.21.15.116:0/2521366970 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd950044070 con 0x7fd960102050 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.667+0000 7fd9695ae740 1 -- start start 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd9695ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b8611d60 con 0x55c2b86acca0 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd9695ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b85bacf0 con 0x55c2b86ad690 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd9695ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b860de00 con 0x55c2b86b2800 2024-03-04T07:42:08.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55100/0 (socket says 172.21.15.116:55100) 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86ad0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60896/0 (socket says 172.21.15.116:60896) 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 -- 172.21.15.116:0/3103689608 learned_addr learned my addr 172.21.15.116:0/3103689608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd956a00640 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 -- 172.21.15.116:0/3103689608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 msgr2=0x55c2b86b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 -- 172.21.15.116:0/3103689608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 msgr2=0x55c2b86ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.668+0000 7fd95f400640 1 -- 172.21.15.116:0/3103689608 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2b83263f0 con 0x55c2b86ad690 2024-03-04T07:42:08.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.669+0000 7fd95f400640 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86b2190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd9500f5a90 tx=0x7fd9500f5b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e5ccc3d03580a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.669+0000 7fd954c00640 1 -- 172.21.15.116:0/3103689608 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9500fe070 con 0x55c2b86ad690 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.669+0000 7fd954c00640 1 -- 172.21.15.116:0/3103689608 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9500f10d0 con 0x55c2b86ad690 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.669+0000 7fd954c00640 1 -- 172.21.15.116:0/3103689608 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9500fa680 con 0x55c2b86ad690 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.669+0000 7fd95cc00640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9440a1e00 0x7fd9440a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd95cc00640 1 -- 172.21.15.116:0/2521366970 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd9500ef8a0 con 0x7fd960102050 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd95fe00640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9440a1e00 0x7fd9440a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 -- 172.21.15.116:0/3103689608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 msgr2=0x55c2b86b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86b2190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd9500f5a90 tx=0x7fd9500f5b80 comp rx=0 tx=0).stop 2024-03-04T07:42:08.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 -- 172.21.15.116:0/3103689608 shutdown_connections 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86b2190 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3103689608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 -- 172.21.15.116:0/3103689608 >> 172.21.15.116:0/3103689608 conn(0x55c2b86a8210 msgr2=0x55c2b86a8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 -- 172.21.15.116:0/3103689608 shutdown_connections 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd95fe00640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9440a1e00 0x7fd9440a42c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x55c2b86adef0 tx=0x7fd958009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.670+0000 7fd9695ae740 1 -- 172.21.15.116:0/3103689608 wait complete. 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 -- start start 2024-03-04T07:42:08.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 Processor -- start 2024-03-04T07:42:08.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86ad690 0x55c2b86b7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86b2800 0x55c2b86be490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b86b5570 con 0x55c2b86acca0 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b85b0220 con 0x55c2b86b2800 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.671+0000 7fd9695ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b86b56f0 con 0x55c2b86ad690 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd9695ae740 10 client.0 _ll_register_callbacks cb 0x55c2b8325be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86b6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60902/0 (socket says 172.21.15.116:60902) 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 -- 172.21.15.116:0/3963850184 learned_addr learned my addr 172.21.15.116:0/3963850184 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95fe00640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86b2800 0x55c2b86be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd956a00640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86ad690 0x55c2b86b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 -- 172.21.15.116:0/3963850184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86ad690 msgr2=0x55c2b86b7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86ad690 0x55c2b86b7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 -- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86b2800 msgr2=0x55c2b86be490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86b2800 0x55c2b86be490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 -- 172.21.15.116:0/3963850184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2b83f07d0 con 0x55c2b86acca0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd956a00640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86ad690 0x55c2b86b7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95fe00640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86b2800 0x55c2b86be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd95f400640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86b6f10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd9500b8f60 tx=0x7fd950005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d6d79cc270b3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.672+0000 7fd94d600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd94d600640 1 -- 172.21.15.116:0/3963850184 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95004b070 con 0x55c2b86acca0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd94d600640 1 -- 172.21.15.116:0/3963850184 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd9500ed470 con 0x55c2b86acca0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9500f10d0 con 0x55c2b86acca0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd94d600640 1 -- 172.21.15.116:0/3963850184 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9500fe070 con 0x55c2b86acca0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd9695ae740 10 client.15465 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c2b86b5570 con 0x55c2b86acca0 2024-03-04T07:42:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd94d600640 1 -- 172.21.15.116:0/3963850184 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9500f10d0 con 0x55c2b86acca0 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd94d600640 1 -- 172.21.15.116:0/3963850184 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd9500035f0 con 0x55c2b86acca0 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd9695ae740 10 client.15465 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd9695ae740 10 client.15465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.673+0000 7fd9695ae740 20 client.15465 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.674+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b86c2b70 0x55c2b86c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.674+0000 7fd9695ae740 4 client.15465 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.674+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2b861f2a0 con 0x55c2b86c2b70 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.674+0000 7fd956a00640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b86c2b70 0x55c2b86c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.674+0000 7fd956a00640 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b86c2b70 0x55c2b86c4f60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51a9540940f6d08 server_cookie=ffcd3937f59f530 in_seq=0 out_seq=0 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.674+0000 7fd94d600640 10 client.15465 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:08.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.675+0000 7fd94d600640 1 -- 172.21.15.116:0/3963850184 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2b861f2a0 con 0x55c2b86c2b70 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.675+0000 7fd94d600640 10 client.15465 handle_command_reply: tid=0 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.675+0000 7fd9695ae740 1 client.15465 shutdown 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 msgr2=0x55c2b86b6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86b6f10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd9500b8f60 tx=0x7fd950005e50 comp rx=0 tx=0).stop 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 shutdown_connections 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b86c2b70 0x55c2b86c4f60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86b2800 0x55c2b86be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86ad690 0x55c2b86b7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3963850184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86b6f10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd94d600640 0 client.15465 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 >> 172.21.15.116:0/3963850184 conn(0x55c2b86a8210 msgr2=0x55c2b86a74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 shutdown_connections 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 1 -- 172.21.15.116:0/3963850184 wait complete. 2024-03-04T07:42:08.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.676+0000 7fd9695ae740 20 client.15465 trim_cache size 0 max 16384 2024-03-04T07:42:08.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.689+0000 7fd9695ae740 1 Processor -- start 2024-03-04T07:42:08.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.689+0000 7fd9695ae740 1 -- start start 2024-03-04T07:42:08.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86acca0 0x55c2b86b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86be420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86b2800 0x55c2b86a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd9695ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601a8d00 con 0x55c2b86b2800 2024-03-04T07:42:08.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd9695ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601ab2e0 con 0x55c2b86ad690 2024-03-04T07:42:08.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd9695ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601a9300 con 0x55c2b86acca0 2024-03-04T07:42:08.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86be420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd95fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86b2800 0x55c2b86a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86be420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55114/0 (socket says 172.21.15.116:55114) 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 -- 172.21.15.116:0/3942108321 learned_addr learned my addr 172.21.15.116:0/3942108321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd95f400640 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86acca0 0x55c2b86b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 -- 172.21.15.116:0/3942108321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86acca0 msgr2=0x55c2b86b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86acca0 0x55c2b86b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 -- 172.21.15.116:0/3942108321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86b2800 msgr2=0x55c2b86a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86b2800 0x55c2b86a6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.690+0000 7fd956a00640 1 -- 172.21.15.116:0/3942108321 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2b86b5fc0 con 0x55c2b86ad690 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.691+0000 7fd956a00640 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86be420 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55c2b86a9180 tx=0x7fd948005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc820b5f096cbb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.691+0000 7fd94fe00640 1 -- 172.21.15.116:0/3942108321 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948020e40 con 0x55c2b86ad690 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.691+0000 7fd94fe00640 1 -- 172.21.15.116:0/3942108321 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd948024030 con 0x55c2b86ad690 2024-03-04T07:42:08.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.691+0000 7fd94fe00640 1 -- 172.21.15.116:0/3942108321 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948005310 con 0x55c2b86ad690 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.691+0000 7fd9695ae740 1 -- 172.21.15.116:0/3942108321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 msgr2=0x55c2b86be420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86be420 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55c2b86a9180 tx=0x7fd948005f90 comp rx=0 tx=0).stop 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 -- 172.21.15.116:0/3942108321 shutdown_connections 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86b2800 0x55c2b86a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86be420 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 --2- 172.21.15.116:0/3942108321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86acca0 0x55c2b86b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 -- 172.21.15.116:0/3942108321 >> 172.21.15.116:0/3942108321 conn(0x55c2b86a8210 msgr2=0x55c2b86b6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 -- 172.21.15.116:0/3942108321 shutdown_connections 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 -- 172.21.15.116:0/3942108321 wait complete. 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 -- start start 2024-03-04T07:42:08.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.692+0000 7fd9695ae740 1 Processor -- start 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86bbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86bc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd9695ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86bd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd9695ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601a9960 con 0x55c2b86acca0 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd9695ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601aac80 con 0x55c2b86ad690 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd9695ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9601a9fc0 con 0x55c2b86b2800 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd9695ae740 10 client.0 _ll_register_callbacks cb 0x7fd958009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86bbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd956a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86bc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86bbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60910/0 (socket says 172.21.15.116:60910) 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 -- 172.21.15.116:0/4186868562 learned_addr learned my addr 172.21.15.116:0/4186868562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:08.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95fe00640 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86bd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 -- 172.21.15.116:0/4186868562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 msgr2=0x55c2b86bd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86bd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 -- 172.21.15.116:0/4186868562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 msgr2=0x55c2b86bc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86bc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 -- 172.21.15.116:0/4186868562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2b86bd700 con 0x55c2b86acca0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95fe00640 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86bd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd95f400640 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86bbbc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd9500039d0 tx=0x7fd9500a70b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ab5d5ee853f7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.693+0000 7fd94d600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd94d600640 1 -- 172.21.15.116:0/4186868562 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95004b070 con 0x55c2b86acca0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd94d600640 1 -- 172.21.15.116:0/4186868562 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd9500bc7f0 con 0x55c2b86acca0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd948024030 con 0x55c2b86acca0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd94d600640 1 -- 172.21.15.116:0/4186868562 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9500bc360 con 0x55c2b86acca0 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 10 client.15471 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:08.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c2b86bd700 con 0x55c2b86acca0 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd94d600640 1 -- 172.21.15.116:0/4186868562 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd948024030 con 0x55c2b86acca0 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd94d600640 1 -- 172.21.15.116:0/4186868562 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd9500c48a0 con 0x55c2b86acca0 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 10 client.15471 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 10 client.15471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 20 client.15471 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b8727110 0x55c2b8729530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 4 client.15471 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.694+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c2b86195f0 con 0x55c2b8727110 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd956a00640 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b8727110 0x55c2b8729530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd956a00640 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b8727110 0x55c2b8729530 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cf173af2c0007a3 server_cookie=b17ac33e1f67ed29 in_seq=0 out_seq=0 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd94d600640 10 client.15471 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd94d600640 1 -- 172.21.15.116:0/4186868562 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c2b86195f0 con 0x55c2b8727110 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd94d600640 10 client.15471 handle_command_reply: tid=0 2024-03-04T07:42:08.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd9695ae740 1 client.15471 shutdown 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 msgr2=0x55c2b86bbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.695+0000 7fd9695ae740 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86bbbc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd9500039d0 tx=0x7fd9500a70b0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 shutdown_connections 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c2b8727110 0x55c2b8729530 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2b86b2800 0x55c2b86bd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c2b86ad690 0x55c2b86bc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 --2- 172.21.15.116:0/4186868562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2b86acca0 0x55c2b86bbbc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd94d600640 0 client.15471 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 >> 172.21.15.116:0/4186868562 conn(0x55c2b86a8210 msgr2=0x55c2b86ab470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 shutdown_connections 2024-03-04T07:42:08.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 1 -- 172.21.15.116:0/4186868562 wait complete. 2024-03-04T07:42:08.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd9695ae740 20 client.15471 trim_cache size 0 max 16384 2024-03-04T07:42:08.701 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9440a1e00 msgr2=0x7fd9440a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd966800640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9440a1e00 0x7fd9440a42c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x55c2b86adef0 tx=0x7fd958009500 comp rx=0 tx=0).stop 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 msgr2=0x7fd960199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.696+0000 7fd966800640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960199b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd95002c260 tx=0x7fd95002c670 comp rx=0 tx=0).stop 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 shutdown_connections 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd9601072b0 0x7fd9601a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9440a1e00 0x7fd9440a42c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd960104980 0x7fd96019a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 --2- 172.21.15.116:0/2521366970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd960102050 0x7fd960199b80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 >> 172.21.15.116:0/2521366970 conn(0x7fd9600fba60 msgr2=0x7fd960101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 shutdown_connections 2024-03-04T07:42:08.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:08.697+0000 7fd966800640 1 -- 172.21.15.116:0/2521366970 wait complete. 2024-03-04T07:42:08.711 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:08 smithi116 ceph-mon[32150]: pgmap v442: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:08.711 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:08 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2347253885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:08 smithi152 ceph-mon[37154]: pgmap v442: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:08 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2347253885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:08 smithi133 ceph-mon[36505]: pgmap v442: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:08 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2347253885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:09.716 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:42:09.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:42:09.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:42:09.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 -- 172.21.15.116:0/2175843730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6941097a0 msgr2=0x7fe694110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 --2- 172.21.15.116:0/2175843730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6941097a0 0x7fe694110030 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009ca0 tx=0x7fe68c02c100 comp rx=0 tx=0).stop 2024-03-04T07:42:09.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 -- 172.21.15.116:0/2175843730 shutdown_connections 2024-03-04T07:42:09.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 --2- 172.21.15.116:0/2175843730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6941097a0 0x7fe694110030 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 --2- 172.21.15.116:0/2175843730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe694108de0 0x7fe694109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 --2- 172.21.15.116:0/2175843730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe694107bf0 0x7fe694108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 -- 172.21.15.116:0/2175843730 >> 172.21.15.116:0/2175843730 conn(0x7fe694076b90 msgr2=0x7fe694078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:09.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 -- 172.21.15.116:0/2175843730 shutdown_connections 2024-03-04T07:42:09.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.942+0000 7fe698c00640 1 -- 172.21.15.116:0/2175843730 wait complete. 2024-03-04T07:42:09.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 Processor -- start 2024-03-04T07:42:09.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 -- start start 2024-03-04T07:42:09.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe694107bf0 0x7fe69419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe694108de0 0x7fe69419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6941097a0 0x7fe6941a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694113850 con 0x7fe6941097a0 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941136d0 con 0x7fe694107bf0 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe698c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941139d0 con 0x7fe694108de0 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe691e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe694107bf0 0x7fe69419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe691e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe694107bf0 0x7fe69419de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55154/0 (socket says 172.21.15.116:55154) 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.943+0000 7fe691e00640 1 -- 172.21.15.116:0/780515030 learned_addr learned my addr 172.21.15.116:0/780515030 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe692800640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6941097a0 0x7fe6941a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe68be00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe694108de0 0x7fe69419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe691e00640 1 -- 172.21.15.116:0/780515030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe694108de0 msgr2=0x7fe69419e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe691e00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe694108de0 0x7fe69419e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe691e00640 1 -- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6941097a0 msgr2=0x7fe6941a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe691e00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6941097a0 0x7fe6941a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe691e00640 1 -- 172.21.15.116:0/780515030 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6941a5ab0 con 0x7fe694107bf0 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe68be00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe694108de0 0x7fe69419e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:09.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe691e00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe694107bf0 0x7fe69419de90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe68400d950 tx=0x7fe68400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe68aa00640 1 -- 172.21.15.116:0/780515030 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684017020 con 0x7fe694107bf0 2024-03-04T07:42:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe68aa00640 1 -- 172.21.15.116:0/780515030 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe684009c30 con 0x7fe694107bf0 2024-03-04T07:42:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe68aa00640 1 -- 172.21.15.116:0/780515030 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68400f720 con 0x7fe694107bf0 2024-03-04T07:42:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.944+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6941a5da0 con 0x7fe694107bf0 2024-03-04T07:42:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.945+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6941a6260 con 0x7fe694107bf0 2024-03-04T07:42:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 Processor -- start 2024-03-04T07:42:09.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe68aa00640 1 -- 172.21.15.116:0/780515030 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe68401a020 con 0x7fe694107bf0 2024-03-04T07:42:09.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 -- start start 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b30d0 0x55d6d34b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b8c30 0x55d6d34bb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6d33c6e20 con 0x55d6d34b3ac0 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6d3418a20 con 0x55d6d34b30d0 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe69b997740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6d33c1b00 con 0x55d6d34b8c30 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.948+0000 7fe692800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b30d0 0x55d6d34b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b30d0 0x55d6d34b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55158/0 (socket says 172.21.15.116:55158) 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 -- 172.21.15.116:0/2318524316 learned_addr learned my addr 172.21.15.116:0/2318524316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:09.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe691e00640 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe68be00640 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b8c30 0x55d6d34bb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 -- 172.21.15.116:0/2318524316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b8c30 msgr2=0x55d6d34bb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b8c30 0x55d6d34bb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 -- 172.21.15.116:0/2318524316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 msgr2=0x55d6d34b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34b85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 -- 172.21.15.116:0/2318524316 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6d33d6e10 con 0x55d6d34b30d0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe692800640 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b30d0 0x55d6d34b34f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe68c02c710 tx=0x7fe68c0323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed7b427ab8ca7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.949+0000 7fe683000640 1 -- 172.21.15.116:0/2318524316 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c043070 con 0x55d6d34b30d0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe683000640 1 -- 172.21.15.116:0/2318524316 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe68c032c90 con 0x55d6d34b30d0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe683000640 1 -- 172.21.15.116:0/2318524316 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c039040 con 0x55d6d34b30d0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe68aa00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6780a1e00 0x7fe6780a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe68aa00640 1 -- 172.21.15.116:0/780515030 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe6840cdcd0 con 0x7fe694107bf0 2024-03-04T07:42:09.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe691e00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6780a1e00 0x7fe6780a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe69b997740 1 -- 172.21.15.116:0/2318524316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b30d0 msgr2=0x55d6d34b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe69b997740 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b30d0 0x55d6d34b34f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe68c02c710 tx=0x7fe68c0323d0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe69b997740 1 -- 172.21.15.116:0/2318524316 shutdown_connections 2024-03-04T07:42:09.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe69b997740 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b8c30 0x55d6d34bb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe69b997740 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe69b997740 1 --2- 172.21.15.116:0/2318524316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b30d0 0x55d6d34b34f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe69b997740 1 -- 172.21.15.116:0/2318524316 >> 172.21.15.116:0/2318524316 conn(0x55d6d34ae640 msgr2=0x55d6d34aea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.950+0000 7fe691e00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6780a1e00 0x7fe6780a42c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe684097390 tx=0x7fe6840d1710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.951+0000 7fe69b997740 1 -- 172.21.15.116:0/2318524316 shutdown_connections 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.951+0000 7fe69b997740 1 -- 172.21.15.116:0/2318524316 wait complete. 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.951+0000 7fe69b997740 1 -- start start 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.951+0000 7fe69b997740 1 Processor -- start 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe69b997740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b30d0 0x55d6d34bd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe69b997740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b3ac0 0x55d6d34bd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe69b997740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe69b997740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6d33d4f40 con 0x55d6d34b30d0 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe69b997740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6d3414ac0 con 0x55d6d34b8c30 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe69b997740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6d34bbb80 con 0x55d6d34b3ac0 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe69b997740 10 client.0 _ll_register_callbacks cb 0x55d6d3413aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b30d0 0x55d6d34bd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b30d0 0x55d6d34bd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60940/0 (socket says 172.21.15.116:60940) 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 -- 172.21.15.116:0/3310094940 learned_addr learned my addr 172.21.15.116:0/3310094940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe68be00640 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b3ac0 0x55d6d34bd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 -- 172.21.15.116:0/3310094940 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b3ac0 msgr2=0x55d6d34bd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b3ac0 0x55d6d34bd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 -- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 msgr2=0x55d6d34c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.952+0000 7fe692800640 1 -- 172.21.15.116:0/3310094940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6d31f57e0 con 0x55d6d34b30d0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe692800640 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b30d0 0x55d6d34bd440 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe68c005e30 tx=0x7fe68c009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c16f991206d953 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe677400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe677400640 1 -- 172.21.15.116:0/3310094940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c053060 con 0x55d6d34b30d0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe677400640 1 -- 172.21.15.116:0/3310094940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe68c0042b0 con 0x55d6d34b30d0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe68c032c90 con 0x55d6d34b30d0 2024-03-04T07:42:09.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe677400640 1 -- 172.21.15.116:0/3310094940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe68c039040 con 0x55d6d34b30d0 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe69b997740 10 client.15483 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d6d3255f70 con 0x55d6d34b30d0 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.953+0000 7fe677400640 1 -- 172.21.15.116:0/3310094940 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe68c032c90 con 0x55d6d34b30d0 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe677400640 1 -- 172.21.15.116:0/3310094940 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe68c0599a0 con 0x55d6d34b30d0 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe69b997740 10 client.15483 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe69b997740 10 client.15483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe69b997740 20 client.15483 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe69b997740 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d34c8eb0 0x55d6d34cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe69b997740 4 client.15483 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6d3425f60 con 0x55d6d34c8eb0 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.954+0000 7fe68be00640 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d34c8eb0 0x55d6d34cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.955+0000 7fe68be00640 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d34c8eb0 0x55d6d34cb2a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35f157409769f4c6 server_cookie=46889a8f4a952948 in_seq=0 out_seq=0 2024-03-04T07:42:09.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.955+0000 7fe677400640 10 client.15483 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.955+0000 7fe677400640 1 -- 172.21.15.116:0/3310094940 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d6d3425f60 con 0x55d6d34c8eb0 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.955+0000 7fe677400640 10 client.15483 handle_command_reply: tid=0 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 client.15483 shutdown 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b30d0 msgr2=0x55d6d34bd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b30d0 0x55d6d34bd440 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe68c005e30 tx=0x7fe68c009540 comp rx=0 tx=0).stop 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 shutdown_connections 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d34c8eb0 0x55d6d34cb2a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b3ac0 0x55d6d34bd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 --2- 172.21.15.116:0/3310094940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b30d0 0x55d6d34bd440 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe677400640 0 client.15483 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:09.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 >> 172.21.15.116:0/3310094940 conn(0x55d6d34ae640 msgr2=0x55d6d34af890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:09.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 shutdown_connections 2024-03-04T07:42:09.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 1 -- 172.21.15.116:0/3310094940 wait complete. 2024-03-04T07:42:09.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.956+0000 7fe69b997740 20 client.15483 trim_cache size 0 max 16384 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 Processor -- start 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 -- start start 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941ad000 con 0x55d6d34b3ac0 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941af5e0 con 0x55d6d34b8c30 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe69b997740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941ad600 con 0x55d6d34b30d0 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe68be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe692800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe692800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34cfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55190/0 (socket says 172.21.15.116:55190) 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe692800640 1 -- 172.21.15.116:0/2268547338 learned_addr learned my addr 172.21.15.116:0/2268547338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe68be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60946/0 (socket says 172.21.15.116:60946) 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.974+0000 7fe692800640 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe692800640 1 -- 172.21.15.116:0/2268547338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 msgr2=0x55d6d34cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe692800640 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34cf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe692800640 1 -- 172.21.15.116:0/2268547338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 msgr2=0x55d6d34c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe692800640 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe692800640 1 -- 172.21.15.116:0/2268547338 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6d30ed170 con 0x55d6d34b8c30 2024-03-04T07:42:09.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe692800640 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34cfef0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe68c059fb0 tx=0x7fe68c032590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af455bc6a805dbda server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe682600640 1 -- 172.21.15.116:0/2268547338 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c043040 con 0x55d6d34b8c30 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe682600640 1 -- 172.21.15.116:0/2268547338 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe68c009b20 con 0x55d6d34b8c30 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.975+0000 7fe682600640 1 -- 172.21.15.116:0/2268547338 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c048910 con 0x55d6d34b8c30 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 -- 172.21.15.116:0/2268547338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 msgr2=0x55d6d34cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34cfef0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe68c059fb0 tx=0x7fe68c032590 comp rx=0 tx=0).stop 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 -- 172.21.15.116:0/2268547338 shutdown_connections 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b8c30 0x55d6d34cfef0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 --2- 172.21.15.116:0/2268547338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 -- 172.21.15.116:0/2268547338 >> 172.21.15.116:0/2268547338 conn(0x55d6d34ae640 msgr2=0x55d6d34b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 -- 172.21.15.116:0/2268547338 shutdown_connections 2024-03-04T07:42:09.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.976+0000 7fe69b997740 1 -- 172.21.15.116:0/2268547338 wait complete. 2024-03-04T07:42:09.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 -- start start 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 Processor -- start 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34c3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b8c30 0x55d6d3524580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941adc60 con 0x55d6d34b8c30 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941aef80 con 0x55d6d34b3ac0 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941ae2c0 con 0x55d6d34b30d0 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe69b997740 10 client.0 _ll_register_callbacks cb 0x7fe6840d15e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe68be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe692800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b8c30 0x55d6d3524580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe68be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55200/0 (socket says 172.21.15.116:55200) 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.977+0000 7fe68be00640 1 -- 172.21.15.116:0/3577064445 learned_addr learned my addr 172.21.15.116:0/3577064445 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe692800640 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe68be00640 1 -- 172.21.15.116:0/3577064445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 msgr2=0x55d6d34c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe68be00640 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34c3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe68be00640 1 -- 172.21.15.116:0/3577064445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b8c30 msgr2=0x55d6d3524580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe68be00640 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b8c30 0x55d6d3524580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe68be00640 1 -- 172.21.15.116:0/3577064445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6d3524ac0 con 0x55d6d34b3ac0 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe692800640 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b8c30 0x55d6d3524580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe68be00640 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c3ac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe67c004700 tx=0x7fe67c004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30eb9c13b4f9cd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe677400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe677400640 1 -- 172.21.15.116:0/3577064445 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c009530 con 0x55d6d34b3ac0 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.978+0000 7fe677400640 1 -- 172.21.15.116:0/3577064445 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe67c01e680 con 0x55d6d34b3ac0 2024-03-04T07:42:09.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe68c009b20 con 0x55d6d34b3ac0 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe677400640 1 -- 172.21.15.116:0/3577064445 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe67c01e880 con 0x55d6d34b3ac0 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 10 client.25601 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d6d3525d40 con 0x55d6d34b3ac0 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe677400640 1 -- 172.21.15.116:0/3577064445 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe68c009b20 con 0x55d6d34b3ac0 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe677400640 1 -- 172.21.15.116:0/3577064445 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe67c027c40 con 0x55d6d34b3ac0 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 10 client.25601 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 10 client.25601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 20 client.25601 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d3526650 0x55d6d3528a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 4 client.25601 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.979+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d6d34202b0 con 0x55d6d3526650 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.980+0000 7fe692800640 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d3526650 0x55d6d3528a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.980+0000 7fe692800640 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d3526650 0x55d6d3528a70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa484b88579081d2 server_cookie=e4971b5b66b135ea in_seq=0 out_seq=0 2024-03-04T07:42:09.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.980+0000 7fe677400640 10 client.25601 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe677400640 1 -- 172.21.15.116:0/3577064445 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d6d34202b0 con 0x55d6d3526650 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe677400640 10 client.25601 handle_command_reply: tid=0 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 client.25601 shutdown 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b3ac0 msgr2=0x55d6d34c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c3ac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe67c004700 tx=0x7fe67c004610 comp rx=0 tx=0).stop 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 shutdown_connections 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d6d3526650 0x55d6d3528a70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6d34b8c30 0x55d6d3524580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d6d34b3ac0 0x55d6d34c3ac0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe69b997740 1 --2- 172.21.15.116:0/3577064445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d6d34b30d0 0x55d6d34c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.981+0000 7fe677400640 0 client.25601 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 >> 172.21.15.116:0/3577064445 conn(0x55d6d34ae640 msgr2=0x55d6d34ad1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:09.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 shutdown_connections 2024-03-04T07:42:09.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe69b997740 1 -- 172.21.15.116:0/3577064445 wait complete. 2024-03-04T07:42:09.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe69b997740 20 client.25601 trim_cache size 0 max 16384 2024-03-04T07:42:09.987 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:09.987 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:42:09.987 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "e5545310-b988-4b6b-953e-818a4af7cf32", 2024-03-04T07:42:09.987 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:42:09.987 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6780a1e00 msgr2=0x7fe6780a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe698c00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6780a1e00 0x7fe6780a42c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe684097390 tx=0x7fe6840d1710 comp rx=0 tx=0).stop 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe694107bf0 msgr2=0x7fe69419de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.982+0000 7fe698c00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe694107bf0 0x7fe69419de90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe68400d950 tx=0x7fe68400de10 comp rx=0 tx=0).stop 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 shutdown_connections 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6941097a0 0x7fe6941a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe6780a1e00 0x7fe6780a42c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe694108de0 0x7fe69419e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 --2- 172.21.15.116:0/780515030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe694107bf0 0x7fe69419de90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:09.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 >> 172.21.15.116:0/780515030 conn(0x7fe694076b90 msgr2=0x7fe69410c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:09.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 shutdown_connections 2024-03-04T07:42:09.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:09.983+0000 7fe698c00640 1 -- 172.21.15.116:0/780515030 wait complete. 2024-03-04T07:42:10.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:42:10.000 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.215+0000 7f5acd200640 1 -- 172.21.15.116:0/2229753996 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 msgr2=0x7f5ac8108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.215+0000 7f5acd200640 1 --2- 172.21.15.116:0/2229753996 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac8108010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009ca0 tx=0x7f5ab802c100 comp rx=0 tx=0).stop 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.216+0000 7f5acd200640 1 -- 172.21.15.116:0/2229753996 shutdown_connections 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.216+0000 7f5acd200640 1 --2- 172.21.15.116:0/2229753996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ac81097a0 0x7f5ac8110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.216+0000 7f5acd200640 1 --2- 172.21.15.116:0/2229753996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac8108de0 0x7f5ac8109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.216+0000 7f5acd200640 1 --2- 172.21.15.116:0/2229753996 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac8108010 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.216+0000 7f5acd200640 1 -- 172.21.15.116:0/2229753996 >> 172.21.15.116:0/2229753996 conn(0x7f5ac8076b90 msgr2=0x7f5ac8078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.216+0000 7f5acd200640 1 -- 172.21.15.116:0/2229753996 shutdown_connections 2024-03-04T07:42:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.216+0000 7f5acd200640 1 -- 172.21.15.116:0/2229753996 wait complete. 2024-03-04T07:42:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 Processor -- start 2024-03-04T07:42:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 -- start start 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ac8108de0 0x7f5ac819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac81097a0 0x7f5ac81a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8113870 con 0x7f5ac81097a0 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81136f0 con 0x7f5ac8107bf0 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5acd200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81139f0 con 0x7f5ac8108de0 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac819def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55216/0 (socket says 172.21.15.116:55216) 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.217+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2219186458 learned_addr learned my addr 172.21.15.116:0/2219186458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac6800640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac81097a0 0x7f5ac81a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5400640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ac8108de0 0x7f5ac819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2219186458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ac8108de0 msgr2=0x7f5ac819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ac8108de0 0x7f5ac819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac81097a0 msgr2=0x7f5ac81a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac81097a0 0x7f5ac81a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2219186458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac81a5b10 con 0x7f5ac8107bf0 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5400640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ac8108de0 0x7f5ac819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac819def0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000f80 tx=0x7f5ab8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5abea00640 1 -- 172.21.15.116:0/2219186458 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8039040 con 0x7f5ac8107bf0 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5abea00640 1 -- 172.21.15.116:0/2219186458 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ab8044070 con 0x7f5ac8107bf0 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac81a5da0 con 0x7f5ac8107bf0 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.218+0000 7f5abea00640 1 -- 172.21.15.116:0/2219186458 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80045a0 con 0x7f5ac8107bf0 2024-03-04T07:42:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.219+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac81a61f0 con 0x7f5ac8107bf0 2024-03-04T07:42:10.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 Processor -- start 2024-03-04T07:42:10.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5abea00640 1 -- 172.21.15.116:0/2219186458 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5ab8032890 con 0x7f5ac8107bf0 2024-03-04T07:42:10.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 -- start start 2024-03-04T07:42:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e032b50 0x56157e032f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e038040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e0386b0 0x56157e03ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157df60090 con 0x56157e0386b0 2024-03-04T07:42:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157df97d30 con 0x56157e033540 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5acfe60740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157df40a30 con 0x56157e032b50 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e038040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5ac6800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e032b50 0x56157e032f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5ac5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e0386b0 0x56157e03ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e038040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55218/0 (socket says 172.21.15.116:55218) 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.222+0000 7f5ac5e00640 1 -- 172.21.15.116:0/3949010928 learned_addr learned my addr 172.21.15.116:0/3949010928 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ac5e00640 1 -- 172.21.15.116:0/3949010928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e032b50 msgr2=0x56157e032f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/3949010928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e032b50 0x56157e032f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ac5e00640 1 -- 172.21.15.116:0/3949010928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e0386b0 msgr2=0x56157e03ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/3949010928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e0386b0 0x56157e03ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ac5e00640 1 -- 172.21.15.116:0/3949010928 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56157dd767d0 con 0x56157e033540 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/3949010928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e038040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80f8440 tx=0x7f5ab80f8530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a6695ca604fb1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5abea00640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5aac0a1e00 0x7f5aac0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ab7400640 1 -- 172.21.15.116:0/3949010928 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80f8ba0 con 0x56157e033540 2024-03-04T07:42:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ab7400640 1 -- 172.21.15.116:0/3949010928 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ab80fd020 con 0x56157e033540 2024-03-04T07:42:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.223+0000 7f5ab7400640 1 -- 172.21.15.116:0/3949010928 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80cdef0 con 0x56157e033540 2024-03-04T07:42:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5abea00640 1 -- 172.21.15.116:0/2219186458 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5ab80f0400 con 0x7f5ac8107bf0 2024-03-04T07:42:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5ac6800640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5aac0a1e00 0x7f5aac0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 -- 172.21.15.116:0/3949010928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 msgr2=0x56157e038040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 --2- 172.21.15.116:0/3949010928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e038040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80f8440 tx=0x7f5ab80f8530 comp rx=0 tx=0).stop 2024-03-04T07:42:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 -- 172.21.15.116:0/3949010928 shutdown_connections 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 --2- 172.21.15.116:0/3949010928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e0386b0 0x56157e03ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 --2- 172.21.15.116:0/3949010928 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e038040 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 --2- 172.21.15.116:0/3949010928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e032b50 0x56157e032f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 -- 172.21.15.116:0/3949010928 >> 172.21.15.116:0/3949010928 conn(0x56157e02e0c0 msgr2=0x56157e02e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5ac6800640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5aac0a1e00 0x7f5aac0a42c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x56157e033da0 tx=0x7f5ac000b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 -- 172.21.15.116:0/3949010928 shutdown_connections 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.224+0000 7f5acfe60740 1 -- 172.21.15.116:0/3949010928 wait complete. 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.225+0000 7f5acfe60740 1 -- start start 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.225+0000 7f5acfe60740 1 Processor -- start 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e03cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e033540 0x56157e03d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e0386b0 0x56157e044120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5acfe60740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157df360c0 con 0x56157e032b50 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5acfe60740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157df93dd0 con 0x56157e0386b0 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5acfe60740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56157e03b420 con 0x56157e033540 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5acfe60740 10 client.0 _ll_register_callbacks cb 0x56157df53860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac5400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e033540 0x56157e03d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e03cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e0386b0 0x56157e044120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac5400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e033540 0x56157e03d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:52994/0 (socket says 172.21.15.116:52994) 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac5400640 1 -- 172.21.15.116:0/577817698 learned_addr learned my addr 172.21.15.116:0/577817698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac6800640 1 -- 172.21.15.116:0/577817698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e033540 msgr2=0x56157e03d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac6800640 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e033540 0x56157e03d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac6800640 1 -- 172.21.15.116:0/577817698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 msgr2=0x56157e03cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac6800640 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e03cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac6800640 1 -- 172.21.15.116:0/577817698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56157dcac3f0 con 0x56157e0386b0 2024-03-04T07:42:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.226+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e03cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5ac6800640 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e0386b0 0x56157e044120 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000bf90 tx=0x7f5ac0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f426886e4af7c804 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5aabe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5aabe00640 1 -- 172.21.15.116:0/577817698 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac002c060 con 0x56157e0386b0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5aabe00640 1 -- 172.21.15.116:0/577817698 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5ac0017030 con 0x56157e0386b0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ab80fd020 con 0x56157e0386b0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5aabe00640 1 -- 172.21.15.116:0/577817698 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ac001fcc0 con 0x56157e0386b0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5acfe60740 10 client.25613 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56157dc6c0f0 con 0x56157e0386b0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.227+0000 7f5aabe00640 1 -- 172.21.15.116:0/577817698 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ab80fd020 con 0x56157e0386b0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5aabe00640 1 -- 172.21.15.116:0/577817698 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5ac0031070 con 0x56157e0386b0 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5acfe60740 10 client.25613 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5acfe60740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5acfe60740 20 client.25613 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5acfe60740 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e048650 0x56157e04aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5acfe60740 4 client.25613 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56157dfa5270 con 0x56157e048650 2024-03-04T07:42:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.228+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e048650 0x56157e04aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.229+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e048650 0x56157e04aaf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=855098edc7901171 server_cookie=c75222aea580aee2 in_seq=0 out_seq=0 2024-03-04T07:42:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.229+0000 7f5aabe00640 10 client.25613 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.229+0000 7f5aabe00640 1 -- 172.21.15.116:0/577817698 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56157dfa5270 con 0x56157e048650 2024-03-04T07:42:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.229+0000 7f5aabe00640 10 client.25613 handle_command_reply: tid=0 2024-03-04T07:42:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.229+0000 7f5acfe60740 1 client.25613 shutdown 2024-03-04T07:42:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.229+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e0386b0 msgr2=0x56157e044120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.229+0000 7f5acfe60740 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e0386b0 0x56157e044120 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000bf90 tx=0x7f5ac0004730 comp rx=0 tx=0).stop 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 shutdown_connections 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e048650 0x56157e04aaf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e0386b0 0x56157e044120 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e033540 0x56157e03d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5aabe00640 0 client.25613 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 --2- 172.21.15.116:0/577817698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e03cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 >> 172.21.15.116:0/577817698 conn(0x56157e02e0c0 msgr2=0x56157e02d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 shutdown_connections 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 1 -- 172.21.15.116:0/577817698 wait complete. 2024-03-04T07:42:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.230+0000 7f5acfe60740 20 client.25613 trim_cache size 0 max 16384 2024-03-04T07:42:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.247+0000 7f5acfe60740 1 Processor -- start 2024-03-04T07:42:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.247+0000 7f5acfe60740 1 -- start start 2024-03-04T07:42:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e032b50 0x56157e031a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e033540 0x56157e0440e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e04f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5acfe60740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81ad080 con 0x56157e033540 2024-03-04T07:42:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5acfe60740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81af660 con 0x56157e032b50 2024-03-04T07:42:10.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5acfe60740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81ad680 con 0x56157e0386b0 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e032b50 0x56157e031a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e032b50 0x56157e031a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55248/0 (socket says 172.21.15.116:55248) 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2815843235 learned_addr learned my addr 172.21.15.116:0/2815843235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5400640 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e033540 0x56157e0440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac6800640 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e04f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2815843235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 msgr2=0x56157e04f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e04f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2815843235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e033540 msgr2=0x56157e0440e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e033540 0x56157e0440e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac5e00640 1 -- 172.21.15.116:0/2815843235 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56157e029b30 con 0x56157e032b50 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.248+0000 7f5ac6800640 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e04f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:10.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.249+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e032b50 0x56157e031a30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80ccc70 tx=0x7f5ab8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb6deb74ed45bc44 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.249+0000 7f5ab6a00640 1 -- 172.21.15.116:0/2815843235 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80cb8e0 con 0x56157e032b50 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.249+0000 7f5ab6a00640 1 -- 172.21.15.116:0/2815843235 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ab80fd040 con 0x56157e032b50 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.249+0000 7f5ab6a00640 1 -- 172.21.15.116:0/2815843235 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80cbae0 con 0x56157e032b50 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 -- 172.21.15.116:0/2815843235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e032b50 msgr2=0x56157e031a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e032b50 0x56157e031a30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80ccc70 tx=0x7f5ab8004040 comp rx=0 tx=0).stop 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 -- 172.21.15.116:0/2815843235 shutdown_connections 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e04f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e033540 0x56157e0440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2815843235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e032b50 0x56157e031a30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 -- 172.21.15.116:0/2815843235 >> 172.21.15.116:0/2815843235 conn(0x56157e02e0c0 msgr2=0x56157e031320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 -- 172.21.15.116:0/2815843235 shutdown_connections 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 -- 172.21.15.116:0/2815843235 wait complete. 2024-03-04T07:42:10.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 -- start start 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.250+0000 7f5acfe60740 1 Processor -- start 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e041ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e0421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5acfe60740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e0431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5acfe60740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81adce0 con 0x56157e032b50 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5acfe60740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81af000 con 0x56157e033540 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5acfe60740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81ae340 con 0x56157e0386b0 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5acfe60740 10 client.0 _ll_register_callbacks cb 0x7f5ac000bdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e041ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5ac5400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e0421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5ac5400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e0421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55250/0 (socket says 172.21.15.116:55250) 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5ac5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e041ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32778/0 (socket says 172.21.15.116:32778) 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5ac5400640 1 -- 172.21.15.116:0/2503164194 learned_addr learned my addr 172.21.15.116:0/2503164194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.251+0000 7f5ac6800640 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e0431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5ac5400640 1 -- 172.21.15.116:0/2503164194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 msgr2=0x56157e0431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5ac5400640 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e0431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5ac5400640 1 -- 172.21.15.116:0/2503164194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 msgr2=0x56157e041ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5ac5400640 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e041ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5ac5400640 1 -- 172.21.15.116:0/2503164194 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56157e0437e0 con 0x56157e033540 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5ac5400640 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e0421e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0011830 tx=0x7f5ab0011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54781fdf6309f259 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5aabe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5aabe00640 1 -- 172.21.15.116:0/2503164194 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0021060 con 0x56157e033540 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5aabe00640 1 -- 172.21.15.116:0/2503164194 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5ab0010cc0 con 0x56157e033540 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ab80fd040 con 0x56157e033540 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5aabe00640 1 -- 172.21.15.116:0/2503164194 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ab001c3f0 con 0x56157e033540 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5acfe60740 10 client.25619 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.252+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56157e0a44a0 con 0x56157e033540 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5aabe00640 1 -- 172.21.15.116:0/2503164194 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ab80fd040 con 0x56157e033540 2024-03-04T07:42:10.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5aabe00640 1 -- 172.21.15.116:0/2503164194 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5ab0026070 con 0x56157e033540 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5acfe60740 10 client.25619 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5acfe60740 10 client.25619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5acfe60740 20 client.25619 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e0a4eb0 0x56157e0a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5acfe60740 4 client.25619 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56157df9f5c0 con 0x56157e0a4eb0 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.253+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e0a4eb0 0x56157e0a72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5ac5e00640 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e0a4eb0 0x56157e0a72d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b245158861ffd702 server_cookie=58dc22c2556cf578 in_seq=0 out_seq=0 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5aabe00640 10 client.25619 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5aabe00640 1 -- 172.21.15.116:0/2503164194 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56157df9f5c0 con 0x56157e0a4eb0 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5aabe00640 10 client.25619 handle_command_reply: tid=0 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 client.25619 shutdown 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 msgr2=0x56157e0421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e0421e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0011830 tx=0x7f5ab0011cf0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 shutdown_connections 2024-03-04T07:42:10.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56157e0a4eb0 0x56157e0a72d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56157e0386b0 0x56157e0431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56157e033540 0x56157e0421e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 --2- 172.21.15.116:0/2503164194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56157e032b50 0x56157e041ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 >> 172.21.15.116:0/2503164194 conn(0x56157e02e0c0 msgr2=0x56157e031bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5aabe00640 0 client.25619 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 shutdown_connections 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 1 -- 172.21.15.116:0/2503164194 wait complete. 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.254+0000 7f5acfe60740 20 client.25619 trim_cache size 0 max 16384 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:10.260 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: "e5545310-b988-4b6b-953e-818a4af7cf32": { 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "e5545310-b988-4b6b-953e-818a4af7cf32", 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:42:10.261 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5aac0a1e00 msgr2=0x7f5aac0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5aac0a1e00 0x7f5aac0a42c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x56157e033da0 tx=0x7f5ac000b2e0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 msgr2=0x7f5ac819def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac819def0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000f80 tx=0x7f5ab8035d10 comp rx=0 tx=0).stop 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 shutdown_connections 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac81097a0 0x7f5ac81a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5aac0a1e00 0x7f5aac0a42c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ac8108de0 0x7f5ac819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 --2- 172.21.15.116:0/2219186458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5ac8107bf0 0x7f5ac819def0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 >> 172.21.15.116:0/2219186458 conn(0x7f5ac8076b90 msgr2=0x7f5ac810c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 shutdown_connections 2024-03-04T07:42:10.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:10.255+0000 7f5acd200640 1 -- 172.21.15.116:0/2219186458 wait complete. 2024-03-04T07:42:10.272 INFO:tasks.cephfs.filesystem:scrub status for tag:e5545310-b988-4b6b-953e-818a4af7cf32 - {'path': '/', 'tag': 'e5545310-b988-4b6b-953e-818a4af7cf32', 'options': 'recursive,force'} 2024-03-04T07:42:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:10 smithi116 ceph-mon[32150]: pgmap v443: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:42:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:10 smithi152 ceph-mon[37154]: pgmap v443: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:42:11.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:10 smithi133 ceph-mon[36505]: pgmap v443: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:42:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:11 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:42:12.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:11 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:12.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:11 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:42:12.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:11 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:12.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:11 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:42:12.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:11 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:12.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:11 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:42:12.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:11 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:11 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:42:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:11 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:11 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:42:12.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:11 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:12 smithi116 ceph-mon[32150]: pgmap v444: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:42:13.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:12 smithi152 ceph-mon[37154]: pgmap v444: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:42:13.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:12 smithi133 ceph-mon[36505]: pgmap v444: 97 pgs: 97 active+clean; 804 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:42:14.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:13 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:42:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:13 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:42:14.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:13 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:42:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:14 smithi116 ceph-mon[32150]: pgmap v445: 97 pgs: 97 active+clean; 862 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:42:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:14 smithi152 ceph-mon[37154]: pgmap v445: 97 pgs: 97 active+clean; 862 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:42:15.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:14 smithi133 ceph-mon[36505]: pgmap v445: 97 pgs: 97 active+clean; 862 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:42:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:16 smithi116 ceph-mon[32150]: pgmap v446: 97 pgs: 97 active+clean; 862 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:42:17.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:16 smithi152 ceph-mon[37154]: pgmap v446: 97 pgs: 97 active+clean; 862 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:42:17.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:16 smithi133 ceph-mon[36505]: pgmap v446: 97 pgs: 97 active+clean; 862 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:42:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.424+0000 7f42bbe00640 1 -- 172.21.15.116:0/888974959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b4108e20 msgr2=0x7f42b41092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.424+0000 7f42bbe00640 1 --2- 172.21.15.116:0/888974959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b4108e20 0x7f42b41092a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009860 tx=0x7f42a002bb70 comp rx=0 tx=0).stop 2024-03-04T07:42:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.424+0000 7f42bbe00640 1 -- 172.21.15.116:0/888974959 shutdown_connections 2024-03-04T07:42:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.424+0000 7f42bbe00640 1 --2- 172.21.15.116:0/888974959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b41097e0 0x7f42b4110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.424+0000 7f42bbe00640 1 --2- 172.21.15.116:0/888974959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b4108e20 0x7f42b41092a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.424+0000 7f42bbe00640 1 --2- 172.21.15.116:0/888974959 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f42b4107c30 0x7f42b4108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.424+0000 7f42bbe00640 1 -- 172.21.15.116:0/888974959 >> 172.21.15.116:0/888974959 conn(0x7f42b4076be0 msgr2=0x7f42b4079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.425+0000 7f42bbe00640 1 -- 172.21.15.116:0/888974959 shutdown_connections 2024-03-04T07:42:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.425+0000 7f42bbe00640 1 -- 172.21.15.116:0/888974959 wait complete. 2024-03-04T07:42:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 Processor -- start 2024-03-04T07:42:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 -- start start 2024-03-04T07:42:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b4107c30 0x7f42b419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f42b4108e20 0x7f42b419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b41097e0 0x7f42b41a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4113890 con 0x7f42b4107c30 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4113710 con 0x7f42b4108e20 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.426+0000 7f42bbe00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4113a10 con 0x7f42b41097e0 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b4107c30 0x7f42b419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f42b4108e20 0x7f42b419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b4107c30 0x7f42b419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42572/0 (socket says 172.21.15.116:42572) 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b9400640 1 -- 172.21.15.116:0/2803874354 learned_addr learned my addr 172.21.15.116:0/2803874354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b9e00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b41097e0 0x7f42b41a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b8a00640 1 -- 172.21.15.116:0/2803874354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b41097e0 msgr2=0x7f42b41a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b8a00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b41097e0 0x7f42b41a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b8a00640 1 -- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b4107c30 msgr2=0x7f42b419dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b8a00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b4107c30 0x7f42b419dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b8a00640 1 -- 172.21.15.116:0/2803874354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b41a5ac0 con 0x7f42b4108e20 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b9e00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b41097e0 0x7f42b41a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b9400640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b4107c30 0x7f42b419dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.427+0000 7f42b8a00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f42b4108e20 0x7f42b419e3e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f42a002c0b0 tx=0x7f42a0032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.428+0000 7f42b2000640 1 -- 172.21.15.116:0/2803874354 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0041020 con 0x7f42b4108e20 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.428+0000 7f42b2000640 1 -- 172.21.15.116:0/2803874354 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42a002c9f0 con 0x7f42b4108e20 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.428+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b41a5d50 con 0x7f42b4108e20 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.428+0000 7f42b2000640 1 -- 172.21.15.116:0/2803874354 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0039040 con 0x7f42b4108e20 2024-03-04T07:42:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.428+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b41a6230 con 0x7f42b4108e20 2024-03-04T07:42:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 Processor -- start 2024-03-04T07:42:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 -- start start 2024-03-04T07:42:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42b2000640 1 -- 172.21.15.116:0/2803874354 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f42a0044020 con 0x7f42b4108e20 2024-03-04T07:42:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc2850 0x564968cc2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cc7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564968c28370 con 0x564968cc83b0 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564968bd1550 con 0x564968cc3240 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42bea61740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564968c24410 con 0x564968cc2850 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42b9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cc7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42b9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cc7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35954/0 (socket says 172.21.15.116:35954) 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.431+0000 7f42b9400640 1 -- 172.21.15.116:0/3708045721 learned_addr learned my addr 172.21.15.116:0/3708045721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42b9e00640 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc2850 0x564968cc2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42b9400640 1 -- 172.21.15.116:0/3708045721 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc2850 msgr2=0x564968cc2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42b9400640 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc2850 0x564968cc2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42b9400640 1 -- 172.21.15.116:0/3708045721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 msgr2=0x564968cca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42b9400640 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42b9400640 1 -- 172.21.15.116:0/3708045721 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564968be53c0 con 0x564968cc3240 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42b9400640 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cc7d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f42a80115c0 tx=0x7f42a8011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51d2eb1015427d27 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42a6600640 1 -- 172.21.15.116:0/3708045721 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8005020 con 0x564968cc3240 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42a6600640 1 -- 172.21.15.116:0/3708045721 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42a80047a0 con 0x564968cc3240 2024-03-04T07:42:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.432+0000 7f42a6600640 1 -- 172.21.15.116:0/3708045721 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a800f9f0 con 0x564968cc3240 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42b2000640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f429c0a1ea0 0x7f429c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 -- 172.21.15.116:0/3708045721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 msgr2=0x564968cc7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cc7d40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f42a80115c0 tx=0x7f42a8011a80 comp rx=0 tx=0).stop 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 -- 172.21.15.116:0/3708045721 shutdown_connections 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cc7d40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 --2- 172.21.15.116:0/3708045721 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc2850 0x564968cc2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 -- 172.21.15.116:0/3708045721 >> 172.21.15.116:0/3708045721 conn(0x564968cbddc0 msgr2=0x564968cbe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42b2000640 1 -- 172.21.15.116:0/2803874354 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f42a00ef600 con 0x7f42b4108e20 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42b9e00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f429c0a1ea0 0x7f429c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 -- 172.21.15.116:0/3708045721 shutdown_connections 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.433+0000 7f42bea61740 1 -- 172.21.15.116:0/3708045721 wait complete. 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 -- start start 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42b9e00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f429c0a1ea0 0x7f429c0a4360 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x564968cc3aa0 tx=0x7f42ac0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 Processor -- start 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968ccd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564968ccb360 con 0x564968cc2850 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564968be49b0 con 0x564968cc3240 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.434+0000 7f42bea61740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564968ccb4e0 con 0x564968cc83b0 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42bea61740 10 client.0 _ll_register_callbacks cb 0x564968c20340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968ccd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968ccd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35964/0 (socket says 172.21.15.116:35964) 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 -- 172.21.15.116:0/421953381 learned_addr learned my addr 172.21.15.116:0/421953381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42590/0 (socket says 172.21.15.116:42590) 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b9e00640 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 -- 172.21.15.116:0/421953381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 msgr2=0x564968cd4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 -- 172.21.15.116:0/421953381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 msgr2=0x564968cccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 -- 172.21.15.116:0/421953381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56496893d210 con 0x564968cc3240 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f42b8a00640 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968ccd0d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f42a00b8cc0 tx=0x7f42a002c130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=362de857272a6905 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f429aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.435+0000 7f429aa00640 1 -- 172.21.15.116:0/421953381 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00b6e60 con 0x564968cc3240 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42a80047a0 con 0x564968cc3240 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f429aa00640 1 -- 172.21.15.116:0/421953381 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f42a00b8960 con 0x564968cc3240 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f429aa00640 1 -- 172.21.15.116:0/421953381 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42a00bbe30 con 0x564968cc3240 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 10 client.25631 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564968a09620 con 0x564968cc3240 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f429aa00640 1 -- 172.21.15.116:0/421953381 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42a80047a0 con 0x564968cc3240 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f429aa00640 1 -- 172.21.15.116:0/421953381 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f42a003c030 con 0x564968cc3240 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 10 client.25631 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 10 client.25631 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 20 client.25631 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968cd8990 0x564968cdada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 4 client.25631 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.436+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564968c358b0 con 0x564968cd8990 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.437+0000 7f42b9400640 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968cd8990 0x564968cdada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.437+0000 7f429aa00640 10 client.25631 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.437+0000 7f42b9400640 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968cd8990 0x564968cdada0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a1febf102f7352c server_cookie=c7919b563297460e in_seq=0 out_seq=0 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f429aa00640 1 -- 172.21.15.116:0/421953381 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564968c358b0 con 0x564968cd8990 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f429aa00640 10 client.25631 handle_command_reply: tid=0 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 client.25631 shutdown 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 msgr2=0x564968ccd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968ccd0d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f42a00b8cc0 tx=0x7f42a002c130 comp rx=0 tx=0).stop 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 shutdown_connections 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968cd8990 0x564968cdada0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968ccd0d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 --2- 172.21.15.116:0/421953381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f429aa00640 0 client.25631 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 >> 172.21.15.116:0/421953381 conn(0x564968cbddc0 msgr2=0x564968cbcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 shutdown_connections 2024-03-04T07:42:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 1 -- 172.21.15.116:0/421953381 wait complete. 2024-03-04T07:42:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.438+0000 7f42bea61740 20 client.25631 trim_cache size 0 max 16384 2024-03-04T07:42:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.453+0000 7f42bea61740 1 Processor -- start 2024-03-04T07:42:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.453+0000 7f42bea61740 1 -- start start 2024-03-04T07:42:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.453+0000 7f42bea61740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc2850 0x564968cca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.453+0000 7f42bea61740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc3240 0x564968cd40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.453+0000 7f42bea61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cbc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42bea61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41acf30 con 0x564968cc83b0 2024-03-04T07:42:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42bea61740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41af510 con 0x564968cc2850 2024-03-04T07:42:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42bea61740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41ad530 con 0x564968cc3240 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cbc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cbc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42592/0 (socket says 172.21.15.116:42592) 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 -- 172.21.15.116:0/1339518702 learned_addr learned my addr 172.21.15.116:0/1339518702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9400640 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc2850 0x564968cca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b8a00640 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc3240 0x564968cd40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 -- 172.21.15.116:0/1339518702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc3240 msgr2=0x564968cd40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc3240 0x564968cd40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 -- 172.21.15.116:0/1339518702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc2850 msgr2=0x564968cca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc2850 0x564968cca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b9e00640 1 -- 172.21.15.116:0/1339518702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564968cbd710 con 0x564968cc83b0 2024-03-04T07:42:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.454+0000 7f42b8a00640 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc3240 0x564968cd40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.455+0000 7f42b9e00640 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cbc750 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42ac01b930 tx=0x7f42ac015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caed1d21fe76b5d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.455+0000 7f42a5c00640 1 -- 172.21.15.116:0/1339518702 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac004b20 con 0x564968cc83b0 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.455+0000 7f42a5c00640 1 -- 172.21.15.116:0/1339518702 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42ac004cc0 con 0x564968cc83b0 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.455+0000 7f42a5c00640 1 -- 172.21.15.116:0/1339518702 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0267e0 con 0x564968cc83b0 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.455+0000 7f42bea61740 1 -- 172.21.15.116:0/1339518702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 msgr2=0x564968cbc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cbc750 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42ac01b930 tx=0x7f42ac015000 comp rx=0 tx=0).stop 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 -- 172.21.15.116:0/1339518702 shutdown_connections 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc83b0 0x564968cbc750 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc3240 0x564968cd40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 --2- 172.21.15.116:0/1339518702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc2850 0x564968cca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 -- 172.21.15.116:0/1339518702 >> 172.21.15.116:0/1339518702 conn(0x564968cbddc0 msgr2=0x564968cbe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 -- 172.21.15.116:0/1339518702 shutdown_connections 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 -- 172.21.15.116:0/1339518702 wait complete. 2024-03-04T07:42:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 -- start start 2024-03-04T07:42:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.456+0000 7f42bea61740 1 Processor -- start 2024-03-04T07:42:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42bea61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cd2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42bea61740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cd25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42bea61740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42bea61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41adb90 con 0x564968cc2850 2024-03-04T07:42:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42bea61740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41aeeb0 con 0x564968cc3240 2024-03-04T07:42:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42bea61740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41ae1f0 con 0x564968cc83b0 2024-03-04T07:42:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42bea61740 10 client.0 _ll_register_callbacks cb 0x7f42a0005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cd25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cd25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:35972/0 (socket says 172.21.15.116:35972) 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 -- 172.21.15.116:0/1884096434 learned_addr learned my addr 172.21.15.116:0/1884096434 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b9400640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cd2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b9e00640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 -- 172.21.15.116:0/1884096434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 msgr2=0x564968cd35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 -- 172.21.15.116:0/1884096434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 msgr2=0x564968cd2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cd2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 -- 172.21.15.116:0/1884096434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564968cd3ba0 con 0x564968cc3240 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b9400640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cd2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b9e00640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f42b8a00640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cd25d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f42a0000c00 tx=0x7f42a0009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a566546d2d1d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.457+0000 7f429aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f429aa00640 1 -- 172.21.15.116:0/1884096434 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00b6e60 con 0x564968cc3240 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f429aa00640 1 -- 172.21.15.116:0/1884096434 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f42a00bb940 con 0x564968cc3240 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42ac004cc0 con 0x564968cc3240 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f429aa00640 1 -- 172.21.15.116:0/1884096434 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42a00bbc50 con 0x564968cc3240 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 10 client.25637 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564968d34b60 con 0x564968cc3240 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f429aa00640 1 -- 172.21.15.116:0/1884096434 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42ac004cc0 con 0x564968cc3240 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f429aa00640 1 -- 172.21.15.116:0/1884096434 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f42a00ef0d0 con 0x564968cc3240 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 10 client.25637 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 10 client.25637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 20 client.25637 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968d35280 0x564968d37670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 4 client.25637 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:42:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.458+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564968c2fc00 con 0x564968d35280 2024-03-04T07:42:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.459+0000 7f42b9400640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968d35280 0x564968d37670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.459+0000 7f42b9400640 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968d35280 0x564968d37670 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=230a6f0777d595e9 server_cookie=3a66852357c6319 in_seq=0 out_seq=0 2024-03-04T07:42:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.459+0000 7f429aa00640 10 client.25637 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f429aa00640 1 -- 172.21.15.116:0/1884096434 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564968c2fc00 con 0x564968d35280 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f429aa00640 10 client.25637 handle_command_reply: tid=0 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 client.25637 shutdown 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 msgr2=0x564968cd25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cd25d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f42a0000c00 tx=0x7f42a0009820 comp rx=0 tx=0).stop 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 shutdown_connections 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564968d35280 0x564968d37670 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564968cc83b0 0x564968cd35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564968cc3240 0x564968cd25d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 --2- 172.21.15.116:0/1884096434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564968cc2850 0x564968cd2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f429aa00640 0 client.25637 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 >> 172.21.15.116:0/1884096434 conn(0x564968cbddc0 msgr2=0x564968cbc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 shutdown_connections 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 1 -- 172.21.15.116:0/1884096434 wait complete. 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.472+0000 7f42bea61740 20 client.25637 trim_cache size 0 max 16384 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:42:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:42:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f429c0a1ea0 msgr2=0x7f429c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f429c0a1ea0 0x7f429c0a4360 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x564968cc3aa0 tx=0x7f42ac0095c0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f42b4108e20 msgr2=0x7f42b419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f42b4108e20 0x7f42b419e3e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f42a002c0b0 tx=0x7f42a0032430 comp rx=0 tx=0).stop 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 shutdown_connections 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f429c0a1ea0 0x7f429c0a4360 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42b41097e0 0x7f42b41a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f42b4108e20 0x7f42b419e3e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 --2- 172.21.15.116:0/2803874354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42b4107c30 0x7f42b419dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 >> 172.21.15.116:0/2803874354 conn(0x7f42b4076be0 msgr2=0x7f42b410c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 shutdown_connections 2024-03-04T07:42:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:17.473+0000 7f42bbe00640 1 -- 172.21.15.116:0/2803874354 wait complete. 2024-03-04T07:42:19.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:18 smithi116 ceph-mon[32150]: pgmap v447: 97 pgs: 97 active+clean; 928 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:42:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:18 smithi152 ceph-mon[37154]: pgmap v447: 97 pgs: 97 active+clean; 928 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:42:19.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:18 smithi133 ceph-mon[36505]: pgmap v447: 97 pgs: 97 active+clean; 928 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:42:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:20 smithi116 ceph-mon[32150]: pgmap v448: 97 pgs: 97 active+clean; 928 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:42:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:20 smithi152 ceph-mon[37154]: pgmap v448: 97 pgs: 97 active+clean; 928 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:42:21.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:20 smithi133 ceph-mon[36505]: pgmap v448: 97 pgs: 97 active+clean; 928 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:42:23.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:22 smithi116 ceph-mon[32150]: pgmap v449: 97 pgs: 97 active+clean; 928 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:42:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:22 smithi152 ceph-mon[37154]: pgmap v449: 97 pgs: 97 active+clean; 928 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:42:23.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:22 smithi133 ceph-mon[36505]: pgmap v449: 97 pgs: 97 active+clean; 928 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:42:25.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:24 smithi116 ceph-mon[32150]: pgmap v450: 97 pgs: 97 active+clean; 1001 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:25.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:24 smithi152 ceph-mon[37154]: pgmap v450: 97 pgs: 97 active+clean; 1001 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:24 smithi133 ceph-mon[36505]: pgmap v450: 97 pgs: 97 active+clean; 1001 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:26 smithi116 ceph-mon[32150]: pgmap v451: 97 pgs: 97 active+clean; 1001 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:42:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:26 smithi152 ceph-mon[37154]: pgmap v451: 97 pgs: 97 active+clean; 1001 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:42:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:26 smithi133 ceph-mon[36505]: pgmap v451: 97 pgs: 97 active+clean; 1001 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:42:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:28 smithi116 ceph-mon[32150]: pgmap v452: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:28 smithi152 ceph-mon[37154]: pgmap v452: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:28 smithi133 ceph-mon[36505]: pgmap v452: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 -- 172.21.15.116:0/1900449305 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124104a20 msgr2=0x7f7124104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 --2- 172.21.15.116:0/1900449305 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124104a20 0x7f7124104ea0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f710c009900 tx=0x7f710c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 -- 172.21.15.116:0/1900449305 shutdown_connections 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 --2- 172.21.15.116:0/1900449305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71241053e0 0x7f712410bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 --2- 172.21.15.116:0/1900449305 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124104a20 0x7f7124104ea0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 --2- 172.21.15.116:0/1900449305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7124103830 0x7f7124103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 -- 172.21.15.116:0/1900449305 >> 172.21.15.116:0/1900449305 conn(0x7f71240fedc0 msgr2=0x7f71241011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 -- 172.21.15.116:0/1900449305 shutdown_connections 2024-03-04T07:42:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.614+0000 7f7129e00640 1 -- 172.21.15.116:0/1900449305 wait complete. 2024-03-04T07:42:30.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 Processor -- start 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 -- start start 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124103830 0x7f7124199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7124104a20 0x7f712419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71241053e0 0x7f71241a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712410f490 con 0x7f7124104a20 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712410f310 con 0x7f7124103830 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.615+0000 7f7129e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712410f610 con 0x7f71241053e0 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124103830 0x7f7124199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124103830 0x7f7124199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51976/0 (socket says 172.21.15.116:51976) 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 -- 172.21.15.116:0/2476125970 learned_addr learned my addr 172.21.15.116:0/2476125970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122000640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7124104a20 0x7f712419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 -- 172.21.15.116:0/2476125970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71241053e0 msgr2=0x7f71241a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71241053e0 0x7f71241a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 -- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7124104a20 msgr2=0x7f712419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7124104a20 0x7f712419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 -- 172.21.15.116:0/2476125970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71241a16f0 con 0x7f7124103830 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7122a00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124103830 0x7f7124199ad0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9c0 tx=0x7f711400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.616+0000 7f7128a00640 1 -- 172.21.15.116:0/2476125970 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711400cde0 con 0x7f7124103830 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.617+0000 7f7128a00640 1 -- 172.21.15.116:0/2476125970 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7114010910 con 0x7f7124103830 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.617+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71241a19e0 con 0x7f7124103830 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.617+0000 7f7128a00640 1 -- 172.21.15.116:0/2476125970 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114018a00 con 0x7f7124103830 2024-03-04T07:42:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.617+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71241a1f80 con 0x7f7124103830 2024-03-04T07:42:30.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f7128a00640 1 -- 172.21.15.116:0/2476125970 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f711401f020 con 0x7f7124103830 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 Processor -- start 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 -- start start 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c815c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c816240 0x56301c81ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c81b3b0 0x56301c81d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56301c77b370 con 0x56301c81b3b0 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56301c724550 con 0x56301c816240 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.620+0000 7f712ca29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56301c777410 con 0x56301c815850 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c816240 0x56301c81ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c81b3b0 0x56301c81d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c816240 0x56301c81ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51980/0 (socket says 172.21.15.116:51980) 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 -- 172.21.15.116:0/4046516379 learned_addr learned my addr 172.21.15.116:0/4046516379 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7123400640 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c815c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 -- 172.21.15.116:0/4046516379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 msgr2=0x56301c815c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c815c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 -- 172.21.15.116:0/4046516379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c81b3b0 msgr2=0x56301c81d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c81b3b0 0x56301c81d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 -- 172.21.15.116:0/4046516379 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56301c7383c0 con 0x56301c816240 2024-03-04T07:42:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7123400640 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c815c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7128a00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f71080a1ea0 0x7f71080a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.621+0000 7f7122a00640 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c816240 0x56301c81ad40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f71140d6290 tx=0x7f71140d6380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0f5cf9c904e6f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f7113e00640 1 -- 172.21.15.116:0/4046516379 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114097010 con 0x56301c816240 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f7113e00640 1 -- 172.21.15.116:0/4046516379 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f711401fb10 con 0x56301c816240 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f7123400640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f71080a1ea0 0x7f71080a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f7113e00640 1 -- 172.21.15.116:0/4046516379 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140b4780 con 0x56301c816240 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f7128a00640 1 -- 172.21.15.116:0/2476125970 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f71140ce040 con 0x7f7124103830 2024-03-04T07:42:30.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f712ca29740 1 -- 172.21.15.116:0/4046516379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c816240 msgr2=0x56301c81ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f7123400640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f71080a1ea0 0x7f71080a4360 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x56301c816aa0 tx=0x7f711c009ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.622+0000 7f712ca29740 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c816240 0x56301c81ad40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f71140d6290 tx=0x7f71140d6380 comp rx=0 tx=0).stop 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 -- 172.21.15.116:0/4046516379 shutdown_connections 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c81b3b0 0x56301c81d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c816240 0x56301c81ad40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 --2- 172.21.15.116:0/4046516379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c815c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 -- 172.21.15.116:0/4046516379 >> 172.21.15.116:0/4046516379 conn(0x56301c810dc0 msgr2=0x56301c8111c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 -- 172.21.15.116:0/4046516379 shutdown_connections 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 -- 172.21.15.116:0/4046516379 wait complete. 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 -- start start 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.623+0000 7f712ca29740 1 Processor -- start 2024-03-04T07:42:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f712ca29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c81fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f712ca29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c8200f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f712ca29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c81b3b0 0x56301c8270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f712ca29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56301c81e360 con 0x56301c816240 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f712ca29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56301c7379b0 con 0x56301c81b3b0 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f712ca29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56301c81e4e0 con 0x56301c815850 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f712ca29740 10 client.0 _ll_register_callbacks cb 0x56301c773340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7122000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c8200f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c81b3b0 0x56301c8270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c81b3b0 0x56301c8270c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51984/0 (socket says 172.21.15.116:51984) 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7122000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c8200f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48946/0 (socket says 172.21.15.116:48946) 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 -- 172.21.15.116:0/594706107 learned_addr learned my addr 172.21.15.116:0/594706107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7122a00640 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c81fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 -- 172.21.15.116:0/594706107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 msgr2=0x56301c81fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c81fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 -- 172.21.15.116:0/594706107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 msgr2=0x56301c8200f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c8200f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.624+0000 7f7123400640 1 -- 172.21.15.116:0/594706107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56301c490210 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f7123400640 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c81b3b0 0x56301c8270c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f711c011c30 tx=0x7f711c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1347f9b8e7982fe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f7110c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f7110c00640 1 -- 172.21.15.116:0/594706107 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c01e620 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f7110c00640 1 -- 172.21.15.116:0/594706107 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f711c01e7c0 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f711401fb10 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f7110c00640 1 -- 172.21.15.116:0/594706107 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f711c00ee90 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f712ca29740 10 client.25649 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.625+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56301c55c620 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f7110c00640 1 -- 172.21.15.116:0/594706107 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f711401fb10 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f7110c00640 1 -- 172.21.15.116:0/594706107 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f711c026a60 con 0x56301c81b3b0 2024-03-04T07:42:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f712ca29740 10 client.25649 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f712ca29740 10 client.25649 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f712ca29740 20 client.25649 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f712ca29740 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c82ba10 0x56301c82de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f712ca29740 4 client.25649 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56301c7888b0 con 0x56301c82ba10 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.626+0000 7f7122a00640 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c82ba10 0x56301c82de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f7122a00640 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c82ba10 0x56301c82de70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17a69e13990e060c server_cookie=2be1670206de7ce3 in_seq=0 out_seq=0 2024-03-04T07:42:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f7110c00640 10 client.25649 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f7110c00640 1 -- 172.21.15.116:0/594706107 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56301c7888b0 con 0x56301c82ba10 2024-03-04T07:42:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f7110c00640 10 client.25649 handle_command_reply: tid=0 2024-03-04T07:42:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 client.25649 shutdown 2024-03-04T07:42:30.632 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c81b3b0 msgr2=0x56301c8270c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c81b3b0 0x56301c8270c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f711c011c30 tx=0x7f711c004a40 comp rx=0 tx=0).stop 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 shutdown_connections 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c82ba10 0x56301c82de70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c81b3b0 0x56301c8270c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c8200f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.627+0000 7f712ca29740 1 --2- 172.21.15.116:0/594706107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c815850 0x56301c81fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.628+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 >> 172.21.15.116:0/594706107 conn(0x56301c810dc0 msgr2=0x56301c80ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.628+0000 7f7110c00640 0 client.25649 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.628+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 shutdown_connections 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.628+0000 7f712ca29740 1 -- 172.21.15.116:0/594706107 wait complete. 2024-03-04T07:42:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.628+0000 7f712ca29740 20 client.25649 trim_cache size 0 max 16384 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 Processor -- start 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 -- start start 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c810dc0 0x56301c814020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c815850 0x56301c827080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c816240 0x56301c814560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a8ce0 con 0x56301c815850 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241ab2c0 con 0x56301c810dc0 2024-03-04T07:42:30.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f712ca29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a92e0 con 0x56301c816240 2024-03-04T07:42:30.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f7122a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c810dc0 0x56301c814020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f7123400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c816240 0x56301c814560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f7122a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c810dc0 0x56301c814020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51990/0 (socket says 172.21.15.116:51990) 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f7122a00640 1 -- 172.21.15.116:0/2270871557 learned_addr learned my addr 172.21.15.116:0/2270871557 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.643+0000 7f7122000640 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c815850 0x56301c827080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7122a00640 1 -- 172.21.15.116:0/2270871557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c816240 msgr2=0x56301c814560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7122a00640 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c816240 0x56301c814560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7122a00640 1 -- 172.21.15.116:0/2270871557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c815850 msgr2=0x56301c827080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7122a00640 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c815850 0x56301c827080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7122a00640 1 -- 172.21.15.116:0/2270871557 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56301c81a7c0 con 0x56301c810dc0 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7123400640 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c816240 0x56301c814560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7122a00640 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c810dc0 0x56301c814020 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f71140d6880 tx=0x7f7114009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2d1cff6ae149914 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7113400640 1 -- 172.21.15.116:0/2270871557 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140b2dd0 con 0x56301c810dc0 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7113400640 1 -- 172.21.15.116:0/2270871557 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f711409b470 con 0x56301c810dc0 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.644+0000 7f7113400640 1 -- 172.21.15.116:0/2270871557 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140b3aa0 con 0x56301c810dc0 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 -- 172.21.15.116:0/2270871557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c810dc0 msgr2=0x56301c814020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c810dc0 0x56301c814020 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f71140d6880 tx=0x7f7114009310 comp rx=0 tx=0).stop 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 -- 172.21.15.116:0/2270871557 shutdown_connections 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c816240 0x56301c814560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c815850 0x56301c827080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 --2- 172.21.15.116:0/2270871557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c810dc0 0x56301c814020 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 -- 172.21.15.116:0/2270871557 >> 172.21.15.116:0/2270871557 conn(0x56301c81aeb0 msgr2=0x56301c817500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 -- 172.21.15.116:0/2270871557 shutdown_connections 2024-03-04T07:42:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 -- 172.21.15.116:0/2270871557 wait complete. 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 -- start start 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.645+0000 7f712ca29740 1 Processor -- start 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f712ca29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c810dc0 0x56301c824e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f712ca29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c815850 0x56301c8253d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f712ca29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c826300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f712ca29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a9940 con 0x56301c816240 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f712ca29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241aac60 con 0x56301c815850 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f712ca29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a9fa0 con 0x56301c810dc0 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f712ca29740 10 client.0 _ll_register_callbacks cb 0x7f711c010f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f7122000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c815850 0x56301c8253d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f7122a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c810dc0 0x56301c824e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f7123400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c826300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f7122000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c815850 0x56301c8253d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51998/0 (socket says 172.21.15.116:51998) 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.646+0000 7f7122000640 1 -- 172.21.15.116:0/4027019100 learned_addr learned my addr 172.21.15.116:0/4027019100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7122000640 1 -- 172.21.15.116:0/4027019100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c810dc0 msgr2=0x56301c824e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7122000640 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c810dc0 0x56301c824e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7122000640 1 -- 172.21.15.116:0/4027019100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 msgr2=0x56301c826300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7122000640 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c826300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7122000640 1 -- 172.21.15.116:0/4027019100 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56301c8268f0 con 0x56301c815850 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7122a00640 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c810dc0 0x56301c824e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7123400640 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c826300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7122000640 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c815850 0x56301c8253d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f710c02c180 tx=0x7f710c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63372e846ac2ac2b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7110c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7110c00640 1 -- 172.21.15.116:0/4027019100 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c005230 con 0x56301c815850 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7110c00640 1 -- 172.21.15.116:0/4027019100 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f710c0053d0 con 0x56301c815850 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f711409b470 con 0x56301c815850 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f7110c00640 1 -- 172.21.15.116:0/4027019100 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f710c0055d0 con 0x56301c815850 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f712ca29740 10 client.25655 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.647+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56301c887a30 con 0x56301c815850 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f7110c00640 1 -- 172.21.15.116:0/4027019100 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f711409b470 con 0x56301c815850 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f7110c00640 1 -- 172.21.15.116:0/4027019100 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f710c003920 con 0x56301c815850 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f712ca29740 10 client.25655 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f712ca29740 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f712ca29740 20 client.25655 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f712ca29740 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c81b3b0 0x56301c88a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f712ca29740 4 client.25655 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:42:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56301c782c00 con 0x56301c81b3b0 2024-03-04T07:42:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f7122a00640 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c81b3b0 0x56301c88a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f7122a00640 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c81b3b0 0x56301c88a440 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb93fbc752a1a1f3 server_cookie=c7935a4997cf12f3 in_seq=0 out_seq=0 2024-03-04T07:42:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.648+0000 7f7110c00640 10 client.25655 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:30.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.661+0000 7f7110c00640 1 -- 172.21.15.116:0/4027019100 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56301c782c00 con 0x56301c81b3b0 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.661+0000 7f7110c00640 10 client.25655 handle_command_reply: tid=0 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.661+0000 7f712ca29740 1 client.25655 shutdown 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.661+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c815850 msgr2=0x56301c8253d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.661+0000 7f712ca29740 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c815850 0x56301c8253d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f710c02c180 tx=0x7f710c004270 comp rx=0 tx=0).stop 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 shutdown_connections 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56301c81b3b0 0x56301c88a440 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56301c816240 0x56301c826300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56301c815850 0x56301c8253d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f7110c00640 0 client.25655 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 --2- 172.21.15.116:0/4027019100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56301c810dc0 0x56301c824e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 >> 172.21.15.116:0/4027019100 conn(0x56301c81aeb0 msgr2=0x56301c817500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 shutdown_connections 2024-03-04T07:42:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 1 -- 172.21.15.116:0/4027019100 wait complete. 2024-03-04T07:42:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.662+0000 7f712ca29740 20 client.25655 trim_cache size 0 max 16384 2024-03-04T07:42:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:42:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:42:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:42:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:42:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f71080a1ea0 msgr2=0x7f71080a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f71080a1ea0 0x7f71080a4360 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x56301c816aa0 tx=0x7f711c009ba0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124103830 msgr2=0x7f7124199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124103830 0x7f7124199ad0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9c0 tx=0x7f711400ee80 comp rx=0 tx=0).stop 2024-03-04T07:42:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 shutdown_connections 2024-03-04T07:42:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f71080a1ea0 0x7f71080a4360 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71241053e0 0x7f71241a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7124104a20 0x7f712419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 --2- 172.21.15.116:0/2476125970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7124103830 0x7f7124199ad0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 >> 172.21.15.116:0/2476125970 conn(0x7f71240fedc0 msgr2=0x7f7124107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 shutdown_connections 2024-03-04T07:42:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:30.663+0000 7f7129e00640 1 -- 172.21.15.116:0/2476125970 wait complete. 2024-03-04T07:42:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:30 smithi116 ceph-mon[32150]: pgmap v453: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:31.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:30 smithi152 ceph-mon[37154]: pgmap v453: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:30 smithi133 ceph-mon[36505]: pgmap v453: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:32 smithi116 ceph-mon[32150]: pgmap v454: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:33.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:32 smithi152 ceph-mon[37154]: pgmap v454: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:33.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:32 smithi133 ceph-mon[36505]: pgmap v454: 97 pgs: 97 active+clean; 1.0 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:42:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:34 smithi116 ceph-mon[32150]: pgmap v455: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:42:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:34 smithi152 ceph-mon[37154]: pgmap v455: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:42:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:34 smithi133 ceph-mon[36505]: pgmap v455: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:42:36.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:35 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:42:36.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:35 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:42:36.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:35 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:42:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:36 smithi116 ceph-mon[32150]: pgmap v456: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:42:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:42:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:42:37.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:42:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:36 smithi152 ceph-mon[37154]: pgmap v456: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:42:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:42:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:42:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:42:37.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:36 smithi133 ceph-mon[36505]: pgmap v456: 97 pgs: 97 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:42:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:42:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:42:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:42:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:38 smithi116 ceph-mon[32150]: pgmap v457: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:42:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:38 smithi152 ceph-mon[37154]: pgmap v457: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:42:39.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:38 smithi133 ceph-mon[36505]: pgmap v457: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:42:40.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:42:40.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:42:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 -- 172.21.15.116:0/2265701836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 msgr2=0x7ff638103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 --2- 172.21.15.116:0/2265701836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638103c10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff628009900 tx=0x7ff62802bc30 comp rx=0 tx=0).stop 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 -- 172.21.15.116:0/2265701836 shutdown_connections 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 --2- 172.21.15.116:0/2265701836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6381053a0 0x7ff63810bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 --2- 172.21.15.116:0/2265701836 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6381049e0 0x7ff638104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 --2- 172.21.15.116:0/2265701836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638103c10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 -- 172.21.15.116:0/2265701836 >> 172.21.15.116:0/2265701836 conn(0x7ff6380fed80 msgr2=0x7ff6381011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 -- 172.21.15.116:0/2265701836 shutdown_connections 2024-03-04T07:42:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.498+0000 7ff63fc00640 1 -- 172.21.15.116:0/2265701836 wait complete. 2024-03-04T07:42:40.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.499+0000 7ff63fc00640 1 Processor -- start 2024-03-04T07:42:40.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63fc00640 1 -- start start 2024-03-04T07:42:40.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63fc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63fc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6381049e0 0x7ff638199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63fc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6381053a0 0x7ff6381a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63fc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63810f4c0 con 0x7ff6381037f0 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63fc00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63810f340 con 0x7ff6381053a0 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63fc00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63810f640 con 0x7ff6381049e0 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47818/0 (socket says 172.21.15.116:47818) 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63dc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6381053a0 0x7ff6381a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff63d200640 1 -- 172.21.15.116:0/499515674 learned_addr learned my addr 172.21.15.116:0/499515674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.500+0000 7ff62fe00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6381049e0 0x7ff638199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63d200640 1 -- 172.21.15.116:0/499515674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6381049e0 msgr2=0x7ff638199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63d200640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6381049e0 0x7ff638199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63d200640 1 -- 172.21.15.116:0/499515674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6381053a0 msgr2=0x7ff6381a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63d200640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6381053a0 0x7ff6381a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63d200640 1 -- 172.21.15.116:0/499515674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6381a16d0 con 0x7ff6381037f0 2024-03-04T07:42:40.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63dc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6381053a0 0x7ff6381a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:40.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63d200640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638199ab0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff628000c00 tx=0x7ff62802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff636000640 1 -- 172.21.15.116:0/499515674 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628043020 con 0x7ff6381037f0 2024-03-04T07:42:40.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff636000640 1 -- 172.21.15.116:0/499515674 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6280054e0 con 0x7ff6381037f0 2024-03-04T07:42:40.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6381a1960 con 0x7ff6381037f0 2024-03-04T07:42:40.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.501+0000 7ff636000640 1 -- 172.21.15.116:0/499515674 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628003c40 con 0x7ff6381037f0 2024-03-04T07:42:40.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.502+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6381a1eb0 con 0x7ff6381037f0 2024-03-04T07:42:40.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 Processor -- start 2024-03-04T07:42:40.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff636000640 1 -- 172.21.15.116:0/499515674 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff628046020 con 0x7ff6381037f0 2024-03-04T07:42:40.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 -- start start 2024-03-04T07:42:40.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6db50 0x55790ea6df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea73040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea75b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55790e99b090 con 0x55790ea736b0 2024-03-04T07:42:40.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55790e9d2d30 con 0x55790ea6e540 2024-03-04T07:42:40.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff64298f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55790e97ba30 con 0x55790ea6db50 2024-03-04T07:42:40.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff63d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea73040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea75b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff63d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea73040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42784/0 (socket says 172.21.15.116:42784) 2024-03-04T07:42:40.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff63d200640 1 -- 172.21.15.116:0/3199839857 learned_addr learned my addr 172.21.15.116:0/3199839857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:40.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.505+0000 7ff63dc00640 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6db50 0x55790ea6df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff63d200640 1 -- 172.21.15.116:0/3199839857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6db50 msgr2=0x55790ea6df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff63d200640 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6db50 0x55790ea6df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff63d200640 1 -- 172.21.15.116:0/3199839857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 msgr2=0x55790ea75b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff63d200640 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea75b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff63d200640 1 -- 172.21.15.116:0/3199839857 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55790e7b17d0 con 0x55790ea6e540 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff63d200640 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea73040 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff6280f3d40 tx=0x7ff6280f3e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2595618bce193eac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff636000640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff61c0a1d80 0x7ff61c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.506+0000 7ff62e000640 1 -- 172.21.15.116:0/3199839857 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280ff070 con 0x55790ea6e540 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff62e000640 1 -- 172.21.15.116:0/3199839857 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6280bd190 con 0x55790ea6e540 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff62e000640 1 -- 172.21.15.116:0/3199839857 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280bccf0 con 0x55790ea6e540 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff636000640 1 -- 172.21.15.116:0/499515674 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff6280efda0 con 0x7ff6381037f0 2024-03-04T07:42:40.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff63dc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff61c0a1d80 0x7ff61c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 -- 172.21.15.116:0/3199839857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 msgr2=0x55790ea73040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea73040 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff6280f3d40 tx=0x7ff6280f3e30 comp rx=0 tx=0).stop 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 -- 172.21.15.116:0/3199839857 shutdown_connections 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea75b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea73040 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 --2- 172.21.15.116:0/3199839857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6db50 0x55790ea6df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff63dc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff61c0a1d80 0x7ff61c0a4240 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x55790ea6eda0 tx=0x7ff630009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 -- 172.21.15.116:0/3199839857 >> 172.21.15.116:0/3199839857 conn(0x55790ea690c0 msgr2=0x55790ea694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 -- 172.21.15.116:0/3199839857 shutdown_connections 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.507+0000 7ff64298f740 1 -- 172.21.15.116:0/3199839857 wait complete. 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.508+0000 7ff64298f740 1 -- start start 2024-03-04T07:42:40.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.508+0000 7ff64298f740 1 Processor -- start 2024-03-04T07:42:40.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.508+0000 7ff64298f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea77bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.508+0000 7ff64298f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea780f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff64298f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea736b0 0x55790ea7f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff64298f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55790e9710c0 con 0x55790ea6db50 2024-03-04T07:42:40.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff64298f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55790e9cedd0 con 0x55790ea6e540 2024-03-04T07:42:40.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff64298f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55790ea76420 con 0x55790ea736b0 2024-03-04T07:42:40.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff64298f740 10 client.0 _ll_register_callbacks cb 0x55790e98e860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea780f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea780f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42798/0 (socket says 172.21.15.116:42798) 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 -- 172.21.15.116:0/1738034765 learned_addr learned my addr 172.21.15.116:0/1738034765 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff63d200640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea77bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff63dc00640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea736b0 0x55790ea7f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 -- 172.21.15.116:0/1738034765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea736b0 msgr2=0x55790ea7f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea736b0 0x55790ea7f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 -- 172.21.15.116:0/1738034765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 msgr2=0x55790ea77bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea77bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 -- 172.21.15.116:0/1738034765 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55790e6e73f0 con 0x55790ea6e540 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff63d200640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea77bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff62fe00640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea780f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff620012720 tx=0x7ff620012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee00ca0f33a5a73 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.509+0000 7ff626a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff626a00640 1 -- 172.21.15.116:0/1738034765 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000f920 con 0x55790ea6e540 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6280bd190 con 0x55790ea6e540 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff626a00640 1 -- 172.21.15.116:0/1738034765 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff62000fac0 con 0x55790ea6e540 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff626a00640 1 -- 172.21.15.116:0/1738034765 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff62000fcc0 con 0x55790ea6e540 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff64298f740 10 client.25667 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55790e6a70f0 con 0x55790ea6e540 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff626a00640 1 -- 172.21.15.116:0/1738034765 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6280bd190 con 0x55790ea6e540 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.510+0000 7ff626a00640 1 -- 172.21.15.116:0/1738034765 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff62001c420 con 0x55790ea6e540 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff64298f740 10 client.25667 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff64298f740 10 client.25667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff64298f740 20 client.25667 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff64298f740 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790ea835c0 0x55790ea85a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff64298f740 4 client.25667 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55790e9e0270 con 0x55790ea835c0 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff63d200640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790ea835c0 0x55790ea85a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff63d200640 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790ea835c0 0x55790ea85a40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40ff8d918f51a9d4 server_cookie=17eb071c2e9587e in_seq=0 out_seq=0 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.511+0000 7ff626a00640 10 client.25667 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff626a00640 1 -- 172.21.15.116:0/1738034765 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55790e9e0270 con 0x55790ea835c0 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff626a00640 10 client.25667 handle_command_reply: tid=0 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 client.25667 shutdown 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 msgr2=0x55790ea780f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea780f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff620012720 tx=0x7ff620012be0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 shutdown_connections 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790ea835c0 0x55790ea85a40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea736b0 0x55790ea7f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6e540 0x55790ea780f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff626a00640 0 client.25667 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:40.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 --2- 172.21.15.116:0/1738034765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea77bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 >> 172.21.15.116:0/1738034765 conn(0x55790ea690c0 msgr2=0x55790ea68360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:40.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 shutdown_connections 2024-03-04T07:42:40.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.512+0000 7ff64298f740 1 -- 172.21.15.116:0/1738034765 wait complete. 2024-03-04T07:42:40.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.513+0000 7ff64298f740 20 client.25667 trim_cache size 0 max 16384 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 Processor -- start 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 -- start start 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6db50 0x55790ea6c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea8a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6381a8cd0 con 0x55790ea736b0 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6381ab2b0 con 0x55790ea6db50 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff64298f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6381a92d0 con 0x55790ea6e540 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff63d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6db50 0x55790ea6c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff63dc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea8a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff63d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6db50 0x55790ea6c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42810/0 (socket says 172.21.15.116:42810) 2024-03-04T07:42:40.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff63d200640 1 -- 172.21.15.116:0/2277109900 learned_addr learned my addr 172.21.15.116:0/2277109900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.530+0000 7ff62fe00640 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff63d200640 1 -- 172.21.15.116:0/2277109900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 msgr2=0x55790ea7f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff63d200640 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff63d200640 1 -- 172.21.15.116:0/2277109900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 msgr2=0x55790ea8a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff63d200640 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea8a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff63d200640 1 -- 172.21.15.116:0/2277109900 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55790ea83360 con 0x55790ea6db50 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff63dc00640 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea8a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff63d200640 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6db50 0x55790ea6c990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff62802ceb0 tx=0x7ff62802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7e698428894c423 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff62d600640 1 -- 172.21.15.116:0/2277109900 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62809a400 con 0x55790ea6db50 2024-03-04T07:42:40.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff62d600640 1 -- 172.21.15.116:0/2277109900 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff628004250 con 0x55790ea6db50 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.531+0000 7ff62d600640 1 -- 172.21.15.116:0/2277109900 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628091460 con 0x55790ea6db50 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 -- 172.21.15.116:0/2277109900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6db50 msgr2=0x55790ea6c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6db50 0x55790ea6c990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff62802ceb0 tx=0x7ff62802bc30 comp rx=0 tx=0).stop 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 -- 172.21.15.116:0/2277109900 shutdown_connections 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea736b0 0x55790ea8a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 --2- 172.21.15.116:0/2277109900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea6db50 0x55790ea6c990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 -- 172.21.15.116:0/2277109900 >> 172.21.15.116:0/2277109900 conn(0x55790ea690c0 msgr2=0x55790ea76fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 -- 172.21.15.116:0/2277109900 shutdown_connections 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 -- 172.21.15.116:0/2277109900 wait complete. 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 -- start start 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.532+0000 7ff64298f740 1 Processor -- start 2024-03-04T07:42:40.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff64298f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea7d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff64298f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff64298f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea736b0 0x55790ea7e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff64298f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6381a9930 con 0x55790ea6db50 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff64298f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6381aac50 con 0x55790ea736b0 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff64298f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6381a9f90 con 0x55790ea6e540 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff64298f740 10 client.0 _ll_register_callbacks cb 0x7ff620009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea7d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63dc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea736b0 0x55790ea7e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea7d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47840/0 (socket says 172.21.15.116:47840) 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 -- 172.21.15.116:0/1375632086 learned_addr learned my addr 172.21.15.116:0/1375632086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff62fe00640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 -- 172.21.15.116:0/1375632086 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 msgr2=0x55790ea7df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 -- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea736b0 msgr2=0x55790ea7e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea736b0 0x55790ea7e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63d200640 1 -- 172.21.15.116:0/1375632086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55790eadef90 con 0x55790ea6db50 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff62fe00640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.533+0000 7ff63dc00640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea736b0 0x55790ea7e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff63d200640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea7d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff6280b8ef0 tx=0x7ff6280bc5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763d6f2a9da6a350 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff626a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff626a00640 1 -- 172.21.15.116:0/1375632086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280ff070 con 0x55790ea6db50 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff626a00640 1 -- 172.21.15.116:0/1375632086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff6280bab30 con 0x55790ea6db50 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55790e9da5c0 con 0x55790ea6db50 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff626a00640 1 -- 172.21.15.116:0/1375632086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff628091650 con 0x55790ea6db50 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff64298f740 10 client.15561 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55790eae02a0 con 0x55790ea6db50 2024-03-04T07:42:40.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.534+0000 7ff626a00640 1 -- 172.21.15.116:0/1375632086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55790e9da5c0 con 0x55790ea6db50 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff626a00640 1 -- 172.21.15.116:0/1375632086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff6280ed8e0 con 0x55790ea6db50 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff64298f740 10 client.15561 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff64298f740 10 client.15561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff64298f740 20 client.15561 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff64298f740 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790eae08c0 0x55790eae2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff64298f740 4 client.15561 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55790e9583a0 con 0x55790eae08c0 2024-03-04T07:42:40.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff62fe00640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790eae08c0 0x55790eae2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff62fe00640 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790eae08c0 0x55790eae2cb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62d633d9ae74f6ff server_cookie=2114fa43cc21873e in_seq=0 out_seq=0 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.535+0000 7ff626a00640 10 client.15561 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff626a00640 1 -- 172.21.15.116:0/1375632086 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55790e9583a0 con 0x55790eae08c0 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff626a00640 10 client.15561 handle_command_reply: tid=0 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 client.15561 shutdown 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 msgr2=0x55790ea7d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea7d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff6280b8ef0 tx=0x7ff6280bc5d0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 shutdown_connections 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55790eae08c0 0x55790eae2cb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55790ea736b0 0x55790ea7e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55790ea6e540 0x55790ea7df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 --2- 172.21.15.116:0/1375632086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55790ea6db50 0x55790ea7d9c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff626a00640 0 client.15561 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 >> 172.21.15.116:0/1375632086 conn(0x55790ea690c0 msgr2=0x55790ea76750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:40.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 shutdown_connections 2024-03-04T07:42:40.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 1 -- 172.21.15.116:0/1375632086 wait complete. 2024-03-04T07:42:40.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.536+0000 7ff64298f740 20 client.15561 trim_cache size 0 max 16384 2024-03-04T07:42:40.542 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:40.542 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:42:40.542 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:42:40.542 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff61c0a1d80 msgr2=0x7ff61c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff61c0a1d80 0x7ff61c0a4240 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x55790ea6eda0 tx=0x7ff630009500 comp rx=0 tx=0).stop 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 msgr2=0x7ff638199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638199ab0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff628000c00 tx=0x7ff62802c650 comp rx=0 tx=0).stop 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 shutdown_connections 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6381053a0 0x7ff6381a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff61c0a1d80 0x7ff61c0a4240 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6381049e0 0x7ff638199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 --2- 172.21.15.116:0/499515674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6381037f0 0x7ff638199ab0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 >> 172.21.15.116:0/499515674 conn(0x7ff6380fed80 msgr2=0x7ff638107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:40.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 shutdown_connections 2024-03-04T07:42:40.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.537+0000 7ff63fc00640 1 -- 172.21.15.116:0/499515674 wait complete. 2024-03-04T07:42:40.554 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:42:40.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.754+0000 7f5db5000640 1 -- 172.21.15.116:0/420388293 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 msgr2=0x7f5db0073a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.754+0000 7f5db5000640 1 --2- 172.21.15.116:0/420388293 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 0x7f5db0073a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009ca0 tx=0x7f5da002c180 comp rx=0 tx=0).stop 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.755+0000 7f5db5000640 1 -- 172.21.15.116:0/420388293 shutdown_connections 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.755+0000 7f5db5000640 1 --2- 172.21.15.116:0/420388293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db01092a0 0x7f5db010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.755+0000 7f5db5000640 1 --2- 172.21.15.116:0/420388293 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5db0073fd0 0x7f5db0074430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.755+0000 7f5db5000640 1 --2- 172.21.15.116:0/420388293 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 0x7f5db0073a00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.755+0000 7f5db5000640 1 -- 172.21.15.116:0/420388293 >> 172.21.15.116:0/420388293 conn(0x7f5db00fc960 msgr2=0x7f5db00fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.755+0000 7f5db5000640 1 -- 172.21.15.116:0/420388293 shutdown_connections 2024-03-04T07:42:40.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.755+0000 7f5db5000640 1 -- 172.21.15.116:0/420388293 wait complete. 2024-03-04T07:42:40.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 Processor -- start 2024-03-04T07:42:40.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 -- start start 2024-03-04T07:42:40.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db0073fd0 0x7f5db0199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 0x7f5db0199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5db01092a0 0x7f5db01a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db010f260 con 0x7f5db0073fd0 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db010f0e0 con 0x7f5db01092a0 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5db5000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db010f3e0 con 0x7f5db00755b0 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5daf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db0073fd0 0x7f5db0199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5daf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db0073fd0 0x7f5db0199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47856/0 (socket says 172.21.15.116:47856) 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5daf400640 1 -- 172.21.15.116:0/2293672290 learned_addr learned my addr 172.21.15.116:0/2293672290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5dafe00640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5db01092a0 0x7f5db01a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.756+0000 7f5daea00640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 0x7f5db0199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5daf400640 1 -- 172.21.15.116:0/2293672290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 msgr2=0x7f5db0199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5daf400640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 0x7f5db0199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5daf400640 1 -- 172.21.15.116:0/2293672290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5db01092a0 msgr2=0x7f5db01a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5daf400640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5db01092a0 0x7f5db01a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:40.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5daf400640 1 -- 172.21.15.116:0/2293672290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db01a1570 con 0x7f5db0073fd0 2024-03-04T07:42:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5daf400640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db0073fd0 0x7f5db0199950 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009dc0 tx=0x7f5da00093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5da7e00640 1 -- 172.21.15.116:0/2293672290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da0039040 con 0x7f5db0073fd0 2024-03-04T07:42:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5da7e00640 1 -- 172.21.15.116:0/2293672290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5da0044070 con 0x7f5db0073fd0 2024-03-04T07:42:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db01a1800 con 0x7f5db0073fd0 2024-03-04T07:42:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5da7e00640 1 -- 172.21.15.116:0/2293672290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da0004330 con 0x7f5db0073fd0 2024-03-04T07:42:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.757+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db01a1cf0 con 0x7f5db0073fd0 2024-03-04T07:42:40.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.759+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db0108060 con 0x7f5db0073fd0 2024-03-04T07:42:40.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.760+0000 7f5da7e00640 1 -- 172.21.15.116:0/2293672290 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5da00044d0 con 0x7f5db0073fd0 2024-03-04T07:42:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.762+0000 7f5da7e00640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d940a1d80 0x7f5d940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.762+0000 7f5da7e00640 1 -- 172.21.15.116:0/2293672290 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5da00f02c0 con 0x7f5db0073fd0 2024-03-04T07:42:40.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.762+0000 7f5daea00640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d940a1d80 0x7f5d940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:40.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.762+0000 7f5daea00640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d940a1d80 0x7f5d940a4240 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5db019ae60 tx=0x7f5d9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:40.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:40.766+0000 7f5da7e00640 1 -- 172.21.15.116:0/2293672290 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5da00bd6b0 con 0x7f5db0073fd0 2024-03-04T07:42:41.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.003+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5db002cff0 con 0x7f5db0073fd0 2024-03-04T07:42:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.005+0000 7f5da7e00640 1 -- 172.21.15.116:0/2293672290 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f5da00bcfb0 con 0x7f5db0073fd0 2024-03-04T07:42:41.006 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:42:41.006 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:42:41.006 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:42:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:40 smithi116 ceph-mon[32150]: pgmap v458: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.007+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d940a1d80 msgr2=0x7f5d940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.007+0000 7f5db5000640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d940a1d80 0x7f5d940a4240 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5db019ae60 tx=0x7f5d9c005f90 comp rx=0 tx=0).stop 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.007+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db0073fd0 msgr2=0x7f5db0199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.007+0000 7f5db5000640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db0073fd0 0x7f5db0199950 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009dc0 tx=0x7f5da00093e0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 shutdown_connections 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5db01092a0 0x7f5db01a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5d940a1d80 0x7f5d940a4240 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5db00755b0 0x7f5db0199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 --2- 172.21.15.116:0/2293672290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db0073fd0 0x7f5db0199950 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 >> 172.21.15.116:0/2293672290 conn(0x7f5db00fc960 msgr2=0x7f5db0105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 shutdown_connections 2024-03-04T07:42:41.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.008+0000 7f5db5000640 1 -- 172.21.15.116:0/2293672290 wait complete. 2024-03-04T07:42:41.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:40 smithi152 ceph-mon[37154]: pgmap v458: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:42:41.028 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:42:41.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:42:41.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:40 smithi133 ceph-mon[36505]: pgmap v458: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:42:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 -- 172.21.15.116:0/3158959430 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01049e0 msgr2=0x7f48d0104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 --2- 172.21.15.116:0/3158959430 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01049e0 0x7f48d0104e60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009900 tx=0x7f48bc02bc40 comp rx=0 tx=0).stop 2024-03-04T07:42:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 -- 172.21.15.116:0/3158959430 shutdown_connections 2024-03-04T07:42:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 --2- 172.21.15.116:0/3158959430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01053a0 0x7f48d010bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 --2- 172.21.15.116:0/3158959430 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01049e0 0x7f48d0104e60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 --2- 172.21.15.116:0/3158959430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d01037f0 0x7f48d0103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 -- 172.21.15.116:0/3158959430 >> 172.21.15.116:0/3158959430 conn(0x7f48d00fed80 msgr2=0x7f48d01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 -- 172.21.15.116:0/3158959430 shutdown_connections 2024-03-04T07:42:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.242+0000 7f48d7600640 1 -- 172.21.15.116:0/3158959430 wait complete. 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 Processor -- start 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 -- start start 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01037f0 0x7f48d0199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d01049e0 0x7f48d019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01053a0 0x7f48d01a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d010f470 con 0x7f48d01037f0 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d010f2f0 con 0x7f48d01053a0 2024-03-04T07:42:41.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d7600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d010f5f0 con 0x7f48d01049e0 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01037f0 0x7f48d0199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01037f0 0x7f48d0199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47876/0 (socket says 172.21.15.116:47876) 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d01049e0 0x7f48d019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d4c00640 1 -- 172.21.15.116:0/3424227033 learned_addr learned my addr 172.21.15.116:0/3424227033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.244+0000 7f48d5600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01053a0 0x7f48d01a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cfe00640 1 -- 172.21.15.116:0/3424227033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01053a0 msgr2=0x7f48d01a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cfe00640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01053a0 0x7f48d01a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cfe00640 1 -- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01037f0 msgr2=0x7f48d0199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cfe00640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01037f0 0x7f48d0199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cfe00640 1 -- 172.21.15.116:0/3424227033 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d01a1730 con 0x7f48d01049e0 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48d4c00640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01037f0 0x7f48d0199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48d5600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01053a0 0x7f48d01a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:41.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cfe00640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d01049e0 0x7f48d019a050 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009230 tx=0x7f48bc02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cd600640 1 -- 172.21.15.116:0/3424227033 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc041070 con 0x7f48d01049e0 2024-03-04T07:42:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cd600640 1 -- 172.21.15.116:0/3424227033 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48bc004450 con 0x7f48d01049e0 2024-03-04T07:42:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48cd600640 1 -- 172.21.15.116:0/3424227033 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc003e50 con 0x7f48d01049e0 2024-03-04T07:42:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.245+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d01a19c0 con 0x7f48d01049e0 2024-03-04T07:42:41.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.246+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d01a1f00 con 0x7f48d01049e0 2024-03-04T07:42:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.249+0000 7f48da3ca740 1 Processor -- start 2024-03-04T07:42:41.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48da3ca740 1 -- start start 2024-03-04T07:42:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d504b0 0x561b45d508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48cd600640 1 -- 172.21.15.116:0/3424227033 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f48bc044070 con 0x7f48d01049e0 2024-03-04T07:42:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48da3ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b45cacd60 con 0x561b45d504b0 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48da3ca740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b45c55cf0 con 0x561b45d56010 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.250+0000 7f48da3ca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b45ca8e00 con 0x561b45d50ea0 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48d5600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d504b0 0x561b45d508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42870/0 (socket says 172.21.15.116:42870) 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 -- 172.21.15.116:0/940172862 learned_addr learned my addr 172.21.15.116:0/940172862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48d4c00640 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 -- 172.21.15.116:0/940172862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 msgr2=0x561b45d559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 -- 172.21.15.116:0/940172862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d504b0 msgr2=0x561b45d508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d504b0 0x561b45d508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 -- 172.21.15.116:0/940172862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b459c13f0 con 0x561b45d56010 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48d5600640 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d504b0 0x561b45d508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48cfe00640 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d584a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0f5a90 tx=0x7f48bc0f5b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4986603706b6b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:41.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48c1c00640 1 -- 172.21.15.116:0/940172862 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc0fe070 con 0x561b45d56010 2024-03-04T07:42:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48c1c00640 1 -- 172.21.15.116:0/940172862 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48bc0f7d60 con 0x561b45d56010 2024-03-04T07:42:41.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.251+0000 7f48c1c00640 1 -- 172.21.15.116:0/940172862 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc0f1910 con 0x561b45d56010 2024-03-04T07:42:41.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48cd600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f48b80a1fa0 0x7f48b80a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 -- 172.21.15.116:0/940172862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 msgr2=0x561b45d584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d584a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0f5a90 tx=0x7f48bc0f5b80 comp rx=0 tx=0).stop 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 -- 172.21.15.116:0/940172862 shutdown_connections 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d584a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 --2- 172.21.15.116:0/940172862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d504b0 0x561b45d508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 -- 172.21.15.116:0/940172862 >> 172.21.15.116:0/940172862 conn(0x561b45d43210 msgr2=0x561b45d43610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48cd600640 1 -- 172.21.15.116:0/3424227033 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f48bc0ef720 con 0x7f48d01049e0 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48d5600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f48b80a1fa0 0x7f48b80a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 -- 172.21.15.116:0/940172862 shutdown_connections 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.252+0000 7f48da3ca740 1 -- 172.21.15.116:0/940172862 wait complete. 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.253+0000 7f48da3ca740 1 -- start start 2024-03-04T07:42:41.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.253+0000 7f48d5600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f48b80a1fa0 0x7f48b80a4460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x561b45d51700 tx=0x7f48c8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.253+0000 7f48da3ca740 1 Processor -- start 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.253+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d61ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48da3ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b45d58d80 con 0x561b45d50ea0 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48da3ca740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b45c4b220 con 0x561b45d56010 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48da3ca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b45d58f00 con 0x561b45d504b0 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48da3ca740 10 client.0 _ll_register_callbacks cb 0x561b459c0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d5600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d61ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d4c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d5600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d61ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42886/0 (socket says 172.21.15.116:42886) 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47902/0 (socket says 172.21.15.116:47902) 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d5600640 1 -- 172.21.15.116:0/1620240536 learned_addr learned my addr 172.21.15.116:0/1620240536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d4c00640 1 -- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 msgr2=0x561b45d61ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d4c00640 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d61ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d4c00640 1 -- 172.21.15.116:0/1620240536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 msgr2=0x561b45d5ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d4c00640 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d4c00640 1 -- 172.21.15.116:0/1620240536 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b45a8b7d0 con 0x561b45d504b0 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48cfe00640 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.254+0000 7f48d5600640 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d61ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48d4c00640 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5a720 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x561b45d512a0 tx=0x7f48c400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0fdb0abbac03aef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48b6000640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48b6000640 1 -- 172.21.15.116:0/1620240536 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c4012110 con 0x561b45d504b0 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48bc0f7d60 con 0x561b45d504b0 2024-03-04T07:42:41.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48b6000640 1 -- 172.21.15.116:0/1620240536 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f48c4004050 con 0x561b45d504b0 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48b6000640 1 -- 172.21.15.116:0/1620240536 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48c40122b0 con 0x561b45d504b0 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48da3ca740 10 client.25651 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561b45d65130 con 0x561b45d504b0 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.255+0000 7f48b6000640 1 -- 172.21.15.116:0/1620240536 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48bc0f7d60 con 0x561b45d504b0 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48b6000640 1 -- 172.21.15.116:0/1620240536 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f48c4010430 con 0x561b45d504b0 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48da3ca740 10 client.25651 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48da3ca740 10 client.25651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48da3ca740 20 client.25651 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48da3ca740 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45d66820 0x561b45d68c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48da3ca740 4 client.25651 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b45cba2a0 con 0x561b45d66820 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.256+0000 7f48cfe00640 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45d66820 0x561b45d68c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.257+0000 7f48b6000640 10 client.25651 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:41.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.257+0000 7f48cfe00640 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45d66820 0x561b45d68c10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=412210a9c260d6e7 server_cookie=d2e4de6793faa93b in_seq=0 out_seq=0 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.257+0000 7f48b6000640 1 -- 172.21.15.116:0/1620240536 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561b45cba2a0 con 0x561b45d66820 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.257+0000 7f48b6000640 10 client.25651 handle_command_reply: tid=0 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.257+0000 7f48da3ca740 1 client.25651 shutdown 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.257+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 msgr2=0x561b45d5a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.257+0000 7f48da3ca740 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5a720 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x561b45d512a0 tx=0x7f48c400f790 comp rx=0 tx=0).stop 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 shutdown_connections 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45d66820 0x561b45d68c10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d61ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48b6000640 0 client.25651 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 --2- 172.21.15.116:0/1620240536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5a720 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 >> 172.21.15.116:0/1620240536 conn(0x561b45d43210 msgr2=0x561b45d424b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 shutdown_connections 2024-03-04T07:42:41.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 1 -- 172.21.15.116:0/1620240536 wait complete. 2024-03-04T07:42:41.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.258+0000 7f48da3ca740 20 client.25651 trim_cache size 0 max 16384 2024-03-04T07:42:41.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.271+0000 7f48da3ca740 1 Processor -- start 2024-03-04T07:42:41.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.271+0000 7f48da3ca740 1 -- start start 2024-03-04T07:42:41.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d504b0 0x561b45d6d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d61c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d56010 0x561b45d6da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48da3ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a4830 con 0x561b45d56010 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48da3ca740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a6e10 con 0x561b45d504b0 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48da3ca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a4e30 con 0x561b45d50ea0 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d5600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d56010 0x561b45d6da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d4c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d504b0 0x561b45d6d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d5600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d56010 0x561b45d6da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47904/0 (socket says 172.21.15.116:47904) 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d4c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d504b0 0x561b45d6d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42890/0 (socket says 172.21.15.116:42890) 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d5600640 1 -- 172.21.15.116:0/3611380540 learned_addr learned my addr 172.21.15.116:0/3611380540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48cfe00640 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d61c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d4c00640 1 -- 172.21.15.116:0/3611380540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 msgr2=0x561b45d61c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d4c00640 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d61c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d4c00640 1 -- 172.21.15.116:0/3611380540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d56010 msgr2=0x561b45d6da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d4c00640 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d56010 0x561b45d6da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.272+0000 7f48d4c00640 1 -- 172.21.15.116:0/3611380540 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b45d59000 con 0x561b45d504b0 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.273+0000 7f48d4c00640 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d504b0 0x561b45d6d4d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f48c4001eb0 tx=0x7f48c40128a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae8e06a0a274cc22 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.273+0000 7f48c1200640 1 -- 172.21.15.116:0/3611380540 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c4000e60 con 0x561b45d504b0 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.273+0000 7f48c1200640 1 -- 172.21.15.116:0/3611380540 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48c4010570 con 0x561b45d504b0 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.273+0000 7f48c1200640 1 -- 172.21.15.116:0/3611380540 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c40038e0 con 0x561b45d504b0 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 -- 172.21.15.116:0/3611380540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d504b0 msgr2=0x561b45d6d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d504b0 0x561b45d6d4d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f48c4001eb0 tx=0x7f48c40128a0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 -- 172.21.15.116:0/3611380540 shutdown_connections 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d56010 0x561b45d6da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d50ea0 0x561b45d61c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 --2- 172.21.15.116:0/3611380540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d504b0 0x561b45d6d4d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 -- 172.21.15.116:0/3611380540 >> 172.21.15.116:0/3611380540 conn(0x561b45d43210 msgr2=0x561b45d46d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 -- 172.21.15.116:0/3611380540 shutdown_connections 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 -- 172.21.15.116:0/3611380540 wait complete. 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 -- start start 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.274+0000 7f48da3ca740 1 Processor -- start 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48da3ca740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d60ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48da3ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a5490 con 0x561b45d50ea0 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48da3ca740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a67b0 con 0x561b45d56010 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48da3ca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01a5af0 con 0x561b45d504b0 2024-03-04T07:42:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48da3ca740 10 client.0 _ll_register_callbacks cb 0x7f48c4012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:59106/0 (socket says 172.21.15.116:59106) 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 -- 172.21.15.116:0/2638735821 learned_addr learned my addr 172.21.15.116:0/2638735821 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d5600640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d60ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48cfe00640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 -- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 msgr2=0x561b45d60ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d60ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 -- 172.21.15.116:0/2638735821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 msgr2=0x561b45d5fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d4c00640 1 -- 172.21.15.116:0/2638735821 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b45d610c0 con 0x561b45d504b0 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.275+0000 7f48d5600640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d60ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48d4c00640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5f580 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48c4009600 tx=0x7f48c400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59cca3e3ad066e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48b6000640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:42:41.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48b6000640 1 -- 172.21.15.116:0/2638735821 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c4000e60 con 0x561b45d504b0 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48b6000640 1 -- 172.21.15.116:0/2638735821 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f48c4005070 con 0x561b45d504b0 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48c4010570 con 0x561b45d504b0 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48b6000640 1 -- 172.21.15.116:0/2638735821 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48c40052b0 con 0x561b45d504b0 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48da3ca740 10 client.25657 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.276+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561b45dc22d0 con 0x561b45d504b0 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48b6000640 1 -- 172.21.15.116:0/2638735821 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48c4010570 con 0x561b45d504b0 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48b6000640 1 -- 172.21.15.116:0/2638735821 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f48c40375f0 con 0x561b45d504b0 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48da3ca740 10 client.25657 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48da3ca740 10 client.25657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48da3ca740 20 client.25657 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48da3ca740 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45dc2b10 0x561b45dc4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48da3ca740 4 client.25657 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561b45cb45f0 con 0x561b45dc2b10 2024-03-04T07:42:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.277+0000 7f48cfe00640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45dc2b10 0x561b45dc4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48cfe00640 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45dc2b10 0x561b45dc4f30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3a3ec80cf6cafb server_cookie=85882b9b47b0055 in_seq=0 out_seq=0 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48b6000640 10 client.25657 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48b6000640 1 -- 172.21.15.116:0/2638735821 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561b45cb45f0 con 0x561b45dc2b10 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48b6000640 10 client.25657 handle_command_reply: tid=0 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 client.25657 shutdown 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 msgr2=0x561b45d5f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5f580 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48c4009600 tx=0x7f48c400f790 comp rx=0 tx=0).stop 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 shutdown_connections 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561b45dc2b10 0x561b45dc4f30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561b45d56010 0x561b45d60ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b45d50ea0 0x561b45d5fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 --2- 172.21.15.116:0/2638735821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561b45d504b0 0x561b45d5f580 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 >> 172.21.15.116:0/2638735821 conn(0x561b45d43210 msgr2=0x561b45d41c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48b6000640 0 client.25657 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 shutdown_connections 2024-03-04T07:42:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 1 -- 172.21.15.116:0/2638735821 wait complete. 2024-03-04T07:42:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.278+0000 7f48da3ca740 20 client.25657 trim_cache size 0 max 16384 2024-03-04T07:42:41.284 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f48b80a1fa0 msgr2=0x7f48b80a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f48b80a1fa0 0x7f48b80a4460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x561b45d51700 tx=0x7f48c8009500 comp rx=0 tx=0).stop 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d01049e0 msgr2=0x7f48d019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d01049e0 0x7f48d019a050 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009230 tx=0x7f48bc02c660 comp rx=0 tx=0).stop 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 shutdown_connections 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48d01053a0 0x7f48d01a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f48b80a1fa0 0x7f48b80a4460 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d01049e0 0x7f48d019a050 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 --2- 172.21.15.116:0/3424227033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48d01037f0 0x7f48d0199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 >> 172.21.15.116:0/3424227033 conn(0x7f48d00fed80 msgr2=0x7f48d0107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 shutdown_connections 2024-03-04T07:42:41.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:41.279+0000 7f48d7600640 1 -- 172.21.15.116:0/3424227033 wait complete. 2024-03-04T07:42:42.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:41 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2293672290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:42.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:41 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2293672290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:41 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2293672290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:42.298 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:42:42.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:42:42.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:42:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 -- 172.21.15.116:0/1169211841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd41097a0 msgr2=0x7f9dd4110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 --2- 172.21.15.116:0/1169211841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd41097a0 0x7f9dd4110030 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc009ca0 tx=0x7f9dcc02c100 comp rx=0 tx=0).stop 2024-03-04T07:42:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 -- 172.21.15.116:0/1169211841 shutdown_connections 2024-03-04T07:42:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 --2- 172.21.15.116:0/1169211841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd41097a0 0x7f9dd4110030 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 --2- 172.21.15.116:0/1169211841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd4108de0 0x7f9dd4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 --2- 172.21.15.116:0/1169211841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dd4107bf0 0x7f9dd4108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 -- 172.21.15.116:0/1169211841 >> 172.21.15.116:0/1169211841 conn(0x7f9dd4076b90 msgr2=0x7f9dd4078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 -- 172.21.15.116:0/1169211841 shutdown_connections 2024-03-04T07:42:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.521+0000 7f9dd9400640 1 -- 172.21.15.116:0/1169211841 wait complete. 2024-03-04T07:42:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.522+0000 7f9dd9400640 1 Processor -- start 2024-03-04T07:42:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.522+0000 7f9dd9400640 1 -- start start 2024-03-04T07:42:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.522+0000 7f9dd9400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dd4107bf0 0x7f9dd419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.522+0000 7f9dd9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd4108de0 0x7f9dd419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd41097a0 0x7f9dd41a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd9400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4113870 con 0x7f9dd41097a0 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd9400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41136f0 con 0x7f9dd4108de0 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd9400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41139f0 con 0x7f9dd4107bf0 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd41097a0 0x7f9dd41a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dcbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd4108de0 0x7f9dd419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd41097a0 0x7f9dd41a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47930/0 (socket says 172.21.15.116:47930) 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 -- 172.21.15.116:0/691151120 learned_addr learned my addr 172.21.15.116:0/691151120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dcbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd4108de0 0x7f9dd419e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42908/0 (socket says 172.21.15.116:42908) 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd2800640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dd4107bf0 0x7f9dd419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 -- 172.21.15.116:0/691151120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dd4107bf0 msgr2=0x7f9dd419def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dd4107bf0 0x7f9dd419def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 -- 172.21.15.116:0/691151120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd4108de0 msgr2=0x7f9dd419e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd4108de0 0x7f9dd419e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dd3200640 1 -- 172.21.15.116:0/691151120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd41a5b10 con 0x7f9dd41097a0 2024-03-04T07:42:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.523+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd4108de0 0x7f9dd419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.524+0000 7f9dd3200640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd41097a0 0x7f9dd41a54a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc037040 tx=0x7f9dcc035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.524+0000 7f9dcb400640 1 -- 172.21.15.116:0/691151120 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc039040 con 0x7f9dd41097a0 2024-03-04T07:42:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.524+0000 7f9dcb400640 1 -- 172.21.15.116:0/691151120 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dcc044070 con 0x7f9dd41097a0 2024-03-04T07:42:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.524+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd41a5da0 con 0x7f9dd41097a0 2024-03-04T07:42:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.524+0000 7f9dcb400640 1 -- 172.21.15.116:0/691151120 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc004460 con 0x7f9dd41097a0 2024-03-04T07:42:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.524+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd41a6230 con 0x7f9dd41097a0 2024-03-04T07:42:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.527+0000 7f9dcb400640 1 -- 172.21.15.116:0/691151120 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9dcc004600 con 0x7f9dd41097a0 2024-03-04T07:42:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.527+0000 7f9ddc1f0740 1 Processor -- start 2024-03-04T07:42:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.527+0000 7f9ddc1f0740 1 -- start start 2024-03-04T07:42:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f6d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6dac0 0x562d38f725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f750c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38e80e20 con 0x562d38f72c30 2024-03-04T07:42:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38ed2a20 con 0x562d38f6dac0 2024-03-04T07:42:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38e7bb00 con 0x562d38f6d0d0 2024-03-04T07:42:42.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6dac0 0x562d38f725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6dac0 0x562d38f725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42912/0 (socket says 172.21.15.116:42912) 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 -- 172.21.15.116:0/810544133 learned_addr learned my addr 172.21.15.116:0/810544133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f6d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f750c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 -- 172.21.15.116:0/810544133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 msgr2=0x562d38f6d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f6d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 -- 172.21.15.116:0/810544133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 msgr2=0x562d38f750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f750c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.528+0000 7f9dd2800640 1 -- 172.21.15.116:0/810544133 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d38e90e10 con 0x562d38f6dac0 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dcb400640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db80a1d80 0x7f9db80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dd2800640 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6dac0 0x562d38f725c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009ca0 tx=0x7f9dc4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a986395ab04f57b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dcb400640 1 -- 172.21.15.116:0/691151120 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9dcc0f0420 con 0x7f9dd41097a0 2024-03-04T07:42:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dc3400640 1 -- 172.21.15.116:0/810544133 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4004220 con 0x562d38f6dac0 2024-03-04T07:42:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db80a1d80 0x7f9db80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dc3400640 1 -- 172.21.15.116:0/810544133 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc40043c0 con 0x562d38f6dac0 2024-03-04T07:42:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.529+0000 7f9dc3400640 1 -- 172.21.15.116:0/810544133 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4019970 con 0x562d38f6dac0 2024-03-04T07:42:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/810544133 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6dac0 msgr2=0x562d38f725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db80a1d80 0x7f9db80a4240 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x562d38f6e320 tx=0x7f9dbc004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6dac0 0x562d38f725c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009ca0 tx=0x7f9dc4011810 comp rx=0 tx=0).stop 2024-03-04T07:42:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/810544133 shutdown_connections 2024-03-04T07:42:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6dac0 0x562d38f725c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/810544133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f6d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/810544133 >> 172.21.15.116:0/810544133 conn(0x562d38f68640 msgr2=0x562d38f68a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/810544133 shutdown_connections 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.530+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/810544133 wait complete. 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 -- start start 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 Processor -- start 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f77530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f77a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f72c30 0x562d38f7ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38e8ef40 con 0x562d38f6dac0 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38eceac0 con 0x562d38f72c30 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38f75be0 con 0x562d38f6d0d0 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9ddc1f0740 10 client.0 _ll_register_callbacks cb 0x562d38ecdaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9dd3200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f72c30 0x562d38f7ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9dd3200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f72c30 0x562d38f7ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42914/0 (socket says 172.21.15.116:42914) 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9dd3200640 1 -- 172.21.15.116:0/4275064967 learned_addr learned my addr 172.21.15.116:0/4275064967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f77a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.531+0000 7f9dd2800640 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f77530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9dd3200640 1 -- 172.21.15.116:0/4275064967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 msgr2=0x562d38f77530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9dd3200640 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f77530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9dd3200640 1 -- 172.21.15.116:0/4275064967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 msgr2=0x562d38f77a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9dd3200640 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f77a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9dd3200640 1 -- 172.21.15.116:0/4275064967 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562d38caf7e0 con 0x562d38f72c30 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9dd3200640 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f72c30 0x562d38f7ea40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc035fd0 tx=0x7f9dcc0bcc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1272c4700fe888d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9db7e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9db7e00640 1 -- 172.21.15.116:0/4275064967 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc0bd110 con 0x562d38f72c30 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9db7e00640 1 -- 172.21.15.116:0/4275064967 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9dcc0bd370 con 0x562d38f72c30 2024-03-04T07:42:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.532+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9dc40043c0 con 0x562d38f72c30 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9db7e00640 1 -- 172.21.15.116:0/4275064967 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9dcc0c1260 con 0x562d38f72c30 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 10 client.25709 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562d38d0ff70 con 0x562d38f72c30 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9db7e00640 1 -- 172.21.15.116:0/4275064967 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc40043c0 con 0x562d38f72c30 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9db7e00640 1 -- 172.21.15.116:0/4275064967 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9dcc03c030 con 0x562d38f72c30 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 10 client.25709 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 10 client.25709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 20 client.25709 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38f83370 0x562d38f857b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 4 client.25709 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d38edff60 con 0x562d38f83370 2024-03-04T07:42:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.533+0000 7f9dd2800640 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38f83370 0x562d38f857b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.534+0000 7f9dd2800640 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38f83370 0x562d38f857b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed97abca4ac40de2 server_cookie=4420238323779958 in_seq=0 out_seq=0 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.534+0000 7f9db7e00640 10 client.25709 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9db7e00640 1 -- 172.21.15.116:0/4275064967 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562d38edff60 con 0x562d38f83370 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9db7e00640 10 client.25709 handle_command_reply: tid=0 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 client.25709 shutdown 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f72c30 msgr2=0x562d38f7ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f72c30 0x562d38f7ea40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc035fd0 tx=0x7f9dcc0bcc50 comp rx=0 tx=0).stop 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 shutdown_connections 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38f83370 0x562d38f857b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f72c30 0x562d38f7ea40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f77a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4275064967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6d0d0 0x562d38f77530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9db7e00640 0 client.25709 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 >> 172.21.15.116:0/4275064967 conn(0x562d38f68640 msgr2=0x562d38f69890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 shutdown_connections 2024-03-04T07:42:42.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4275064967 wait complete. 2024-03-04T07:42:42.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.535+0000 7f9ddc1f0740 20 client.25709 trim_cache size 0 max 16384 2024-03-04T07:42:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 Processor -- start 2024-03-04T07:42:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 -- start start 2024-03-04T07:42:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f67a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6dac0 0x562d38f7ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f6bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41acea0 con 0x562d38f72c30 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41af480 con 0x562d38f6d0d0 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41ad4a0 con 0x562d38f6dac0 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9dd3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f6bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f67a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f67a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42926/0 (socket says 172.21.15.116:42926) 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9dd3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f6bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47948/0 (socket says 172.21.15.116:47948) 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9dd2800640 1 -- 172.21.15.116:0/3757863421 learned_addr learned my addr 172.21.15.116:0/3757863421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.553+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6dac0 0x562d38f7ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dd2800640 1 -- 172.21.15.116:0/3757863421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6dac0 msgr2=0x562d38f7ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dd2800640 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6dac0 0x562d38f7ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dd2800640 1 -- 172.21.15.116:0/3757863421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 msgr2=0x562d38f6bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dd2800640 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f6bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dd2800640 1 -- 172.21.15.116:0/3757863421 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d38ba7170 con 0x562d38f6d0d0 2024-03-04T07:42:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dd2800640 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f67a60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009710 tx=0x7f9dc4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83e1d3b22e9d9a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dc2a00640 1 -- 172.21.15.116:0/3757863421 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4031070 con 0x562d38f6d0d0 2024-03-04T07:42:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dc2a00640 1 -- 172.21.15.116:0/3757863421 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc4003ad0 con 0x562d38f6d0d0 2024-03-04T07:42:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.554+0000 7f9dc2a00640 1 -- 172.21.15.116:0/3757863421 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc402b780 con 0x562d38f6d0d0 2024-03-04T07:42:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/3757863421 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 msgr2=0x562d38f67a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f67a60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009710 tx=0x7f9dc4011810 comp rx=0 tx=0).stop 2024-03-04T07:42:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/3757863421 shutdown_connections 2024-03-04T07:42:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f72c30 0x562d38f6bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f6dac0 0x562d38f7ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/3757863421 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f67a60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/3757863421 >> 172.21.15.116:0/3757863421 conn(0x562d38f68640 msgr2=0x562d38f6b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/3757863421 shutdown_connections 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.555+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/3757863421 wait complete. 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 -- start start 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 Processor -- start 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f7e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f7ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f72c30 0x562d38f7d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41adb00 con 0x562d38f6dac0 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41aee20 con 0x562d38f6d0d0 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41ae160 con 0x562d38f72c30 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.556+0000 7f9ddc1f0740 10 client.0 _ll_register_callbacks cb 0x7f9dcc0b9480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dcbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f7ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f7e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f7e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42940/0 (socket says 172.21.15.116:42940) 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 -- 172.21.15.116:0/4128260139 learned_addr learned my addr 172.21.15.116:0/4128260139 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dcbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f7ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47958/0 (socket says 172.21.15.116:47958) 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd3200640 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f72c30 0x562d38f7d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 -- 172.21.15.116:0/4128260139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f72c30 msgr2=0x562d38f7d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f72c30 0x562d38f7d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 -- 172.21.15.116:0/4128260139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 msgr2=0x562d38f7ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f7ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 -- 172.21.15.116:0/4128260139 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562d38f7dc00 con 0x562d38f6d0d0 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f7ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9dd2800640 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f7e880 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009250 tx=0x7f9dc4005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c46742f59d294f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9db7e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.557+0000 7f9db7e00640 1 -- 172.21.15.116:0/4128260139 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc403c060 con 0x562d38f6d0d0 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9db7e00640 1 -- 172.21.15.116:0/4128260139 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9dc40035f0 con 0x562d38f6d0d0 2024-03-04T07:42:42.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9dc4003ad0 con 0x562d38f6d0d0 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9db7e00640 1 -- 172.21.15.116:0/4128260139 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9dc4031070 con 0x562d38f6d0d0 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 10 client.25715 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562d38fdfc90 con 0x562d38f6d0d0 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9db7e00640 1 -- 172.21.15.116:0/4128260139 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc4003ad0 con 0x562d38f6d0d0 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9db7e00640 1 -- 172.21.15.116:0/4128260139 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9dc4041070 con 0x562d38f6d0d0 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 10 client.25715 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 10 client.25715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 20 client.25715 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38fe0450 0x562d38fe2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 4 client.25715 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:42:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.558+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562d38eda2b0 con 0x562d38fe0450 2024-03-04T07:42:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.559+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38fe0450 0x562d38fe2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.559+0000 7f9dcbe00640 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38fe0450 0x562d38fe2870 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec3c6012aea1014d server_cookie=5676a55eb4ecc9c0 in_seq=0 out_seq=0 2024-03-04T07:42:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.559+0000 7f9db7e00640 10 client.25715 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9db7e00640 1 -- 172.21.15.116:0/4128260139 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562d38eda2b0 con 0x562d38fe0450 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9db7e00640 10 client.25715 handle_command_reply: tid=0 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 client.25715 shutdown 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 msgr2=0x562d38f7e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f7e880 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009250 tx=0x7f9dc4005dc0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 shutdown_connections 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562d38fe0450 0x562d38fe2870 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38f72c30 0x562d38f7d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d38f6dac0 0x562d38f7ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9db7e00640 0 client.25715 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 --2- 172.21.15.116:0/4128260139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562d38f6d0d0 0x562d38f7e880 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 >> 172.21.15.116:0/4128260139 conn(0x562d38f68640 msgr2=0x562d38f6c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 shutdown_connections 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.560+0000 7f9ddc1f0740 1 -- 172.21.15.116:0/4128260139 wait complete. 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9ddc1f0740 20 client.25715 trim_cache size 0 max 16384 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "1459b486-7dc4-41d2-8490-7650bbaf5bb0", 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:42:42.566 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db80a1d80 msgr2=0x7f9db80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db80a1d80 0x7f9db80a4240 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x562d38f6e320 tx=0x7f9dbc004730 comp rx=0 tx=0).stop 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd41097a0 msgr2=0x7f9dd41a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd41097a0 0x7f9dd41a54a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc037040 tx=0x7f9dcc035d90 comp rx=0 tx=0).stop 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 shutdown_connections 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dd41097a0 0x7f9dd41a54a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9dd4108de0 0x7f9dd419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9db80a1d80 0x7f9db80a4240 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 --2- 172.21.15.116:0/691151120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dd4107bf0 0x7f9dd419def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 >> 172.21.15.116:0/691151120 conn(0x7f9dd4076b90 msgr2=0x7f9dd410c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.561+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 shutdown_connections 2024-03-04T07:42:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.562+0000 7f9dd9400640 1 -- 172.21.15.116:0/691151120 wait complete. 2024-03-04T07:42:42.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:42:42.580 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.775+0000 7f874aa00640 1 -- 172.21.15.116:0/3180119682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87441096a0 msgr2=0x7f874410ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.775+0000 7f874aa00640 1 --2- 172.21.15.116:0/3180119682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87441096a0 0x7f874410ff30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f873c009ca0 tx=0x7f873c02c0f0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.775+0000 7f874aa00640 1 -- 172.21.15.116:0/3180119682 shutdown_connections 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.775+0000 7f874aa00640 1 --2- 172.21.15.116:0/3180119682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87441096a0 0x7f874410ff30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.775+0000 7f874aa00640 1 --2- 172.21.15.116:0/3180119682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8744108ce0 0x7f8744109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.775+0000 7f874aa00640 1 --2- 172.21.15.116:0/3180119682 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 0x7f8744107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.775+0000 7f874aa00640 1 -- 172.21.15.116:0/3180119682 >> 172.21.15.116:0/3180119682 conn(0x7f8744076aa0 msgr2=0x7f8744078ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.776+0000 7f874aa00640 1 -- 172.21.15.116:0/3180119682 shutdown_connections 2024-03-04T07:42:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.776+0000 7f874aa00640 1 -- 172.21.15.116:0/3180119682 wait complete. 2024-03-04T07:42:42.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.776+0000 7f874aa00640 1 Processor -- start 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874aa00640 1 -- start start 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 0x7f874419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8744108ce0 0x7f874419e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874aa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87441096a0 0x7f87441a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8744113750 con 0x7f8744108ce0 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874aa00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441135d0 con 0x7f8744107af0 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874aa00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441138d0 con 0x7f87441096a0 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f8748c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8744108ce0 0x7f874419e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f8749600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 0x7f874419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f8749600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 0x7f874419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42962/0 (socket says 172.21.15.116:42962) 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f8749600640 1 -- 172.21.15.116:0/485329665 learned_addr learned my addr 172.21.15.116:0/485329665 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.777+0000 7f874a000640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87441096a0 0x7f87441a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8749600640 1 -- 172.21.15.116:0/485329665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87441096a0 msgr2=0x7f87441a52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8749600640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87441096a0 0x7f87441a52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8749600640 1 -- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8744108ce0 msgr2=0x7f874419e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8749600640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8744108ce0 0x7f874419e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8749600640 1 -- 172.21.15.116:0/485329665 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87441a5940 con 0x7f8744107af0 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8748c00640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8744108ce0 0x7f874419e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8749600640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 0x7f874419dd30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f873800e9c0 tx=0x7f873800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8742000640 1 -- 172.21.15.116:0/485329665 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873800cd80 con 0x7f8744107af0 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8742000640 1 -- 172.21.15.116:0/485329665 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87380035f0 con 0x7f8744107af0 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87441a5c30 con 0x7f8744107af0 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f8742000640 1 -- 172.21.15.116:0/485329665 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738010690 con 0x7f8744107af0 2024-03-04T07:42:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.778+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87441a6180 con 0x7f8744107af0 2024-03-04T07:42:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.781+0000 7f8742000640 1 -- 172.21.15.116:0/485329665 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f87380037a0 con 0x7f8744107af0 2024-03-04T07:42:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 Processor -- start 2024-03-04T07:42:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 -- start start 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b960764130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960769200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960769870 0x55b96076bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b960691090 con 0x55b960764700 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9606c8d30 con 0x55b960763d10 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874ede4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b960671a30 con 0x55b960769870 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f8749600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960769200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b960764130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b960764130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42976/0 (socket says 172.21.15.116:42976) 2024-03-04T07:42:42.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f8748c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960769870 0x55b96076bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.782+0000 7f874a000640 1 -- 172.21.15.116:0/946487783 learned_addr learned my addr 172.21.15.116:0/946487783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f874a000640 1 -- 172.21.15.116:0/946487783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960769870 msgr2=0x55b96076bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f874a000640 1 --2- 172.21.15.116:0/946487783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960769870 0x55b96076bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f874a000640 1 -- 172.21.15.116:0/946487783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 msgr2=0x55b960769200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f874a000640 1 --2- 172.21.15.116:0/946487783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960769200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f874a000640 1 -- 172.21.15.116:0/946487783 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9604a77d0 con 0x55b960763d10 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f8742000640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f872c0a1d80 0x7f872c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f874a000640 1 --2- 172.21.15.116:0/946487783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b960764130 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f873c02c700 tx=0x7f873c0323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb7e6b81c3921ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f8742000640 1 -- 172.21.15.116:0/485329665 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f87380ce210 con 0x7f8744107af0 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f8749600640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f872c0a1d80 0x7f872c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f8736600640 1 -- 172.21.15.116:0/946487783 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c043070 con 0x55b960763d10 2024-03-04T07:42:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f8736600640 1 -- 172.21.15.116:0/946487783 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f873c02cca0 con 0x55b960763d10 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.783+0000 7f8736600640 1 -- 172.21.15.116:0/946487783 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c039040 con 0x55b960763d10 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 -- 172.21.15.116:0/946487783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 msgr2=0x55b960764130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f8749600640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f872c0a1d80 0x7f872c0a4240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8738097840 tx=0x7f87380d1c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 --2- 172.21.15.116:0/946487783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b960764130 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f873c02c700 tx=0x7f873c0323d0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 -- 172.21.15.116:0/946487783 shutdown_connections 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 --2- 172.21.15.116:0/946487783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960769870 0x55b96076bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 --2- 172.21.15.116:0/946487783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960769200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 --2- 172.21.15.116:0/946487783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b960764130 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 -- 172.21.15.116:0/946487783 >> 172.21.15.116:0/946487783 conn(0x55b96075f280 msgr2=0x55b96075f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 -- 172.21.15.116:0/946487783 shutdown_connections 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.784+0000 7f874ede4740 1 -- 172.21.15.116:0/946487783 wait complete. 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 -- start start 2024-03-04T07:42:42.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 Processor -- start 2024-03-04T07:42:42.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b96076e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960764700 0x55b96076e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960769870 0x55b9607755e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9606670c0 con 0x55b960769870 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9606c4dd0 con 0x55b960763d10 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b96076c7c0 con 0x55b960764700 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.785+0000 7f874ede4740 10 client.0 _ll_register_callbacks cb 0x55b960684860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b96076e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b96076e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42988/0 (socket says 172.21.15.116:42988) 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 -- 172.21.15.116:0/1013123461 learned_addr learned my addr 172.21.15.116:0/1013123461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960769870 0x55b9607755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f8748c00640 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960764700 0x55b96076e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 -- 172.21.15.116:0/1013123461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960764700 msgr2=0x55b96076e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960764700 0x55b96076e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 -- 172.21.15.116:0/1013123461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960769870 msgr2=0x55b9607755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960769870 0x55b9607755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 -- 172.21.15.116:0/1013123461 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9603dd3f0 con 0x55b960763d10 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f874a000640 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b96076e060 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f873c009620 tx=0x7f873c02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22f1468790d660cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.786+0000 7f872aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f872aa00640 1 -- 172.21.15.116:0/1013123461 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c039070 con 0x55b960763d10 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f872aa00640 1 -- 172.21.15.116:0/1013123461 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f873c004040 con 0x55b960763d10 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f873c02cca0 con 0x55b960763d10 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f872aa00640 1 -- 172.21.15.116:0/1013123461 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f873c043040 con 0x55b960763d10 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f874ede4740 10 client.25727 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b96039d0f0 con 0x55b960763d10 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f872aa00640 1 -- 172.21.15.116:0/1013123461 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f873c02cca0 con 0x55b960763d10 2024-03-04T07:42:42.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.787+0000 7f872aa00640 1 -- 172.21.15.116:0/1013123461 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f873c048730 con 0x55b960763d10 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f874ede4740 10 client.25727 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f874ede4740 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f874ede4740 20 client.25727 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f874ede4740 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b960779740 0x55b96077bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f874ede4740 4 client.25727 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9606d6270 con 0x55b960779740 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f8748c00640 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b960779740 0x55b96077bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f8748c00640 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b960779740 0x55b96077bb30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68160218597095f2 server_cookie=9b5b699faf7bd327 in_seq=0 out_seq=0 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.788+0000 7f872aa00640 10 client.25727 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f872aa00640 1 -- 172.21.15.116:0/1013123461 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b9606d6270 con 0x55b960779740 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f872aa00640 10 client.25727 handle_command_reply: tid=0 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 client.25727 shutdown 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 msgr2=0x55b96076e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b96076e060 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f873c009620 tx=0x7f873c02e040 comp rx=0 tx=0).stop 2024-03-04T07:42:42.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 shutdown_connections 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b960779740 0x55b96077bb30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960769870 0x55b9607755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f872aa00640 0 client.25727 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960764700 0x55b96076e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 --2- 172.21.15.116:0/1013123461 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960763d10 0x55b96076e060 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.789+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 >> 172.21.15.116:0/1013123461 conn(0x55b96075f280 msgr2=0x55b96075e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.790+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 shutdown_connections 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.790+0000 7f874ede4740 1 -- 172.21.15.116:0/1013123461 wait complete. 2024-03-04T07:42:42.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.790+0000 7f874ede4740 20 client.25727 trim_cache size 0 max 16384 2024-03-04T07:42:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.807+0000 7f874ede4740 1 Processor -- start 2024-03-04T07:42:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874ede4740 1 -- start start 2024-03-04T07:42:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874ede4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b96076e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874ede4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b9607755a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874ede4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b96075dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874ede4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441ace00 con 0x55b960764700 2024-03-04T07:42:42.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874ede4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441af3e0 con 0x55b960769870 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874ede4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441ad400 con 0x55b960763d10 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f8748c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b9607755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f874a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b96075dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f8748c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b9607755a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48000/0 (socket says 172.21.15.116:48000) 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.808+0000 7f8748c00640 1 -- 172.21.15.116:0/1323277197 learned_addr learned my addr 172.21.15.116:0/1323277197 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f874a000640 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b96076e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8748c00640 1 -- 172.21.15.116:0/1323277197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 msgr2=0x55b96076e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8748c00640 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b96076e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8748c00640 1 -- 172.21.15.116:0/1323277197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 msgr2=0x55b96075dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8748c00640 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b96075dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8748c00640 1 -- 172.21.15.116:0/1323277197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b96075fff0 con 0x55b960764700 2024-03-04T07:42:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8748c00640 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b9607755a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55b96077c240 tx=0x7f873000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3484377cdb790 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8735c00640 1 -- 172.21.15.116:0/1323277197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730004020 con 0x55b960764700 2024-03-04T07:42:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8735c00640 1 -- 172.21.15.116:0/1323277197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8730024030 con 0x55b960764700 2024-03-04T07:42:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.809+0000 7f8735c00640 1 -- 172.21.15.116:0/1323277197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873001d650 con 0x55b960764700 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 -- 172.21.15.116:0/1323277197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 msgr2=0x55b9607755a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b9607755a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x55b96077c240 tx=0x7f873000b390 comp rx=0 tx=0).stop 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 -- 172.21.15.116:0/1323277197 shutdown_connections 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b96075dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b9607755a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 --2- 172.21.15.116:0/1323277197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b96076e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 -- 172.21.15.116:0/1323277197 >> 172.21.15.116:0/1323277197 conn(0x55b96075f280 msgr2=0x55b960762d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 -- 172.21.15.116:0/1323277197 shutdown_connections 2024-03-04T07:42:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 -- 172.21.15.116:0/1323277197 wait complete. 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.810+0000 7f874ede4740 1 -- start start 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 1 Processor -- start 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b960774fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960775500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b960773680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441ada60 con 0x55b960764700 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441aed80 con 0x55b960769870 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441ae0c0 con 0x55b960763d10 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874ede4740 10 client.0 _ll_register_callbacks cb 0x7f873c003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f8748c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960775500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f8748c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960775500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48002/0 (socket says 172.21.15.116:48002) 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f8748c00640 1 -- 172.21.15.116:0/1537102676 learned_addr learned my addr 172.21.15.116:0/1537102676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874a000640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b960773680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.811+0000 7f874a000640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b960774fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f8748c00640 1 -- 172.21.15.116:0/1537102676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 msgr2=0x55b960774fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f8748c00640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b960774fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f8748c00640 1 -- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 msgr2=0x55b960773680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f8748c00640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b960773680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f8748c00640 1 -- 172.21.15.116:0/1537102676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b960773eb0 con 0x55b960764700 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f874a000640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b960774fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f8748c00640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960775500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8730011650 tx=0x7f87300058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37a53577a05a9ad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f872aa00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f872aa00640 1 -- 172.21.15.116:0/1537102676 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873002d030 con 0x55b960764700 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f872aa00640 1 -- 172.21.15.116:0/1537102676 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f87300201f0 con 0x55b960764700 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.812+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8730024030 con 0x55b960764700 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f872aa00640 1 -- 172.21.15.116:0/1537102676 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8730004020 con 0x55b960764700 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 10 client.15618 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b9607d59f0 con 0x55b960764700 2024-03-04T07:42:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f872aa00640 1 -- 172.21.15.116:0/1537102676 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8730024030 con 0x55b960764700 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f872aa00640 1 -- 172.21.15.116:0/1537102676 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f87300053f0 con 0x55b960764700 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 10 client.15618 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 10 client.15618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 20 client.15618 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b9607d6370 0x55b9607d8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 4 client.15618 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b9606d05c0 con 0x55b9607d6370 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.813+0000 7f874a000640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b9607d6370 0x55b9607d8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874a000640 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b9607d6370 0x55b9607d8790 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d766838938f4cfaf server_cookie=64ba634afeb1c43f in_seq=0 out_seq=0 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f872aa00640 10 client.15618 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f872aa00640 1 -- 172.21.15.116:0/1537102676 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b9606d05c0 con 0x55b9607d6370 2024-03-04T07:42:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f872aa00640 10 client.15618 handle_command_reply: tid=0 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 client.15618 shutdown 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 msgr2=0x55b960775500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960775500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8730011650 tx=0x7f87300058e0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 shutdown_connections 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b9607d6370 0x55b9607d8790 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b960769870 0x55b960773680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b960764700 0x55b960775500 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f872aa00640 0 client.15618 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 --2- 172.21.15.116:0/1537102676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b960763d10 0x55b960774fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 >> 172.21.15.116:0/1537102676 conn(0x55b96075f280 msgr2=0x55b960762d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 shutdown_connections 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 1 -- 172.21.15.116:0/1537102676 wait complete. 2024-03-04T07:42:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.814+0000 7f874ede4740 20 client.15618 trim_cache size 0 max 16384 2024-03-04T07:42:42.822 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:42:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:42:42.822 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:42.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f872c0a1d80 msgr2=0x7f872c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f872c0a1d80 0x7f872c0a4240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8738097840 tx=0x7f87380d1c50 comp rx=0 tx=0).stop 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 msgr2=0x7f874419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 0x7f874419dd30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f873800e9c0 tx=0x7f873800ee80 comp rx=0 tx=0).stop 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 shutdown_connections 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f872c0a1d80 0x7f872c0a4240 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87441096a0 0x7f87441a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8744108ce0 0x7f874419e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 --2- 172.21.15.116:0/485329665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8744107af0 0x7f874419dd30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 >> 172.21.15.116:0/485329665 conn(0x7f8744076aa0 msgr2=0x7f874410bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 shutdown_connections 2024-03-04T07:42:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:42.815+0000 7f874aa00640 1 -- 172.21.15.116:0/485329665 wait complete. 2024-03-04T07:42:42.834 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:42:42.834 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:42:42.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:42 smithi116 ceph-mon[32150]: pgmap v459: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:42:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:42 smithi152 ceph-mon[37154]: pgmap v459: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:42:43.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 -- 172.21.15.116:0/3433151022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001016e0 msgr2=0x7fb800101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 --2- 172.21.15.116:0/3433151022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001016e0 0x7fb800101b60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009900 tx=0x7fb7e802bc30 comp rx=0 tx=0).stop 2024-03-04T07:42:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 -- 172.21.15.116:0/3433151022 shutdown_connections 2024-03-04T07:42:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 --2- 172.21.15.116:0/3433151022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8001020a0 0x7fb800108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 --2- 172.21.15.116:0/3433151022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001016e0 0x7fb800101b60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 --2- 172.21.15.116:0/3433151022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 0x7fb800100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 -- 172.21.15.116:0/3433151022 >> 172.21.15.116:0/3433151022 conn(0x7fb8000fba60 msgr2=0x7fb8000fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 -- 172.21.15.116:0/3433151022 shutdown_connections 2024-03-04T07:42:43.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.039+0000 7fb807400640 1 -- 172.21.15.116:0/3433151022 wait complete. 2024-03-04T07:42:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 Processor -- start 2024-03-04T07:42:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 -- start start 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 0x7fb800197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8001016e0 0x7fb800197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001020a0 0x7fb80019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80010c1d0 con 0x7fb8001016e0 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80010c050 con 0x7fb8001020a0 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.041+0000 7fb807400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80010c350 con 0x7fb8001004f0 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 0x7fb800197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 0x7fb800197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:59272/0 (socket says 172.21.15.116:59272) 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 -- 172.21.15.116:0/70507167 learned_addr learned my addr 172.21.15.116:0/70507167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb805400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001020a0 0x7fb80019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb7f7e00640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8001016e0 0x7fb800197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 -- 172.21.15.116:0/70507167 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001020a0 msgr2=0x7fb80019eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001020a0 0x7fb80019eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 -- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8001016e0 msgr2=0x7fb800197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8001016e0 0x7fb800197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb804a00640 1 -- 172.21.15.116:0/70507167 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80019f540 con 0x7fb8001004f0 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.042+0000 7fb805400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001020a0 0x7fb80019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.043+0000 7fb804a00640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 0x7fb800197920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d770 tx=0x7fb7f000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.043+0000 7fb7fe000640 1 -- 172.21.15.116:0/70507167 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0017070 con 0x7fb8001004f0 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.043+0000 7fb7fe000640 1 -- 172.21.15.116:0/70507167 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7f0009c30 con 0x7fb8001004f0 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.043+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80019f830 con 0x7fb8001004f0 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.043+0000 7fb7fe000640 1 -- 172.21.15.116:0/70507167 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0003dd0 con 0x7fb8001004f0 2024-03-04T07:42:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.043+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80019fd50 con 0x7fb8001004f0 2024-03-04T07:42:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.044+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80019ffb0 con 0x7fb8001004f0 2024-03-04T07:42:43.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.046+0000 7fb7fe000640 1 -- 172.21.15.116:0/70507167 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb7f0003fb0 con 0x7fb8001004f0 2024-03-04T07:42:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.047+0000 7fb7fe000640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7e40a1d80 0x7fb7e40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.047+0000 7fb7fe000640 1 -- 172.21.15.116:0/70507167 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb7f00cd6a0 con 0x7fb8001004f0 2024-03-04T07:42:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.048+0000 7fb7f7e00640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7e40a1d80 0x7fb7e40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.048+0000 7fb7f7e00640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7e40a1d80 0x7fb7e40a4240 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7e802c240 tx=0x7fb7e802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.052+0000 7fb7fe000640 1 -- 172.21.15.116:0/70507167 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb7f0009de0 con 0x7fb8001004f0 2024-03-04T07:42:43.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:42 smithi133 ceph-mon[36505]: pgmap v459: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:42:43.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.298+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8001a07e0 con 0x7fb8001004f0 2024-03-04T07:42:43.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.301+0000 7fb7fe000640 1 -- 172.21.15.116:0/70507167 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fb7f009a440 con 0x7fb8001004f0 2024-03-04T07:42:43.301 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:42:43.301 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:42:43.301 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:42:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.302+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7e40a1d80 msgr2=0x7fb7e40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.302+0000 7fb807400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7e40a1d80 0x7fb7e40a4240 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb7e802c240 tx=0x7fb7e802df90 comp rx=0 tx=0).stop 2024-03-04T07:42:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.302+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 msgr2=0x7fb800197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.302+0000 7fb807400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 0x7fb800197920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d770 tx=0x7fb7f000dc30 comp rx=0 tx=0).stop 2024-03-04T07:42:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 shutdown_connections 2024-03-04T07:42:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb8001020a0 0x7fb80019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8001016e0 0x7fb800197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7e40a1d80 0x7fb7e40a4240 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 --2- 172.21.15.116:0/70507167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8001004f0 0x7fb800197920 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 >> 172.21.15.116:0/70507167 conn(0x7fb8000fba60 msgr2=0x7fb8000fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 shutdown_connections 2024-03-04T07:42:43.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.303+0000 7fb807400640 1 -- 172.21.15.116:0/70507167 wait complete. 2024-03-04T07:42:43.324 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:42:43.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.538+0000 7fe75b800640 1 -- 172.21.15.116:0/3325154979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 msgr2=0x7fe754104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.538+0000 7fe75b800640 1 --2- 172.21.15.116:0/3325154979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe754104e60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe740009900 tx=0x7fe74002bc40 comp rx=0 tx=0).stop 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.538+0000 7fe75b800640 1 -- 172.21.15.116:0/3325154979 shutdown_connections 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.538+0000 7fe75b800640 1 --2- 172.21.15.116:0/3325154979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7541053a0 0x7fe75410bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.538+0000 7fe75b800640 1 --2- 172.21.15.116:0/3325154979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe754104e60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.538+0000 7fe75b800640 1 --2- 172.21.15.116:0/3325154979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7541037f0 0x7fe754103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.538+0000 7fe75b800640 1 -- 172.21.15.116:0/3325154979 >> 172.21.15.116:0/3325154979 conn(0x7fe7540fed80 msgr2=0x7fe7541011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.539+0000 7fe75b800640 1 -- 172.21.15.116:0/3325154979 shutdown_connections 2024-03-04T07:42:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.539+0000 7fe75b800640 1 -- 172.21.15.116:0/3325154979 wait complete. 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 Processor -- start 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 -- start start 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7541037f0 0x7fe754199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe75419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7541053a0 0x7fe7541a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75410f450 con 0x7fe7541053a0 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75410f2d0 con 0x7fe7541049e0 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.540+0000 7fe75b800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75410f5d0 con 0x7fe7541037f0 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe75419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe759800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7541053a0 0x7fe7541a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe75419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43028/0 (socket says 172.21.15.116:43028) 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 -- 172.21.15.116:0/2552177122 learned_addr learned my addr 172.21.15.116:0/2552177122 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe758e00640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7541037f0 0x7fe754199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 -- 172.21.15.116:0/2552177122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7541037f0 msgr2=0x7fe754199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7541037f0 0x7fe754199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 -- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7541053a0 msgr2=0x7fe7541a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7541053a0 0x7fe7541a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 -- 172.21.15.116:0/2552177122 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7541a1710 con 0x7fe7541049e0 2024-03-04T07:42:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.541+0000 7fe753e00640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe75419a030 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe740009230 tx=0x7fe740005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.542+0000 7fe751600640 1 -- 172.21.15.116:0/2552177122 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740041070 con 0x7fe7541049e0 2024-03-04T07:42:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.542+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7541a19a0 con 0x7fe7541049e0 2024-03-04T07:42:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.542+0000 7fe751600640 1 -- 172.21.15.116:0/2552177122 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7400054e0 con 0x7fe7541049e0 2024-03-04T07:42:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.542+0000 7fe751600640 1 -- 172.21.15.116:0/2552177122 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740034440 con 0x7fe7541049e0 2024-03-04T07:42:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.542+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7541a1e90 con 0x7fe7541049e0 2024-03-04T07:42:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.545+0000 7fe751600640 1 -- 172.21.15.116:0/2552177122 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe740044070 con 0x7fe7541049e0 2024-03-04T07:42:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.545+0000 7fe75e633740 1 Processor -- start 2024-03-04T07:42:43.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.545+0000 7fe75e633740 1 -- start start 2024-03-04T07:42:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe75e633740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc41750c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe75e633740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe75e633740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe75e633740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cc40d9d60 con 0x559cc417a800 2024-03-04T07:42:43.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe75e633740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cc4082cf0 con 0x559cc4174ca0 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe75e633740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cc40d5e00 con 0x559cc4175690 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe753e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc41750c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc41750c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43030/0 (socket says 172.21.15.116:43030) 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 -- 172.21.15.116:0/1629538027 learned_addr learned my addr 172.21.15.116:0/1629538027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe758e00640 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 -- 172.21.15.116:0/1629538027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 msgr2=0x559cc417a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 -- 172.21.15.116:0/1629538027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 msgr2=0x559cc417cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe759800640 1 -- 172.21.15.116:0/1629538027 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cc3dee3f0 con 0x559cc4174ca0 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.546+0000 7fe758e00640 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.547+0000 7fe751600640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe73c0a1d80 0x7fe73c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.547+0000 7fe759800640 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc41750c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0039f0 tx=0x7fe74c003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e49a55250e45840 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.547+0000 7fe751600640 1 -- 172.21.15.116:0/2552177122 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe7400efa80 con 0x7fe7541049e0 2024-03-04T07:42:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.547+0000 7fe758e00640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe73c0a1d80 0x7fe73c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.547+0000 7fe746000640 1 -- 172.21.15.116:0/1629538027 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c01a470 con 0x559cc4174ca0 2024-03-04T07:42:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.547+0000 7fe746000640 1 -- 172.21.15.116:0/1629538027 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe74c015070 con 0x559cc4174ca0 2024-03-04T07:42:43.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.547+0000 7fe746000640 1 -- 172.21.15.116:0/1629538027 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c010900 con 0x559cc4174ca0 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 -- 172.21.15.116:0/1629538027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 msgr2=0x559cc41750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe758e00640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe73c0a1d80 0x7fe73c0a4240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x559cc4175a90 tx=0x7fe74800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc41750c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0039f0 tx=0x7fe74c003eb0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 -- 172.21.15.116:0/1629538027 shutdown_connections 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 --2- 172.21.15.116:0/1629538027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc41750c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 -- 172.21.15.116:0/1629538027 >> 172.21.15.116:0/1629538027 conn(0x559cc4170210 msgr2=0x559cc4170610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 -- 172.21.15.116:0/1629538027 shutdown_connections 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.548+0000 7fe75e633740 1 -- 172.21.15.116:0/1629538027 wait complete. 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 -- start start 2024-03-04T07:42:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 Processor -- start 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc417a800 0x559cc41867c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cc417d7b0 con 0x559cc4174ca0 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cc4078220 con 0x559cc417a800 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cc417d930 con 0x559cc4175690 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe75e633740 10 client.0 _ll_register_callbacks cb 0x559cc3dedbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe758e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe759800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc417a800 0x559cc41867c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.549+0000 7fe758e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48050/0 (socket says 172.21.15.116:48050) 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe758e00640 1 -- 172.21.15.116:0/2978048348 learned_addr learned my addr 172.21.15.116:0/2978048348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe753e00640 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe759800640 1 -- 172.21.15.116:0/2978048348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 msgr2=0x559cc417f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe759800640 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe759800640 1 -- 172.21.15.116:0/2978048348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc4174ca0 msgr2=0x559cc417f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe759800640 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe759800640 1 -- 172.21.15.116:0/2978048348 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559cc3eb87d0 con 0x559cc417a800 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe759800640 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc417a800 0x559cc41867c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe74c004410 tx=0x7fe74c011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf726c6b8491beda server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe73aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe73aa00640 1 -- 172.21.15.116:0/2978048348 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c011850 con 0x559cc417a800 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe73aa00640 1 -- 172.21.15.116:0/2978048348 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe74c0119f0 con 0x559cc417a800 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.550+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe74c015070 con 0x559cc417a800 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe73aa00640 1 -- 172.21.15.116:0/2978048348 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe74c010680 con 0x559cc417a800 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 10 client.25751 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559cc4189b30 con 0x559cc417a800 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe73aa00640 1 -- 172.21.15.116:0/2978048348 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe74c015070 con 0x559cc417a800 2024-03-04T07:42:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe73aa00640 1 -- 172.21.15.116:0/2978048348 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe74c023770 con 0x559cc417a800 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 10 client.25751 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 10 client.25751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 20 client.25751 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc418af30 0x559cc418d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 4 client.25751 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.551+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559cc40e72a0 con 0x559cc418af30 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.552+0000 7fe758e00640 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc418af30 0x559cc418d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.552+0000 7fe758e00640 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc418af30 0x559cc418d320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=631e8e8f49ca1040 server_cookie=669962b925dce5f6 in_seq=0 out_seq=0 2024-03-04T07:42:43.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.552+0000 7fe73aa00640 10 client.25751 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.553+0000 7fe73aa00640 1 -- 172.21.15.116:0/2978048348 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559cc40e72a0 con 0x559cc418af30 2024-03-04T07:42:43.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.553+0000 7fe73aa00640 10 client.25751 handle_command_reply: tid=0 2024-03-04T07:42:43.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 client.25751 shutdown 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc417a800 msgr2=0x559cc41867c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc417a800 0x559cc41867c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe74c004410 tx=0x7fe74c011d10 comp rx=0 tx=0).stop 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 shutdown_connections 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc418af30 0x559cc418d320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc417a800 0x559cc41867c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc417f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 --2- 172.21.15.116:0/2978048348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 >> 172.21.15.116:0/2978048348 conn(0x559cc4170210 msgr2=0x559cc416f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe73aa00640 0 client.25751 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 shutdown_connections 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 1 -- 172.21.15.116:0/2978048348 wait complete. 2024-03-04T07:42:43.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.554+0000 7fe75e633740 20 client.25751 trim_cache size 0 max 16384 2024-03-04T07:42:43.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 Processor -- start 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 -- start start 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc4186780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541a8cb0 con 0x559cc417a800 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541ab290 con 0x559cc4174ca0 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe75e633740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541a92b0 con 0x559cc4175690 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43048/0 (socket says 172.21.15.116:43048) 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe759800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 -- 172.21.15.116:0/2510078968 learned_addr learned my addr 172.21.15.116:0/2510078968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe753e00640 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc4186780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 -- 172.21.15.116:0/2510078968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 msgr2=0x559cc4186780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc4186780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 -- 172.21.15.116:0/2510078968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 msgr2=0x559cc417f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.568+0000 7fe758e00640 1 -- 172.21.15.116:0/2510078968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cc417f030 con 0x559cc4174ca0 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.569+0000 7fe758e00640 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc417f240 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe748011d30 tx=0x7fe748019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c6446b2e52b5db3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.569+0000 7fe745600640 1 -- 172.21.15.116:0/2510078968 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74801eda0 con 0x559cc4174ca0 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.569+0000 7fe745600640 1 -- 172.21.15.116:0/2510078968 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe74802f070 con 0x559cc4174ca0 2024-03-04T07:42:43.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.569+0000 7fe745600640 1 -- 172.21.15.116:0/2510078968 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74801c950 con 0x559cc4174ca0 2024-03-04T07:42:43.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 -- 172.21.15.116:0/2510078968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 msgr2=0x559cc417f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc417f240 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe748011d30 tx=0x7fe748019000 comp rx=0 tx=0).stop 2024-03-04T07:42:43.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 -- 172.21.15.116:0/2510078968 shutdown_connections 2024-03-04T07:42:43.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc417f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4175690 0x559cc4186780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 --2- 172.21.15.116:0/2510078968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4174ca0 0x559cc417f240 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 -- 172.21.15.116:0/2510078968 >> 172.21.15.116:0/2510078968 conn(0x559cc4170210 msgr2=0x559cc4176b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 -- 172.21.15.116:0/2510078968 shutdown_connections 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 -- 172.21.15.116:0/2510078968 wait complete. 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 -- start start 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.570+0000 7fe75e633740 1 Processor -- start 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe75e633740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4174ca0 0x559cc41845d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe75e633740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4175690 0x559cc4184b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe75e633740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc4185b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe75e633740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541a9910 con 0x559cc417a800 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe75e633740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541aac30 con 0x559cc4175690 2024-03-04T07:42:43.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe75e633740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541a9f70 con 0x559cc4174ca0 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe75e633740 10 client.0 _ll_register_callbacks cb 0x7fe7400f7040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4175690 0x559cc4184b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4175690 0x559cc4184b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43058/0 (socket says 172.21.15.116:43058) 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 -- 172.21.15.116:0/1802610353 learned_addr learned my addr 172.21.15.116:0/1802610353 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe759800640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc4185b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe758e00640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4174ca0 0x559cc41845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 -- 172.21.15.116:0/1802610353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4174ca0 msgr2=0x559cc41845d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4174ca0 0x559cc41845d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 -- 172.21.15.116:0/1802610353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 msgr2=0x559cc4185b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc4185b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe753e00640 1 -- 172.21.15.116:0/1802610353 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559cc4186110 con 0x559cc4175690 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe759800640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc4185b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.571+0000 7fe758e00640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4174ca0 0x559cc41845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:43.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe753e00640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4175690 0x559cc4184b10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe7400efd20 tx=0x7fe7400f30b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ff68a2ca5f9bceb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe73aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe73aa00640 1 -- 172.21.15.116:0/1802610353 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74004b040 con 0x559cc4175690 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe73aa00640 1 -- 172.21.15.116:0/1802610353 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe7400c4a80 con 0x559cc4175690 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe74802f070 con 0x559cc4175690 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe73aa00640 1 -- 172.21.15.116:0/1802610353 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7400bc360 con 0x559cc4175690 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe75e633740 10 client.25757 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559cc41e6ec0 con 0x559cc4175690 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe73aa00640 1 -- 172.21.15.116:0/1802610353 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe74802f070 con 0x559cc4175690 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.572+0000 7fe73aa00640 1 -- 172.21.15.116:0/1802610353 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe7400d5120 con 0x559cc4175690 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe75e633740 10 client.25757 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe75e633740 10 client.25757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe75e633740 20 client.25757 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe75e633740 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc41e7890 0x559cc41e9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe75e633740 4 client.25757 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559cc40e15f0 con 0x559cc41e7890 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe758e00640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc41e7890 0x559cc41e9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe758e00640 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc41e7890 0x559cc41e9c80 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=459d5ef82717a9b1 server_cookie=b7e96ece9d307d07 in_seq=0 out_seq=0 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe73aa00640 10 client.25757 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.573+0000 7fe73aa00640 1 -- 172.21.15.116:0/1802610353 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559cc40e15f0 con 0x559cc41e7890 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe73aa00640 10 client.25757 handle_command_reply: tid=0 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 client.25757 shutdown 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4175690 msgr2=0x559cc4184b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4175690 0x559cc4184b10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe7400efd20 tx=0x7fe7400f30b0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 shutdown_connections 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559cc41e7890 0x559cc41e9c80 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559cc417a800 0x559cc4185b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559cc4175690 0x559cc4184b10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 --2- 172.21.15.116:0/1802610353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559cc4174ca0 0x559cc41845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 >> 172.21.15.116:0/1802610353 conn(0x559cc4170210 msgr2=0x559cc417f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe73aa00640 0 client.25757 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 shutdown_connections 2024-03-04T07:42:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 1 -- 172.21.15.116:0/1802610353 wait complete. 2024-03-04T07:42:43.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.574+0000 7fe75e633740 20 client.25757 trim_cache size 0 max 16384 2024-03-04T07:42:43.580 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe73c0a1d80 msgr2=0x7fe73c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe73c0a1d80 0x7fe73c0a4240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x559cc4175a90 tx=0x7fe74800b390 comp rx=0 tx=0).stop 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 msgr2=0x7fe75419a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe75419a030 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe740009230 tx=0x7fe740005dd0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 shutdown_connections 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7541053a0 0x7fe7541a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe7541049e0 0x7fe75419a030 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe73c0a1d80 0x7fe73c0a4240 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 --2- 172.21.15.116:0/2552177122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7541037f0 0x7fe754199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 >> 172.21.15.116:0/2552177122 conn(0x7fe7540fed80 msgr2=0x7fe754107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 shutdown_connections 2024-03-04T07:42:43.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:43.575+0000 7fe75b800640 1 -- 172.21.15.116:0/2552177122 wait complete. 2024-03-04T07:42:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 -- 172.21.15.116:0/916092665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8108e20 msgr2=0x7f47c81092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 --2- 172.21.15.116:0/916092665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8108e20 0x7f47c81092a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009900 tx=0x7f47b402bc30 comp rx=0 tx=0).stop 2024-03-04T07:42:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 -- 172.21.15.116:0/916092665 shutdown_connections 2024-03-04T07:42:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 --2- 172.21.15.116:0/916092665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47c81097e0 0x7f47c8110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 --2- 172.21.15.116:0/916092665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8108e20 0x7f47c81092a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 --2- 172.21.15.116:0/916092665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47c8107c30 0x7f47c8108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 -- 172.21.15.116:0/916092665 >> 172.21.15.116:0/916092665 conn(0x7f47c8076be0 msgr2=0x7f47c8079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 -- 172.21.15.116:0/916092665 shutdown_connections 2024-03-04T07:42:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.802+0000 7f47d0200640 1 -- 172.21.15.116:0/916092665 wait complete. 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 Processor -- start 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 -- start start 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8107c30 0x7f47c819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47c8108e20 0x7f47c819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47c81097e0 0x7f47c81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81138e0 con 0x7f47c81097e0 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c8113760 con 0x7f47c8107c30 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47d0200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c8113a60 con 0x7f47c8108e20 2024-03-04T07:42:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.803+0000 7f47cd800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8107c30 0x7f47c819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8107c30 0x7f47c819dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43082/0 (socket says 172.21.15.116:43082) 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47ce200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47c81097e0 0x7f47c81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 -- 172.21.15.116:0/3692291331 learned_addr learned my addr 172.21.15.116:0/3692291331 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cce00640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47c8108e20 0x7f47c819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 -- 172.21.15.116:0/3692291331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47c8108e20 msgr2=0x7f47c819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47c8108e20 0x7f47c819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 -- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47c81097e0 msgr2=0x7f47c81a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47c81097e0 0x7f47c81a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 -- 172.21.15.116:0/3692291331 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c81a5ac0 con 0x7f47c8107c30 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47cd800640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8107c30 0x7f47c819dea0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00e9c0 tx=0x7f47bc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47c6000640 1 -- 172.21.15.116:0/3692291331 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc00cde0 con 0x7f47c8107c30 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.804+0000 7f47c6000640 1 -- 172.21.15.116:0/3692291331 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47bc010910 con 0x7f47c8107c30 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.805+0000 7f47c6000640 1 -- 172.21.15.116:0/3692291331 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc018a00 con 0x7f47c8107c30 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.805+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c81a5db0 con 0x7f47c8107c30 2024-03-04T07:42:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.805+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c81a6270 con 0x7f47c8107c30 2024-03-04T07:42:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.807+0000 7f47c6000640 1 -- 172.21.15.116:0/3692291331 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f47bc01f020 con 0x7f47c8107c30 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 Processor -- start 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 -- start start 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acce12f370 con 0x55acce1ca240 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acce0d8550 con 0x55acce1cf3b0 2024-03-04T07:42:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47d2f92740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acce12b410 con 0x55acce1c9850 2024-03-04T07:42:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.808+0000 7f47cce00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cd800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43096/0 (socket says 172.21.15.116:43096) 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 -- 172.21.15.116:0/1271126393 learned_addr learned my addr 172.21.15.116:0/1271126393 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47ce200640 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 -- 172.21.15.116:0/1271126393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 msgr2=0x55acce1c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 -- 172.21.15.116:0/1271126393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 msgr2=0x55acce1ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 -- 172.21.15.116:0/1271126393 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acce0ec3c0 con 0x55acce1cf3b0 2024-03-04T07:42:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47c6000640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47b00a1ea0 0x7f47b00a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47cce00640 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1d1840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f47b4000c00 tx=0x7f47b4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b7bdd1e13f546a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47c6000640 1 -- 172.21.15.116:0/3692291331 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f47bc0cdf40 con 0x7f47c8107c30 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47ce200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47b00a1ea0 0x7f47b00a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.809+0000 7f47baa00640 1 -- 172.21.15.116:0/1271126393 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4045070 con 0x55acce1cf3b0 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47baa00640 1 -- 172.21.15.116:0/1271126393 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47b4004040 con 0x55acce1cf3b0 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47baa00640 1 -- 172.21.15.116:0/1271126393 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4004eb0 con 0x55acce1cf3b0 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 -- 172.21.15.116:0/1271126393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 msgr2=0x55acce1d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47ce200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47b00a1ea0 0x7f47b00a4360 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55acce1caaa0 tx=0x7f47c0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1d1840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f47b4000c00 tx=0x7f47b4009560 comp rx=0 tx=0).stop 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 -- 172.21.15.116:0/1271126393 shutdown_connections 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1d1840 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 --2- 172.21.15.116:0/1271126393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 -- 172.21.15.116:0/1271126393 >> 172.21.15.116:0/1271126393 conn(0x55acce1c4dc0 msgr2=0x55acce1c51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.810+0000 7f47d2f92740 1 -- 172.21.15.116:0/1271126393 shutdown_connections 2024-03-04T07:42:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.811+0000 7f47d2f92740 1 -- 172.21.15.116:0/1271126393 wait complete. 2024-03-04T07:42:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.811+0000 7f47d2f92740 1 -- start start 2024-03-04T07:42:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.811+0000 7f47d2f92740 1 Processor -- start 2024-03-04T07:42:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c9850 0x55acce1d3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1d4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1db030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47d2f92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acce1d2300 con 0x55acce1c9850 2024-03-04T07:42:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47d2f92740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acce0eb9b0 con 0x55acce1cf3b0 2024-03-04T07:42:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47d2f92740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acce1d2480 con 0x55acce1ca240 2024-03-04T07:42:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47d2f92740 10 client.0 _ll_register_callbacks cb 0x55acce127340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47cd800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c9850 0x55acce1d3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1db030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1db030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43104/0 (socket says 172.21.15.116:43104) 2024-03-04T07:42:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 -- 172.21.15.116:0/332647202 learned_addr learned my addr 172.21.15.116:0/332647202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47cce00640 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1d4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 -- 172.21.15.116:0/332647202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 msgr2=0x55acce1d4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1d4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 -- 172.21.15.116:0/332647202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c9850 msgr2=0x55acce1d3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c9850 0x55acce1d3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 -- 172.21.15.116:0/332647202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55accde44210 con 0x55acce1cf3b0 2024-03-04T07:42:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47ce200640 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1db030 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009cd0 tx=0x7f47c0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e2bde122b53100b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.812+0000 7f47af400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47af400640 1 -- 172.21.15.116:0/332647202 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000f7b0 con 0x55acce1cf3b0 2024-03-04T07:42:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47af400640 1 -- 172.21.15.116:0/332647202 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f47c000f950 con 0x55acce1cf3b0 2024-03-04T07:42:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47b4004040 con 0x55acce1cf3b0 2024-03-04T07:42:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47af400640 1 -- 172.21.15.116:0/332647202 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47c000fb50 con 0x55acce1cf3b0 2024-03-04T07:42:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47d2f92740 10 client.25769 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55accdf10620 con 0x55acce1cf3b0 2024-03-04T07:42:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47af400640 1 -- 172.21.15.116:0/332647202 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47b4004040 con 0x55acce1cf3b0 2024-03-04T07:42:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47af400640 1 -- 172.21.15.116:0/332647202 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f47c0026070 con 0x55acce1cf3b0 2024-03-04T07:42:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47d2f92740 10 client.25769 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.813+0000 7f47d2f92740 10 client.25769 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.814+0000 7f47d2f92740 20 client.25769 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.814+0000 7f47d2f92740 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1df980 0x55acce1e1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.814+0000 7f47d2f92740 4 client.25769 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.814+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acce13c8b0 con 0x55acce1df980 2024-03-04T07:42:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.814+0000 7f47cd800640 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1df980 0x55acce1e1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.814+0000 7f47cd800640 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1df980 0x55acce1e1df0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7903e8e1fb4ebbb4 server_cookie=eec4d99c40458225 in_seq=0 out_seq=0 2024-03-04T07:42:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.814+0000 7f47af400640 10 client.25769 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47af400640 1 -- 172.21.15.116:0/332647202 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55acce13c8b0 con 0x55acce1df980 2024-03-04T07:42:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47af400640 10 client.25769 handle_command_reply: tid=0 2024-03-04T07:42:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 client.25769 shutdown 2024-03-04T07:42:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 msgr2=0x55acce1db030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1db030 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009cd0 tx=0x7f47c0004ab0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 shutdown_connections 2024-03-04T07:42:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1df980 0x55acce1e1df0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1cf3b0 0x55acce1db030 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1d4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47af400640 0 client.25769 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 --2- 172.21.15.116:0/332647202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c9850 0x55acce1d3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 >> 172.21.15.116:0/332647202 conn(0x55acce1c4dc0 msgr2=0x55acce1c3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 shutdown_connections 2024-03-04T07:42:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.815+0000 7f47d2f92740 1 -- 172.21.15.116:0/332647202 wait complete. 2024-03-04T07:42:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.816+0000 7f47d2f92740 20 client.25769 trim_cache size 0 max 16384 2024-03-04T07:42:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.830+0000 7f47d2f92740 1 Processor -- start 2024-03-04T07:42:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47d2f92740 1 -- start start 2024-03-04T07:42:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c4dc0 0x55acce1c8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c9850 0x55acce1daff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1c8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47d2f92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81acf60 con 0x55acce1c4dc0 2024-03-04T07:42:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47d2f92740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81af540 con 0x55acce1c9850 2024-03-04T07:42:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47d2f92740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81ad560 con 0x55acce1ca240 2024-03-04T07:42:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47cce00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c9850 0x55acce1daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47cce00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c9850 0x55acce1daff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43110/0 (socket says 172.21.15.116:43110) 2024-03-04T07:42:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47cce00640 1 -- 172.21.15.116:0/823960474 learned_addr learned my addr 172.21.15.116:0/823960474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47cd800640 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c4dc0 0x55acce1c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.831+0000 7f47ce200640 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47cce00640 1 -- 172.21.15.116:0/823960474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 msgr2=0x55acce1c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47cce00640 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1c8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47cce00640 1 -- 172.21.15.116:0/823960474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c4dc0 msgr2=0x55acce1c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47cce00640 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c4dc0 0x55acce1c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47cce00640 1 -- 172.21.15.116:0/823960474 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acce1caad0 con 0x55acce1c9850 2024-03-04T07:42:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47cd800640 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c4dc0 0x55acce1c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47cce00640 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c9850 0x55acce1daff0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f47b402c170 tx=0x7f47b402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1526364eab133de server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47ba000640 1 -- 172.21.15.116:0/823960474 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b404e030 con 0x55acce1c9850 2024-03-04T07:42:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47ba000640 1 -- 172.21.15.116:0/823960474 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47b402cc90 con 0x55acce1c9850 2024-03-04T07:42:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.832+0000 7f47ba000640 1 -- 172.21.15.116:0/823960474 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4045040 con 0x55acce1c9850 2024-03-04T07:42:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 -- 172.21.15.116:0/823960474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c9850 msgr2=0x55acce1daff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c9850 0x55acce1daff0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f47b402c170 tx=0x7f47b402bc30 comp rx=0 tx=0).stop 2024-03-04T07:42:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 -- 172.21.15.116:0/823960474 shutdown_connections 2024-03-04T07:42:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1ca240 0x55acce1c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c9850 0x55acce1daff0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 --2- 172.21.15.116:0/823960474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1c4dc0 0x55acce1c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 -- 172.21.15.116:0/823960474 >> 172.21.15.116:0/823960474 conn(0x55acce1ceeb0 msgr2=0x55acce1cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 -- 172.21.15.116:0/823960474 shutdown_connections 2024-03-04T07:42:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 -- 172.21.15.116:0/823960474 wait complete. 2024-03-04T07:42:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.833+0000 7f47d2f92740 1 -- start start 2024-03-04T07:42:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 1 Processor -- start 2024-03-04T07:42:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c4dc0 0x55acce1d8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1d9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1da230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81adbc0 con 0x55acce1ca240 2024-03-04T07:42:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81aeee0 con 0x55acce1c4dc0 2024-03-04T07:42:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c81ae220 con 0x55acce1c9850 2024-03-04T07:42:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47d2f92740 10 client.0 _ll_register_callbacks cb 0x7f47c0009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47ce200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1da230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47cd800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c4dc0 0x55acce1d8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47ce200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1da230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48120/0 (socket says 172.21.15.116:48120) 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47ce200640 1 -- 172.21.15.116:0/2359359681 learned_addr learned my addr 172.21.15.116:0/2359359681 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.834+0000 7f47cce00640 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1d9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47ce200640 1 -- 172.21.15.116:0/2359359681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 msgr2=0x55acce1d9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47ce200640 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1d9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47ce200640 1 -- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c4dc0 msgr2=0x55acce1d8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47ce200640 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c4dc0 0x55acce1d8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47ce200640 1 -- 172.21.15.116:0/2359359681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55acce1da820 con 0x55acce1ca240 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47cd800640 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c4dc0 0x55acce1d8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47ce200640 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1da230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009f50 tx=0x7f47c000fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23aec61fbfe80b81 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47af400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47af400640 1 -- 172.21.15.116:0/2359359681 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0011240 con 0x55acce1ca240 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47af400640 1 -- 172.21.15.116:0/2359359681 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f47c001e6f0 con 0x55acce1ca240 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47b402cc90 con 0x55acce1ca240 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47af400640 1 -- 172.21.15.116:0/2359359681 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47c0042050 con 0x55acce1ca240 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47d2f92740 10 client.15660 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.835+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55acce23b880 con 0x55acce1ca240 2024-03-04T07:42:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47af400640 1 -- 172.21.15.116:0/2359359681 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47b402cc90 con 0x55acce1ca240 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47af400640 1 -- 172.21.15.116:0/2359359681 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f47c0026070 con 0x55acce1ca240 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47d2f92740 10 client.15660 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47d2f92740 10 client.15660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47d2f92740 20 client.15660 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47d2f92740 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1cf3b0 0x55acce23e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47d2f92740 4 client.15660 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55acce136c00 con 0x55acce1cf3b0 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.836+0000 7f47cd800640 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1cf3b0 0x55acce23e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.837+0000 7f47af400640 10 client.15660 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.837+0000 7f47cd800640 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1cf3b0 0x55acce23e1e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d3add7bb25ae9e4 server_cookie=3df1befa9c9c6865 in_seq=0 out_seq=0 2024-03-04T07:42:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47af400640 1 -- 172.21.15.116:0/2359359681 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55acce136c00 con 0x55acce1cf3b0 2024-03-04T07:42:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47af400640 10 client.15660 handle_command_reply: tid=0 2024-03-04T07:42:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 client.15660 shutdown 2024-03-04T07:42:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 msgr2=0x55acce1da230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1da230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009f50 tx=0x7f47c000fd70 comp rx=0 tx=0).stop 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 shutdown_connections 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55acce1cf3b0 0x55acce23e1e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55acce1ca240 0x55acce1da230 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acce1c9850 0x55acce1d9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 --2- 172.21.15.116:0/2359359681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55acce1c4dc0 0x55acce1d8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47af400640 0 client.15660 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 >> 172.21.15.116:0/2359359681 conn(0x55acce1ceeb0 msgr2=0x55acce1cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 shutdown_connections 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.849+0000 7f47d2f92740 1 -- 172.21.15.116:0/2359359681 wait complete. 2024-03-04T07:42:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.850+0000 7f47d2f92740 20 client.15660 trim_cache size 0 max 16384 2024-03-04T07:42:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:42:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:42:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:42:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.850+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47b00a1ea0 msgr2=0x7f47b00a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.850+0000 7f47d0200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47b00a1ea0 0x7f47b00a4360 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55acce1caaa0 tx=0x7f47c0009500 comp rx=0 tx=0).stop 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.850+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8107c30 msgr2=0x7f47c819dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.850+0000 7f47d0200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8107c30 0x7f47c819dea0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00e9c0 tx=0x7f47bc00ee80 comp rx=0 tx=0).stop 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 shutdown_connections 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47c81097e0 0x7f47c81a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47b00a1ea0 0x7f47b00a4360 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47c8108e20 0x7f47c819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 --2- 172.21.15.116:0/3692291331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47c8107c30 0x7f47c819dea0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 >> 172.21.15.116:0/3692291331 conn(0x7f47c8076be0 msgr2=0x7f47c810c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 shutdown_connections 2024-03-04T07:42:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:43.851+0000 7f47d0200640 1 -- 172.21.15.116:0/3692291331 wait complete. 2024-03-04T07:42:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:43 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:42:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:43 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:43 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:42:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:43 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:43 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:42:44.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:43 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/70507167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:43 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:42:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:43 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:43 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:42:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:43 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:43 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:42:44.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:43 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/70507167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:43 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:42:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:43 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:43 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:42:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:43 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:43 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:42:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:43 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/70507167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:42:44.594 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:42:44.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:42:44.594 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.820+0000 7f05ad400640 1 -- 172.21.15.116:0/2420412975 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8104a00 msgr2=0x7f05a8104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.820+0000 7f05ad400640 1 --2- 172.21.15.116:0/2420412975 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8104a00 0x7f05a8104e80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0590009900 tx=0x7f059002bc40 comp rx=0 tx=0).stop 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.821+0000 7f05ad400640 1 -- 172.21.15.116:0/2420412975 shutdown_connections 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.821+0000 7f05ad400640 1 --2- 172.21.15.116:0/2420412975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a81053c0 0x7f05a810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.821+0000 7f05ad400640 1 --2- 172.21.15.116:0/2420412975 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8104a00 0x7f05a8104e80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.821+0000 7f05ad400640 1 --2- 172.21.15.116:0/2420412975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05a8103810 0x7f05a8103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.821+0000 7f05ad400640 1 -- 172.21.15.116:0/2420412975 >> 172.21.15.116:0/2420412975 conn(0x7f05a80fed80 msgr2=0x7f05a81011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.821+0000 7f05ad400640 1 -- 172.21.15.116:0/2420412975 shutdown_connections 2024-03-04T07:42:44.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.821+0000 7f05ad400640 1 -- 172.21.15.116:0/2420412975 wait complete. 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 Processor -- start 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 -- start start 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8103810 0x7f05a8199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8104a00 0x7f05a819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05a81053c0 0x7f05a81a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a810f4e0 con 0x7f05a8104a00 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a810f360 con 0x7f05a8103810 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05ad400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a810f660 con 0x7f05a81053c0 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.822+0000 7f05a6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8103810 0x7f05a8199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8103810 0x7f05a8199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43134/0 (socket says 172.21.15.116:43134) 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 -- 172.21.15.116:0/662935214 learned_addr learned my addr 172.21.15.116:0/662935214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:44.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a5e00640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8104a00 0x7f05a819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a7200640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05a81053c0 0x7f05a81a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 -- 172.21.15.116:0/662935214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05a81053c0 msgr2=0x7f05a81a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05a81053c0 0x7f05a81a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 -- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8104a00 msgr2=0x7f05a819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8104a00 0x7f05a819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 -- 172.21.15.116:0/662935214 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a81a17b0 con 0x7f05a8103810 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05a6800640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8103810 0x7f05a8199b90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f059800eaa0 tx=0x7f059800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f059f400640 1 -- 172.21.15.116:0/662935214 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598016070 con 0x7f05a8103810 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f059f400640 1 -- 172.21.15.116:0/662935214 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0598003e50 con 0x7f05a8103810 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f059f400640 1 -- 172.21.15.116:0/662935214 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05980116d0 con 0x7f05a8103810 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.823+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a81a1aa0 con 0x7f05a8103810 2024-03-04T07:42:44.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.824+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a81a1f70 con 0x7f05a8103810 2024-03-04T07:42:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.826+0000 7f059f400640 1 -- 172.21.15.116:0/662935214 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f0598019070 con 0x7f05a8103810 2024-03-04T07:42:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 Processor -- start 2024-03-04T07:42:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 -- start start 2024-03-04T07:42:44.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x5638801774f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x56388017c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x56388017f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56388008ae20 con 0x56388017cc30 2024-03-04T07:42:44.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638800dca20 con 0x563880177ac0 2024-03-04T07:42:44.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05b0188740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563880085b00 con 0x5638801770d0 2024-03-04T07:42:44.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05a6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x56388017c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.827+0000 7f05a5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x56388017f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a7200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x5638801774f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x56388017f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48154/0 (socket says 172.21.15.116:48154) 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x56388017c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43148/0 (socket says 172.21.15.116:43148) 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a7200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x5638801774f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:59412/0 (socket says 172.21.15.116:59412) 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 -- 172.21.15.116:0/636873162 learned_addr learned my addr 172.21.15.116:0/636873162 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f059f400640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f058c0a1e00 0x7f058c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 -- 172.21.15.116:0/636873162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 msgr2=0x5638801774f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x5638801774f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 -- 172.21.15.116:0/636873162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 msgr2=0x56388017c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x56388017c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 -- 172.21.15.116:0/636873162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56388009ae10 con 0x56388017cc30 2024-03-04T07:42:44.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a6800640 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x56388017c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:44.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f059f400640 1 -- 172.21.15.116:0/662935214 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f05980cdee0 con 0x7f05a8103810 2024-03-04T07:42:44.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a7200640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f058c0a1e00 0x7f058c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.828+0000 7f05a5e00640 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x56388017f0c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0590005ab0 tx=0x7f0590009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10e58dd65a27fde server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:44.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.829+0000 7f0597400640 1 -- 172.21.15.116:0/636873162 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590042070 con 0x56388017cc30 2024-03-04T07:42:44.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.829+0000 7f0597400640 1 -- 172.21.15.116:0/636873162 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f059002cb90 con 0x56388017cc30 2024-03-04T07:42:44.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.829+0000 7f0597400640 1 -- 172.21.15.116:0/636873162 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900048c0 con 0x56388017cc30 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.829+0000 7f05b0188740 1 -- 172.21.15.116:0/636873162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 msgr2=0x56388017f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.829+0000 7f05a7200640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f058c0a1e00 0x7f058c0a42c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x563880178320 tx=0x7f05a00046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.829+0000 7f05b0188740 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x56388017f0c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0590005ab0 tx=0x7f0590009560 comp rx=0 tx=0).stop 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 -- 172.21.15.116:0/636873162 shutdown_connections 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x56388017f0c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x56388017c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 --2- 172.21.15.116:0/636873162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x5638801774f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 -- 172.21.15.116:0/636873162 >> 172.21.15.116:0/636873162 conn(0x563880172640 msgr2=0x563880172a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 -- 172.21.15.116:0/636873162 shutdown_connections 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 -- 172.21.15.116:0/636873162 wait complete. 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 -- start start 2024-03-04T07:42:44.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.830+0000 7f05b0188740 1 Processor -- start 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05b0188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638801770d0 0x5638801813a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05b0188740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x5638801818e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05b0188740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56388017cc30 0x563880188920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05b0188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563880098f40 con 0x5638801770d0 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05b0188740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638800d8ac0 con 0x563880177ac0 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05b0188740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56388017fb80 con 0x56388017cc30 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05b0188740 10 client.0 _ll_register_callbacks cb 0x5638800d7aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x5638801818e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638801770d0 0x5638801813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x5638801818e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43160/0 (socket says 172.21.15.116:43160) 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 -- 172.21.15.116:0/31727513 learned_addr learned my addr 172.21.15.116:0/31727513 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a7200640 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56388017cc30 0x563880188920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 -- 172.21.15.116:0/31727513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56388017cc30 msgr2=0x563880188920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56388017cc30 0x563880188920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 -- 172.21.15.116:0/31727513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638801770d0 msgr2=0x5638801813a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638801770d0 0x5638801813a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a5e00640 1 -- 172.21.15.116:0/31727513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56387feb97e0 con 0x563880177ac0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.831+0000 7f05a6800640 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638801770d0 0x5638801813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f05a5e00640 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x5638801818e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f059002c180 tx=0x7f059002c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77c82b780e51d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f058be00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f058be00640 1 -- 172.21.15.116:0/31727513 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590042050 con 0x563880177ac0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f059002cb90 con 0x563880177ac0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f058be00640 1 -- 172.21.15.116:0/31727513 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0590003800 con 0x563880177ac0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f058be00640 1 -- 172.21.15.116:0/31727513 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0590003a00 con 0x563880177ac0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f05b0188740 10 client.25787 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56387ff19f70 con 0x563880177ac0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.832+0000 7f058be00640 1 -- 172.21.15.116:0/31727513 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f059002cb90 con 0x563880177ac0 2024-03-04T07:42:44.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f058be00640 1 -- 172.21.15.116:0/31727513 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0590034440 con 0x563880177ac0 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05b0188740 10 client.25787 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05b0188740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05b0188740 20 client.25787 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05b0188740 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56388018ce10 0x56388018f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05b0188740 4 client.25787 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638800e9f60 con 0x56388018ce10 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05a6800640 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56388018ce10 0x56388018f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f05a6800640 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56388018ce10 0x56388018f200 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dc04d7ba13e5105 server_cookie=232e661d69ab02c4 in_seq=0 out_seq=0 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.833+0000 7f058be00640 10 client.25787 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f058be00640 1 -- 172.21.15.116:0/31727513 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5638800e9f60 con 0x56388018ce10 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f058be00640 10 client.25787 handle_command_reply: tid=0 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 client.25787 shutdown 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 msgr2=0x5638801818e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x5638801818e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f059002c180 tx=0x7f059002c340 comp rx=0 tx=0).stop 2024-03-04T07:42:44.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 shutdown_connections 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56388018ce10 0x56388018f200 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56388017cc30 0x563880188920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563880177ac0 0x5638801818e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 --2- 172.21.15.116:0/31727513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638801770d0 0x5638801813a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f058be00640 0 client.25787 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.834+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 >> 172.21.15.116:0/31727513 conn(0x563880172640 msgr2=0x563880173890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.835+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 shutdown_connections 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.835+0000 7f05b0188740 1 -- 172.21.15.116:0/31727513 wait complete. 2024-03-04T07:42:44.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.835+0000 7f05b0188740 20 client.25787 trim_cache size 0 max 16384 2024-03-04T07:42:44.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 Processor -- start 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 -- start start 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x563880172a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563880177ac0 0x5638801888e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56388017cc30 0x5638801711d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81a8c80 con 0x563880177ac0 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81ab260 con 0x56388017cc30 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.852+0000 7f05b0188740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81a9280 con 0x5638801770d0 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563880177ac0 0x5638801888e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a7200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56388017cc30 0x5638801711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563880177ac0 0x5638801888e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48178/0 (socket says 172.21.15.116:48178) 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 -- 172.21.15.116:0/272212631 learned_addr learned my addr 172.21.15.116:0/272212631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:44.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a6800640 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x563880172a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 -- 172.21.15.116:0/272212631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 msgr2=0x563880172a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x563880172a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 -- 172.21.15.116:0/272212631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56388017cc30 msgr2=0x5638801711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56388017cc30 0x5638801711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 -- 172.21.15.116:0/272212631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56387fdb1170 con 0x563880177ac0 2024-03-04T07:42:44.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a6800640 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x563880172a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.853+0000 7f05a5e00640 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563880177ac0 0x5638801888e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0590004650 tx=0x7f05900058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1736f02fd2ca5757 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f0596a00640 1 -- 172.21.15.116:0/272212631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590042020 con 0x563880177ac0 2024-03-04T07:42:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f0596a00640 1 -- 172.21.15.116:0/272212631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0590034440 con 0x563880177ac0 2024-03-04T07:42:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f0596a00640 1 -- 172.21.15.116:0/272212631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900053c0 con 0x563880177ac0 2024-03-04T07:42:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 -- 172.21.15.116:0/272212631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563880177ac0 msgr2=0x5638801888e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563880177ac0 0x5638801888e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0590004650 tx=0x7f05900058b0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 -- 172.21.15.116:0/272212631 shutdown_connections 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56388017cc30 0x5638801711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563880177ac0 0x5638801888e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 --2- 172.21.15.116:0/272212631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638801770d0 0x563880172a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 -- 172.21.15.116:0/272212631 >> 172.21.15.116:0/272212631 conn(0x563880172640 msgr2=0x56388016d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 -- 172.21.15.116:0/272212631 shutdown_connections 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.854+0000 7f05b0188740 1 -- 172.21.15.116:0/272212631 wait complete. 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 -- start start 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 Processor -- start 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638801770d0 0x563880186440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563880177ac0 0x563880186980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x5638801878d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81a98e0 con 0x56388017cc30 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81aac00 con 0x5638801770d0 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a81a9f40 con 0x563880177ac0 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05b0188740 10 client.0 _ll_register_callbacks cb 0x7f0598096e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05a6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638801770d0 0x563880186440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05a6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638801770d0 0x563880186440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43168/0 (socket says 172.21.15.116:43168) 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05a6800640 1 -- 172.21.15.116:0/517762706 learned_addr learned my addr 172.21.15.116:0/517762706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.855+0000 7f05a7200640 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x5638801878d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05a5e00640 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563880177ac0 0x563880186980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05a6800640 1 -- 172.21.15.116:0/517762706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563880177ac0 msgr2=0x563880186980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05a6800640 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563880177ac0 0x563880186980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05a6800640 1 -- 172.21.15.116:0/517762706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 msgr2=0x5638801878d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05a6800640 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x5638801878d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05a6800640 1 -- 172.21.15.116:0/517762706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563880187ec0 con 0x5638801770d0 2024-03-04T07:42:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05a6800640 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638801770d0 0x563880186440 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0598021000 tx=0x7f05980d16f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17bde7b08a74a486 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f058be00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f058be00640 1 -- 172.21.15.116:0/517762706 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05980b2070 con 0x5638801770d0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f058be00640 1 -- 172.21.15.116:0/517762706 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f05980041c0 con 0x5638801770d0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.856+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0590034440 con 0x5638801770d0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f058be00640 1 -- 172.21.15.116:0/517762706 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0598020040 con 0x5638801770d0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 10 client.25793 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5638801e8880 con 0x5638801770d0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f058be00640 1 -- 172.21.15.116:0/517762706 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0590034440 con 0x5638801770d0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f058be00640 1 -- 172.21.15.116:0/517762706 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f059800f040 con 0x5638801770d0 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 10 client.25793 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 20 client.25793 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638801e90d0 0x5638801eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 4 client.25793 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:42:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5638800e42b0 con 0x5638801e90d0 2024-03-04T07:42:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.857+0000 7f05a5e00640 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638801e90d0 0x5638801eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.858+0000 7f05a5e00640 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638801e90d0 0x5638801eb4c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad18f813d02cb087 server_cookie=98f7ffd79f9de4e4 in_seq=0 out_seq=0 2024-03-04T07:42:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.858+0000 7f058be00640 10 client.25793 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f058be00640 1 -- 172.21.15.116:0/517762706 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5638800e42b0 con 0x5638801e90d0 2024-03-04T07:42:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f058be00640 10 client.25793 handle_command_reply: tid=0 2024-03-04T07:42:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 client.25793 shutdown 2024-03-04T07:42:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638801770d0 msgr2=0x563880186440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638801770d0 0x563880186440 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0598021000 tx=0x7f05980d16f0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 shutdown_connections 2024-03-04T07:42:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638801e90d0 0x5638801eb4c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56388017cc30 0x5638801878d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563880177ac0 0x563880186980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f058be00640 0 client.25793 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 --2- 172.21.15.116:0/517762706 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638801770d0 0x563880186440 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 >> 172.21.15.116:0/517762706 conn(0x563880172640 msgr2=0x563880172a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 shutdown_connections 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.859+0000 7f05b0188740 1 -- 172.21.15.116:0/517762706 wait complete. 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05b0188740 20 client.25793 trim_cache size 0 max 16384 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:44.864 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:42:44.865 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "6e45eafc-bc0b-4c23-ab47-dd2216d6c78e", 2024-03-04T07:42:44.865 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:42:44.865 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f058c0a1e00 msgr2=0x7f058c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f058c0a1e00 0x7f058c0a42c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x563880178320 tx=0x7f05a00046c0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8103810 msgr2=0x7f05a8199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8103810 0x7f05a8199b90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f059800eaa0 tx=0x7f059800ef60 comp rx=0 tx=0).stop 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 shutdown_connections 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f058c0a1e00 0x7f058c0a42c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05a81053c0 0x7f05a81a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05a8104a00 0x7f05a819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 --2- 172.21.15.116:0/662935214 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05a8103810 0x7f05a8199b90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 >> 172.21.15.116:0/662935214 conn(0x7f05a80fed80 msgr2=0x7f05a8107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.860+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 shutdown_connections 2024-03-04T07:42:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:44.861+0000 7f05ad400640 1 -- 172.21.15.116:0/662935214 wait complete. 2024-03-04T07:42:44.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:42:44.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:42:44.951 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:44 smithi116 ceph-mon[32150]: pgmap v460: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 -- 172.21.15.116:0/2276328589 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31041097a0 msgr2=0x7f3104110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 --2- 172.21.15.116:0/2276328589 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31041097a0 0x7f3104110030 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009ca0 tx=0x7f30fc02c100 comp rx=0 tx=0).stop 2024-03-04T07:42:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 -- 172.21.15.116:0/2276328589 shutdown_connections 2024-03-04T07:42:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 --2- 172.21.15.116:0/2276328589 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f31041097a0 0x7f3104110030 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 --2- 172.21.15.116:0/2276328589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 0x7f3104109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 --2- 172.21.15.116:0/2276328589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104107bf0 0x7f3104108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 -- 172.21.15.116:0/2276328589 >> 172.21.15.116:0/2276328589 conn(0x7f3104076b90 msgr2=0x7f3104078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 -- 172.21.15.116:0/2276328589 shutdown_connections 2024-03-04T07:42:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.084+0000 7f3109200640 1 -- 172.21.15.116:0/2276328589 wait complete. 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.085+0000 7f3109200640 1 Processor -- start 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.085+0000 7f3109200640 1 -- start start 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.085+0000 7f3109200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3104107bf0 0x7f310419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.085+0000 7f3109200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 0x7f310419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.085+0000 7f3109200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31041097a0 0x7f31041a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3109200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3104113870 con 0x7f31041097a0 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3109200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041136f0 con 0x7f3104107bf0 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3109200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041139f0 con 0x7f3104108de0 2024-03-04T07:42:45.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 0x7f310419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3101e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3104107bf0 0x7f310419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3102800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31041097a0 0x7f31041a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3101e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3104107bf0 0x7f310419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43198/0 (socket says 172.21.15.116:43198) 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 0x7f310419e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:59452/0 (socket says 172.21.15.116:59452) 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3101e00640 1 -- 172.21.15.116:0/2866108694 learned_addr learned my addr 172.21.15.116:0/2866108694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 -- 172.21.15.116:0/2866108694 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3104107bf0 msgr2=0x7f310419de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3104107bf0 0x7f310419de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 -- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31041097a0 msgr2=0x7f31041a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31041097a0 0x7f31041a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 -- 172.21.15.116:0/2866108694 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31041a5aa0 con 0x7f3104108de0 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3101e00640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3104107bf0 0x7f310419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f3102800640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31041097a0 0x7f31041a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.086+0000 7f30f9400640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 0x7f310419e3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00e9f0 tx=0x7f30ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:45.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.087+0000 7f30fb400640 1 -- 172.21.15.116:0/2866108694 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec00ccc0 con 0x7f3104108de0 2024-03-04T07:42:45.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.087+0000 7f30fb400640 1 -- 172.21.15.116:0/2866108694 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30ec005020 con 0x7f3104108de0 2024-03-04T07:42:45.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.087+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31041a5d90 con 0x7f3104108de0 2024-03-04T07:42:45.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.087+0000 7f30fb400640 1 -- 172.21.15.116:0/2866108694 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec0049e0 con 0x7f3104108de0 2024-03-04T07:42:45.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.087+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31041a6230 con 0x7f3104108de0 2024-03-04T07:42:45.090 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:44 smithi133 ceph-mon[36505]: pgmap v460: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:45.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f30fb400640 1 -- 172.21.15.116:0/2866108694 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f30ec005560 con 0x7f3104108de0 2024-03-04T07:42:45.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f310bf84740 1 Processor -- start 2024-03-04T07:42:45.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f310bf84740 1 -- start start 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f310bf84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x559841880f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f310bf84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841881540 0x559841886040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f310bf84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598418866b0 0x559841888b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f310bf84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598417ae090 con 0x559841881540 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.090+0000 7f310bf84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598417e5d30 con 0x559841880b50 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f310bf84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55984178ea30 con 0x5598418866b0 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x559841880f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x559841880f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43208/0 (socket says 172.21.15.116:43208) 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 -- 172.21.15.116:0/852752702 learned_addr learned my addr 172.21.15.116:0/852752702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f30f9400640 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598418866b0 0x559841888b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3101e00640 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841881540 0x559841886040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 -- 172.21.15.116:0/852752702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598418866b0 msgr2=0x559841888b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598418866b0 0x559841888b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 -- 172.21.15.116:0/852752702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841881540 msgr2=0x559841886040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841881540 0x559841886040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 -- 172.21.15.116:0/852752702 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598415c47d0 con 0x559841880b50 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.091+0000 7f3102800640 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x559841880f70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f30fc02c710 tx=0x7f30fc0323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dafcac97195edb69 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.092+0000 7f30fb400640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30e80a1d80 0x7f30e80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.092+0000 7f30f3400640 1 -- 172.21.15.116:0/852752702 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc043070 con 0x559841880b50 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.092+0000 7f30f3400640 1 -- 172.21.15.116:0/852752702 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30fc032c90 con 0x559841880b50 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.092+0000 7f30f3400640 1 -- 172.21.15.116:0/852752702 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc039040 con 0x559841880b50 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.092+0000 7f30fb400640 1 -- 172.21.15.116:0/2866108694 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f30ec0cdd60 con 0x7f3104108de0 2024-03-04T07:42:45.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.092+0000 7f3101e00640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30e80a1d80 0x7f30e80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 -- 172.21.15.116:0/852752702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 msgr2=0x559841880f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x559841880f70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f30fc02c710 tx=0x7f30fc0323d0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f3101e00640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30e80a1d80 0x7f30e80a4240 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x559841881940 tx=0x7f30f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 -- 172.21.15.116:0/852752702 shutdown_connections 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5598418866b0 0x559841888b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841881540 0x559841886040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 --2- 172.21.15.116:0/852752702 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x559841880f70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 -- 172.21.15.116:0/852752702 >> 172.21.15.116:0/852752702 conn(0x55984187c0c0 msgr2=0x55984187c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 -- 172.21.15.116:0/852752702 shutdown_connections 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 -- 172.21.15.116:0/852752702 wait complete. 2024-03-04T07:42:45.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 -- start start 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.093+0000 7f310bf84740 1 Processor -- start 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f310bf84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f310bf84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x55984188b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f310bf84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5598418866b0 0x559841892470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f310bf84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598417840c0 con 0x559841880b50 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f310bf84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598417e1dd0 con 0x5598418866b0 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f310bf84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559841889600 con 0x559841881540 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f310bf84740 10 client.0 _ll_register_callbacks cb 0x5598417a1860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f3101e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f30f9400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x55984188b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f3102800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5598418866b0 0x559841892470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f3101e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48224/0 (socket says 172.21.15.116:48224) 2024-03-04T07:42:45.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.094+0000 7f3101e00640 1 -- 172.21.15.116:0/841362267 learned_addr learned my addr 172.21.15.116:0/841362267 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f3102800640 1 -- 172.21.15.116:0/841362267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 msgr2=0x55984188b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f3102800640 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x55984188b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f3102800640 1 -- 172.21.15.116:0/841362267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841880b50 msgr2=0x55984188aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f3102800640 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f3102800640 1 -- 172.21.15.116:0/841362267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598414fa3f0 con 0x5598418866b0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f3102800640 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5598418866b0 0x559841892470 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009ca0 tx=0x7f30fc0093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a021f261a901e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f30e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f30e7e00640 1 -- 172.21.15.116:0/841362267 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc04c030 con 0x5598418866b0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f30e7e00640 1 -- 172.21.15.116:0/841362267 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f30fc003970 con 0x5598418866b0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30fc032c90 con 0x5598418866b0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.095+0000 7f30e7e00640 1 -- 172.21.15.116:0/841362267 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30fc039070 con 0x5598418866b0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 10 client.25805 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5598414ba0f0 con 0x5598418866b0 2024-03-04T07:42:45.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f30e7e00640 1 -- 172.21.15.116:0/841362267 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30fc032c90 con 0x5598418866b0 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f30e7e00640 1 -- 172.21.15.116:0/841362267 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f30fc004490 con 0x5598418866b0 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 10 client.25805 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 20 client.25805 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418965d0 0x5598418989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 4 client.25805 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598417f3270 con 0x5598418965d0 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.096+0000 7f3101e00640 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418965d0 0x5598418989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.097+0000 7f3101e00640 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418965d0 0x5598418989c0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e95eb326eb22b4e server_cookie=b39d5c0ff19b920e in_seq=0 out_seq=0 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.097+0000 7f30e7e00640 10 client.25805 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.097+0000 7f30e7e00640 1 -- 172.21.15.116:0/841362267 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5598417f3270 con 0x5598418965d0 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.097+0000 7f30e7e00640 10 client.25805 handle_command_reply: tid=0 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.097+0000 7f310bf84740 1 client.25805 shutdown 2024-03-04T07:42:45.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5598418866b0 msgr2=0x559841892470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5598418866b0 0x559841892470 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009ca0 tx=0x7f30fc0093f0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 shutdown_connections 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418965d0 0x5598418989c0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5598418866b0 0x559841892470 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x55984188b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 --2- 172.21.15.116:0/841362267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 >> 172.21.15.116:0/841362267 conn(0x55984187c0c0 msgr2=0x55984187b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f30e7e00640 0 client.25805 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 shutdown_connections 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 1 -- 172.21.15.116:0/841362267 wait complete. 2024-03-04T07:42:45.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.098+0000 7f310bf84740 20 client.25805 trim_cache size 0 max 16384 2024-03-04T07:42:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 Processor -- start 2024-03-04T07:42:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 -- start start 2024-03-04T07:42:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841881540 0x559841892430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x55984188b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041acfa0 con 0x5598418866b0 2024-03-04T07:42:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041af580 con 0x559841881540 2024-03-04T07:42:45.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.115+0000 7f310bf84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041ad5a0 con 0x559841880b50 2024-03-04T07:42:45.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841881540 0x559841892430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f3101e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841881540 0x559841892430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43230/0 (socket says 172.21.15.116:43230) 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 -- 172.21.15.116:0/1982254648 learned_addr learned my addr 172.21.15.116:0/1982254648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f3102800640 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x55984188b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 -- 172.21.15.116:0/1982254648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841880b50 msgr2=0x55984188aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 -- 172.21.15.116:0/1982254648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 msgr2=0x55984188b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x55984188b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 -- 172.21.15.116:0/1982254648 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559841889760 con 0x559841881540 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.116+0000 7f30f9400640 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841881540 0x559841892430 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00e9c0 tx=0x7f30ec0d17a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b5f89b12d4e670 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f30f2a00640 1 -- 172.21.15.116:0/1982254648 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec01c070 con 0x559841881540 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f30f2a00640 1 -- 172.21.15.116:0/1982254648 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30ec0051d0 con 0x559841881540 2024-03-04T07:42:45.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f30f2a00640 1 -- 172.21.15.116:0/1982254648 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec0970c0 con 0x559841881540 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 -- 172.21.15.116:0/1982254648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841881540 msgr2=0x559841892430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841881540 0x559841892430 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00e9c0 tx=0x7f30ec0d17a0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 -- 172.21.15.116:0/1982254648 shutdown_connections 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x55984188b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841881540 0x559841892430 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 --2- 172.21.15.116:0/1982254648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841880b50 0x55984188aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 -- 172.21.15.116:0/1982254648 >> 172.21.15.116:0/1982254648 conn(0x55984187c0c0 msgr2=0x5598418899d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 -- 172.21.15.116:0/1982254648 shutdown_connections 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.117+0000 7f310bf84740 1 -- 172.21.15.116:0/1982254648 wait complete. 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 -- start start 2024-03-04T07:42:45.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 Processor -- start 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x55984188ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x5598418904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x559841891500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041adc00 con 0x5598418866b0 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041aef20 con 0x559841880b50 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.118+0000 7f310bf84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041ae260 con 0x559841881540 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f310bf84740 10 client.0 _ll_register_callbacks cb 0x7f30ec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:45.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f3101e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x55984188ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f3101e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x55984188ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:43232/0 (socket says 172.21.15.116:43232) 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f3101e00640 1 -- 172.21.15.116:0/898912941 learned_addr learned my addr 172.21.15.116:0/898912941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30f9400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x5598418904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f3102800640 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x559841891500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30f9400640 1 -- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 msgr2=0x55984188ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30f9400640 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x55984188ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30f9400640 1 -- 172.21.15.116:0/898912941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 msgr2=0x559841891500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30f9400640 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x559841891500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30f9400640 1 -- 172.21.15.116:0/898912941 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559841891af0 con 0x559841881540 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f3101e00640 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x55984188ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30f9400640 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x5598418904f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55984187c720 tx=0x7f30ec09eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96d9c15f88e82a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.119+0000 7f30e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:42:45.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f30e7e00640 1 -- 172.21.15.116:0/898912941 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec01c040 con 0x559841881540 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f30e7e00640 1 -- 172.21.15.116:0/898912941 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f30ec004ca0 con 0x559841881540 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30ec0051d0 con 0x559841881540 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f30e7e00640 1 -- 172.21.15.116:0/898912941 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30ec09aa50 con 0x559841881540 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 10 client.25771 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5598418f20b0 con 0x559841881540 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f30e7e00640 1 -- 172.21.15.116:0/898912941 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30ec0051d0 con 0x559841881540 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f30e7e00640 1 -- 172.21.15.116:0/898912941 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f30ec0cbce0 con 0x559841881540 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 10 client.25771 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 10 client.25771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 20 client.25771 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418f2a10 0x5598418f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 4 client.25771 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.120+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55984188ace0 con 0x5598418f2a10 2024-03-04T07:42:45.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f3101e00640 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418f2a10 0x5598418f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f3101e00640 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418f2a10 0x5598418f4e30 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dd116a77d099956 server_cookie=54443f86c83dcf0f in_seq=0 out_seq=0 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f30e7e00640 10 client.25771 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f30e7e00640 1 -- 172.21.15.116:0/898912941 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55984188ace0 con 0x5598418f2a10 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f30e7e00640 10 client.25771 handle_command_reply: tid=0 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f310bf84740 1 client.25771 shutdown 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 msgr2=0x5598418904f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.121+0000 7f310bf84740 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x5598418904f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55984187c720 tx=0x7f30ec09eba0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 shutdown_connections 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5598418f2a10 0x5598418f4e30 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598418866b0 0x559841891500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559841881540 0x5598418904f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 --2- 172.21.15.116:0/898912941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559841880b50 0x55984188ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f30e7e00640 0 client.25771 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:45.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 >> 172.21.15.116:0/898912941 conn(0x55984187c0c0 msgr2=0x55984188aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 shutdown_connections 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 1 -- 172.21.15.116:0/898912941 wait complete. 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f310bf84740 20 client.25771 trim_cache size 0 max 16384 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stdout: "6e45eafc-bc0b-4c23-ab47-dd2216d6c78e": { 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:42:45.128 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "6e45eafc-bc0b-4c23-ab47-dd2216d6c78e", 2024-03-04T07:42:45.129 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:42:45.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:42:45.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:42:45.129 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:42:45.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30e80a1d80 msgr2=0x7f30e80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30e80a1d80 0x7f30e80a4240 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x559841881940 tx=0x7f30f4005f90 comp rx=0 tx=0).stop 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 msgr2=0x7f310419e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 0x7f310419e3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30ec00e9f0 tx=0x7f30ec00eeb0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 shutdown_connections 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31041097a0 0x7f31041a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30e80a1d80 0x7f30e80a4240 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3104108de0 0x7f310419e3c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 --2- 172.21.15.116:0/2866108694 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3104107bf0 0x7f310419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.122+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 >> 172.21.15.116:0/2866108694 conn(0x7f3104076b90 msgr2=0x7f310410c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.123+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 shutdown_connections 2024-03-04T07:42:45.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:42:45.123+0000 7f3109200640 1 -- 172.21.15.116:0/2866108694 wait complete. 2024-03-04T07:42:45.142 INFO:tasks.cephfs.filesystem:scrub status for tag:6e45eafc-bc0b-4c23-ab47-dd2216d6c78e - {'path': '/', 'tag': '6e45eafc-bc0b-4c23-ab47-dd2216d6c78e', 'options': 'recursive,force'} 2024-03-04T07:42:45.200 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:44 smithi152 ceph-mon[37154]: pgmap v460: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:42:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:46 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:42:47.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:46 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:47.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:46 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:42:47.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:46 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:47.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:46 smithi133 ceph-mon[36505]: pgmap v461: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:42:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:46 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:42:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:46 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:46 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:42:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:46 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:46 smithi116 ceph-mon[32150]: pgmap v461: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:42:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:46 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:42:47.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:46 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:47.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:46 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:42:47.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:46 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:42:47.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:46 smithi152 ceph-mon[37154]: pgmap v461: 97 pgs: 97 active+clean; 1.2 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:42:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:48 smithi133 ceph-mon[36505]: pgmap v462: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:42:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:48 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:42:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:48 smithi116 ceph-mon[32150]: pgmap v462: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:42:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:48 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:42:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:48 smithi152 ceph-mon[37154]: pgmap v462: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:42:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:48 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:42:51.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:50 smithi133 ceph-mon[36505]: pgmap v463: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:42:51.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:50 smithi116 ceph-mon[32150]: pgmap v463: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:42:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:50 smithi152 ceph-mon[37154]: pgmap v463: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:42:53.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:52 smithi133 ceph-mon[36505]: pgmap v464: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:42:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:52 smithi116 ceph-mon[32150]: pgmap v464: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:42:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:52 smithi152 ceph-mon[37154]: pgmap v464: 97 pgs: 97 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:42:55.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:54 smithi133 ceph-mon[36505]: pgmap v465: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:42:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:54 smithi116 ceph-mon[32150]: pgmap v465: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:42:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:54 smithi152 ceph-mon[37154]: pgmap v465: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:42:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.991+0000 7f2c2ae00640 1 -- 172.21.15.116:0/1697769275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c241097e0 msgr2=0x7f2c24110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.991+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/1697769275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c241097e0 0x7f2c24110070 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009be0 tx=0x7f2c1c02bef0 comp rx=0 tx=0).stop 2024-03-04T07:42:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.992+0000 7f2c2ae00640 1 -- 172.21.15.116:0/1697769275 shutdown_connections 2024-03-04T07:42:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.992+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/1697769275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c241097e0 0x7f2c24110070 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.992+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/1697769275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c24108e20 0x7f2c241092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.992+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/1697769275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24107c30 0x7f2c24108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.992+0000 7f2c2ae00640 1 -- 172.21.15.116:0/1697769275 >> 172.21.15.116:0/1697769275 conn(0x7f2c24076be0 msgr2=0x7f2c24079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.992+0000 7f2c2ae00640 1 -- 172.21.15.116:0/1697769275 shutdown_connections 2024-03-04T07:42:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.992+0000 7f2c2ae00640 1 -- 172.21.15.116:0/1697769275 wait complete. 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 Processor -- start 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 -- start start 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24107c30 0x7f2c2419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c24108e20 0x7f2c2419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c241097e0 0x7f2c241a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c24113890 con 0x7f2c241097e0 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c24113710 con 0x7f2c24108e20 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.993+0000 7f2c2ae00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c24113a10 con 0x7f2c24107c30 2024-03-04T07:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c24108e20 0x7f2c2419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24107c30 0x7f2c2419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c28e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c241097e0 0x7f2c241a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c24108e20 0x7f2c2419e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52090/0 (socket says 172.21.15.116:52090) 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 -- 172.21.15.116:0/3254664035 learned_addr learned my addr 172.21.15.116:0/3254664035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 -- 172.21.15.116:0/3254664035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24107c30 msgr2=0x7f2c2419de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24107c30 0x7f2c2419de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 -- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c241097e0 msgr2=0x7f2c241a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c241097e0 0x7f2c241a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 -- 172.21.15.116:0/3254664035 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c241a5a60 con 0x7f2c24108e20 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c23400640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c24108e20 0x7f2c2419e380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000e9c0 tx=0x7f2c1000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c20c00640 1 -- 172.21.15.116:0/3254664035 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1000cde0 con 0x7f2c24108e20 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c20c00640 1 -- 172.21.15.116:0/3254664035 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c100035f0 con 0x7f2c24108e20 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.994+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c241a5d50 con 0x7f2c24108e20 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.995+0000 7f2c20c00640 1 -- 172.21.15.116:0/3254664035 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10010690 con 0x7f2c24108e20 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.995+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c241a6260 con 0x7f2c24108e20 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 Processor -- start 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 -- start start 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c20c00640 1 -- 172.21.15.116:0/3254664035 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2c100037a0 con 0x7f2c24108e20 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd51ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51f240 0x558fcd523d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd5243b0 0x558fcd526840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fcd484370 con 0x558fcd51e850 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fcd42d550 con 0x558fcd5243b0 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fcd480410 con 0x558fcd51f240 2024-03-04T07:42:57.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c28e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd51ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c28e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd51ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53310/0 (socket says 172.21.15.116:53310) 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c28e00640 1 -- 172.21.15.116:0/2192358483 learned_addr learned my addr 172.21.15.116:0/2192358483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c23e00640 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51f240 0x558fcd523d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.998+0000 7f2c23400640 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd5243b0 0x558fcd526840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.999+0000 7f2c28e00640 1 -- 172.21.15.116:0/2192358483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51f240 msgr2=0x558fcd523d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.999+0000 7f2c28e00640 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51f240 0x558fcd523d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.999+0000 7f2c28e00640 1 -- 172.21.15.116:0/2192358483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd5243b0 msgr2=0x558fcd526840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.999+0000 7f2c28e00640 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd5243b0 0x558fcd526840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.999+0000 7f2c28e00640 1 -- 172.21.15.116:0/2192358483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fcd4413c0 con 0x558fcd51e850 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.999+0000 7f2c28e00640 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd51ec70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009be0 tx=0x7f2c1c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1adb030dd6116ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:56.999+0000 7f2c19600640 1 -- 172.21.15.116:0/2192358483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c043030 con 0x558fcd51e850 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c19600640 1 -- 172.21.15.116:0/2192358483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c1c033b90 con 0x558fcd51e850 2024-03-04T07:42:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c19600640 1 -- 172.21.15.116:0/2192358483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c02c890 con 0x558fcd51e850 2024-03-04T07:42:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c20c00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2c080a1ea0 0x7f2c080a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c20c00640 1 -- 172.21.15.116:0/3254664035 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2c100cdfe0 con 0x7f2c24108e20 2024-03-04T07:42:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c23e00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2c080a1ea0 0x7f2c080a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/2192358483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 msgr2=0x558fcd51ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd51ec70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009be0 tx=0x7f2c1c009310 comp rx=0 tx=0).stop 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/2192358483 shutdown_connections 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd5243b0 0x558fcd526840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51f240 0x558fcd523d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/2192358483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd51ec70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.000+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/2192358483 >> 172.21.15.116:0/2192358483 conn(0x558fcd519dc0 msgr2=0x558fcd51a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.001+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/2192358483 shutdown_connections 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.001+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/2192358483 wait complete. 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.001+0000 7f2c23e00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2c080a1ea0 0x7f2c080a4360 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x558fcd51f640 tx=0x7f2c14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.001+0000 7f2c2d9f9740 1 -- start start 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.001+0000 7f2c2d9f9740 1 Processor -- start 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5290a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd5300e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fcd527300 con 0x558fcd51e850 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fcd4409b0 con 0x558fcd51f240 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fcd527480 con 0x558fcd5243b0 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c2d9f9740 10 client.0 _ll_register_callbacks cb 0x558fcd47c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5290a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52114/0 (socket says 172.21.15.116:52114) 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 -- 172.21.15.116:0/1390309502 learned_addr learned my addr 172.21.15.116:0/1390309502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23e00640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c28e00640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd5300e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 -- 172.21.15.116:0/1390309502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 msgr2=0x558fcd5300e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd5300e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 -- 172.21.15.116:0/1390309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 msgr2=0x558fcd528b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c23400640 1 -- 172.21.15.116:0/1390309502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fcd199210 con 0x558fcd51f240 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.002+0000 7f2c28e00640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd5300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c23400640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5290a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2c100a2f80 tx=0x7f2c10096890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdedd33619d6c144 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c0e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c0e000640 1 -- 172.21.15.116:0/1390309502 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1009a850 con 0x558fcd51f240 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c0e000640 1 -- 172.21.15.116:0/1390309502 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2c100d0110 con 0x558fcd51f240 2024-03-04T07:42:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c1c033b90 con 0x558fcd51f240 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c0e000640 1 -- 172.21.15.116:0/1390309502 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c1009a670 con 0x558fcd51f240 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 10 client.25823 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558fcd265620 con 0x558fcd51f240 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c0e000640 1 -- 172.21.15.116:0/1390309502 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c1c033b90 con 0x558fcd51f240 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c0e000640 1 -- 172.21.15.116:0/1390309502 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2c100a3020 con 0x558fcd51f240 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 10 client.25823 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 10 client.25823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 20 client.25823 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd534980 0x558fcd536d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 4 client.25823 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.003+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fcd4918b0 con 0x558fcd534980 2024-03-04T07:42:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.004+0000 7f2c23e00640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd534980 0x558fcd536d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.004+0000 7f2c23e00640 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd534980 0x558fcd536d90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65a00c114671e066 server_cookie=13250224f8cbe2fc in_seq=0 out_seq=0 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.004+0000 7f2c0e000640 10 client.25823 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c0e000640 1 -- 172.21.15.116:0/1390309502 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558fcd4918b0 con 0x558fcd534980 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c0e000640 10 client.25823 handle_command_reply: tid=0 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 client.25823 shutdown 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 msgr2=0x558fcd5290a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5290a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2c100a2f80 tx=0x7f2c10096890 comp rx=0 tx=0).stop 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 shutdown_connections 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd534980 0x558fcd536d90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd5300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5290a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1390309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c0e000640 0 client.25823 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 >> 172.21.15.116:0/1390309502 conn(0x558fcd519dc0 msgr2=0x558fcd518fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 shutdown_connections 2024-03-04T07:42:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.005+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1390309502 wait complete. 2024-03-04T07:42:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.006+0000 7f2c2d9f9740 20 client.25823 trim_cache size 0 max 16384 2024-03-04T07:42:57.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.020+0000 7f2c2d9f9740 1 Processor -- start 2024-03-04T07:42:57.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.020+0000 7f2c2d9f9740 1 -- start start 2024-03-04T07:42:57.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.020+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd51d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241acf80 con 0x558fcd51e850 2024-03-04T07:42:57.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241af560 con 0x558fcd51f240 2024-03-04T07:42:57.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241ad580 con 0x558fcd5243b0 2024-03-04T07:42:57.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5300a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52116/0 (socket says 172.21.15.116:52116) 2024-03-04T07:42:57.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 -- 172.21.15.116:0/3863447847 learned_addr learned my addr 172.21.15.116:0/3863447847 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:57.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 -- 172.21.15.116:0/3863447847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 msgr2=0x558fcd51d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c28e00640 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd51d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd51d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 -- 172.21.15.116:0/3863447847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 msgr2=0x558fcd528180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 -- 172.21.15.116:0/3863447847 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fcd537410 con 0x558fcd51f240 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c28e00640 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd51d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.021+0000 7f2c23400640 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5300a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2c100976e0 tx=0x7f2c100cbb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aa208adbd490046 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c18c00640 1 -- 172.21.15.116:0/3863447847 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1009a850 con 0x558fcd51f240 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c18c00640 1 -- 172.21.15.116:0/3863447847 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c1009b410 con 0x558fcd51f240 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c18c00640 1 -- 172.21.15.116:0/3863447847 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10096fb0 con 0x558fcd51f240 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/3863447847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 msgr2=0x558fcd5300a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5300a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2c100976e0 tx=0x7f2c100cbb80 comp rx=0 tx=0).stop 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/3863447847 shutdown_connections 2024-03-04T07:42:57.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd5243b0 0x558fcd51d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd5300a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/3863447847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd51e850 0x558fcd528180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/3863447847 >> 172.21.15.116:0/3863447847 conn(0x558fcd519dc0 msgr2=0x558fcd51d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/3863447847 shutdown_connections 2024-03-04T07:42:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.022+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/3863447847 wait complete. 2024-03-04T07:42:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 -- start start 2024-03-04T07:42:57.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 Processor -- start 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51e850 0x558fcd52fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd52e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd5243b0 0x558fcd52e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241adbe0 con 0x558fcd5243b0 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241aef00 con 0x558fcd51f240 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c241ae240 con 0x558fcd51e850 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.023+0000 7f2c2d9f9740 10 client.0 _ll_register_callbacks cb 0x7f2c10003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd52e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd52e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52130/0 (socket says 172.21.15.116:52130) 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 -- 172.21.15.116:0/1465680158 learned_addr learned my addr 172.21.15.116:0/1465680158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23e00640 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51e850 0x558fcd52fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c28e00640 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd5243b0 0x558fcd52e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 -- 172.21.15.116:0/1465680158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51e850 msgr2=0x558fcd52fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51e850 0x558fcd52fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 -- 172.21.15.116:0/1465680158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd5243b0 msgr2=0x558fcd52e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd5243b0 0x558fcd52e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 -- 172.21.15.116:0/1465680158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fcd52ee40 con 0x558fcd51f240 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c23400640 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd52e390 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2c1009aca0 tx=0x7f2c100a32c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9575d13cfe23177c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c0e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c0e000640 1 -- 172.21.15.116:0/1465680158 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100b2070 con 0x558fcd51f240 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c0e000640 1 -- 172.21.15.116:0/1465680158 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2c10004210 con 0x558fcd51f240 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.024+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c1009b410 con 0x558fcd51f240 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c0e000640 1 -- 172.21.15.116:0/1465680158 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c1009a200 con 0x558fcd51f240 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 10 client.25829 fetch_fsmap learned FSMap version 19 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558fcd590f80 con 0x558fcd51f240 2024-03-04T07:42:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c0e000640 1 -- 172.21.15.116:0/1465680158 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c1009b410 con 0x558fcd51f240 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c0e000640 1 -- 172.21.15.116:0/1465680158 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2c10099bd0 con 0x558fcd51f240 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 10 client.25829 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 10 client.25829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 20 client.25829 populate_metadata read hostname 'smithi116' 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd5916c0 0x558fcd593ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 4 client.25829 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558fcd528950 con 0x558fcd5916c0 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.025+0000 7f2c23e00640 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd5916c0 0x558fcd593ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.026+0000 7f2c23e00640 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd5916c0 0x558fcd593ae0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ee7e8aa1b19e8bc server_cookie=7f15be255c97f1f1 in_seq=0 out_seq=0 2024-03-04T07:42:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.026+0000 7f2c0e000640 10 client.25829 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c0e000640 1 -- 172.21.15.116:0/1465680158 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558fcd528950 con 0x558fcd5916c0 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c0e000640 10 client.25829 handle_command_reply: tid=0 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 client.25829 shutdown 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 msgr2=0x558fcd52e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd52e390 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2c1009aca0 tx=0x7f2c100a32c0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 shutdown_connections 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558fcd5916c0 0x558fcd593ae0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fcd5243b0 0x558fcd52e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558fcd51f240 0x558fcd52e390 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 --2- 172.21.15.116:0/1465680158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558fcd51e850 0x558fcd52fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c0e000640 0 client.25829 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 >> 172.21.15.116:0/1465680158 conn(0x558fcd519dc0 msgr2=0x558fcd51d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 shutdown_connections 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 1 -- 172.21.15.116:0/1465680158 wait complete. 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.041+0000 7f2c2d9f9740 20 client.25829 trim_cache size 0 max 16384 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:42:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:42:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:42:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2c080a1ea0 msgr2=0x7f2c080a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2c080a1ea0 0x7f2c080a4360 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x558fcd51f640 tx=0x7f2c14005f90 comp rx=0 tx=0).stop 2024-03-04T07:42:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c24108e20 msgr2=0x7f2c2419e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:42:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c24108e20 0x7f2c2419e380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000e9c0 tx=0x7f2c1000ee80 comp rx=0 tx=0).stop 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 shutdown_connections 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c241097e0 0x7f2c241a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2c24108e20 0x7f2c2419e380 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2c080a1ea0 0x7f2c080a4360 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 --2- 172.21.15.116:0/3254664035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24107c30 0x7f2c2419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 >> 172.21.15.116:0/3254664035 conn(0x7f2c24076be0 msgr2=0x7f2c2410c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 shutdown_connections 2024-03-04T07:42:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:42:57.042+0000 7f2c2ae00640 1 -- 172.21.15.116:0/3254664035 wait complete. 2024-03-04T07:42:57.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:56 smithi133 ceph-mon[36505]: pgmap v466: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:42:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:56 smithi116 ceph-mon[32150]: pgmap v466: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:42:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:56 smithi152 ceph-mon[37154]: pgmap v466: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:42:59.061 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:42:58 smithi116 ceph-mon[32150]: pgmap v467: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:42:59.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:42:58 smithi133 ceph-mon[36505]: pgmap v467: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:42:59.257 INFO:tasks.workunit.client.0.smithi116.stdout:1/600: open fd = 3 2024-03-04T07:42:59.257 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:2/600: open fd = 3 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:3/600: open fd = 3 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:4/600: open fd = 3 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.258 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:5/600: open fd = 3 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:6/600: open fd = 3 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:7/600: open fd = 3 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.259 INFO:tasks.workunit.client.0.smithi116.stdout:8/600: open fd = 3 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:9/600: open fd = 3 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:10/600: open fd = 3 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:11/600: open fd = 3 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.260 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:12/600: open fd = 3 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:13/600: open fd = 3 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:14/600: open fd = 3 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:15/600: open fd = 3 2024-03-04T07:42:59.261 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:16/600: open fd = 3 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:17/600: open fd = 3 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:18/600: open fd = 3 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.262 INFO:tasks.workunit.client.0.smithi116.stdout:19/600: open fd = 3 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:20/600: open fd = 3 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:21/600: open fd = 3 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:22/600: open fd = 3 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.263 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:23/600: open fd = 3 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:24/600: open fd = 3 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:25/600: open fd = 3 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.264 INFO:tasks.workunit.client.0.smithi116.stdout:26/600: open fd = 3 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:27/600: open fd = 3 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:28/600: open fd = 3 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:29/600: open fd = 3 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.265 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:30/600: open fd = 3 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:31/600: open fd = 3 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:32/600: open fd = 3 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:33/600: open fd = 3 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.266 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:34/600: open fd = 3 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:35/600: open fd = 3 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:36/600: open fd = 3 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:37/600: open fd = 3 2024-03-04T07:42:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:38/600: open fd = 3 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:39/600: open fd = 3 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:40/600: open fd = 3 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.268 INFO:tasks.workunit.client.0.smithi116.stdout:41/600: open fd = 3 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:42/600: open fd = 3 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:43/600: open fd = 3 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:44/600: open fd = 3 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.269 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:45/600: open fd = 3 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:46/600: open fd = 3 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:47/600: open fd = 3 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:48/600: open fd = 3 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.270 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:49/600: open fd = 3 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:50/600: open fd = 3 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:51/600: open fd = 3 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:52/600: open fd = 3 2024-03-04T07:42:59.271 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:53/600: open fd = 3 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:54/600: open fd = 3 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:55/600: open fd = 3 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.272 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:56/600: open fd = 3 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:57/600: open fd = 3 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:58/600: open fd = 3 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:59/600: open fd = 3 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.273 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:60/600: open fd = 3 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:61/600: open fd = 3 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:62/600: open fd = 3 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:63/600: open fd = 3 2024-03-04T07:42:59.274 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.275 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:42:59.275 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:42:59.275 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:42:58 smithi152 ceph-mon[37154]: pgmap v467: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:43:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:00 smithi133 ceph-mon[36505]: pgmap v468: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:43:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:00 smithi116 ceph-mon[32150]: pgmap v468: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:43:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:00 smithi152 ceph-mon[37154]: pgmap v468: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:43:03.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:02 smithi133 ceph-mon[36505]: pgmap v469: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:43:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:02 smithi116 ceph-mon[32150]: pgmap v469: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:43:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:02 smithi152 ceph-mon[37154]: pgmap v469: 97 pgs: 97 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:43:05.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:04 smithi133 ceph-mon[36505]: pgmap v470: 97 pgs: 97 active+clean; 1.5 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:43:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:04 smithi116 ceph-mon[32150]: pgmap v470: 97 pgs: 97 active+clean; 1.5 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:43:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:04 smithi152 ceph-mon[37154]: pgmap v470: 97 pgs: 97 active+clean; 1.5 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:43:07.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:06 smithi133 ceph-mon[36505]: pgmap v471: 97 pgs: 97 active+clean; 1.5 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:06 smithi116 ceph-mon[32150]: pgmap v471: 97 pgs: 97 active+clean; 1.5 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:06 smithi152 ceph-mon[37154]: pgmap v471: 97 pgs: 97 active+clean; 1.5 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:09.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:08 smithi133 ceph-mon[36505]: pgmap v472: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:08 smithi116 ceph-mon[32150]: pgmap v472: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:08 smithi152 ceph-mon[37154]: pgmap v472: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.184+0000 7f67a3800640 1 -- 172.21.15.116:0/388837770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 msgr2=0x7f679c078410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.184+0000 7f67a3800640 1 --2- 172.21.15.116:0/388837770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 0x7f679c078410 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6794009ca0 tx=0x7f679402c0f0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.185+0000 7f67a3800640 1 -- 172.21.15.116:0/388837770 shutdown_connections 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.185+0000 7f67a3800640 1 --2- 172.21.15.116:0/388837770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 0x7f679c078410 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.185+0000 7f67a3800640 1 --2- 172.21.15.116:0/388837770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f679c0771d0 0x7f679c077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.185+0000 7f67a3800640 1 --2- 172.21.15.116:0/388837770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f679c075f90 0x7f679c0763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.185+0000 7f67a3800640 1 -- 172.21.15.116:0/388837770 >> 172.21.15.116:0/388837770 conn(0x7f679c0fed90 msgr2=0x7f679c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.185+0000 7f67a3800640 1 -- 172.21.15.116:0/388837770 shutdown_connections 2024-03-04T07:43:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.185+0000 7f67a3800640 1 -- 172.21.15.116:0/388837770 wait complete. 2024-03-04T07:43:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 Processor -- start 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 -- start start 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f679c075f90 0x7f679c19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f679c0771d0 0x7f679c19e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 0x7f679c1a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1138b0 con 0x7f679c0771d0 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c113730 con 0x7f679c077b70 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a3800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c113a30 con 0x7f679c075f90 2024-03-04T07:43:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f679c0771d0 0x7f679c19e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a1800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 0x7f679c1a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f679c0771d0 0x7f679c19e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56442/0 (socket says 172.21.15.116:56442) 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f679be00640 1 -- 172.21.15.116:0/4088752427 learned_addr learned my addr 172.21.15.116:0/4088752427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.186+0000 7f67a0e00640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f679c075f90 0x7f679c19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a1800640 1 -- 172.21.15.116:0/4088752427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f679c075f90 msgr2=0x7f679c19de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a1800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f679c075f90 0x7f679c19de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a1800640 1 -- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f679c0771d0 msgr2=0x7f679c19e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a1800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f679c0771d0 0x7f679c19e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a1800640 1 -- 172.21.15.116:0/4088752427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c1a5a30 con 0x7f679c077b70 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a1800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 0x7f679c1a53c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6794037040 tx=0x7f6794032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f6799600640 1 -- 172.21.15.116:0/4088752427 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794041020 con 0x7f679c077b70 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f6799600640 1 -- 172.21.15.116:0/4088752427 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6794034cd0 con 0x7f679c077b70 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c1a5cc0 con 0x7f679c077b70 2024-03-04T07:43:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f6799600640 1 -- 172.21.15.116:0/4088752427 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794039040 con 0x7f679c077b70 2024-03-04T07:43:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.187+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c1a61d0 con 0x7f679c077b70 2024-03-04T07:43:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.190+0000 7f6799600640 1 -- 172.21.15.116:0/4088752427 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f6794044020 con 0x7f679c077b70 2024-03-04T07:43:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 Processor -- start 2024-03-04T07:43:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 -- start start 2024-03-04T07:43:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f507c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f50cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f50f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6f46d370 con 0x559c6f507850 2024-03-04T07:43:10.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6f416550 con 0x559c6f50d3b0 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a63f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6f469410 con 0x559c6f508240 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f507c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f507c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56446/0 (socket says 172.21.15.116:56446) 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f679be00640 1 -- 172.21.15.116:0/648742292 learned_addr learned my addr 172.21.15.116:0/648742292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f67a0e00640 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f50cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.191+0000 7f679be00640 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f50f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f67a0e00640 1 -- 172.21.15.116:0/648742292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 msgr2=0x559c6f50f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f67a0e00640 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f50f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f67a0e00640 1 -- 172.21.15.116:0/648742292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 msgr2=0x559c6f507c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f67a0e00640 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f507c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f67a0e00640 1 -- 172.21.15.116:0/648742292 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6f42a3c0 con 0x559c6f508240 2024-03-04T07:43:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f6799600640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f67840a1ea0 0x7f67840a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f6799600640 1 -- 172.21.15.116:0/4088752427 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f67940efb10 con 0x7f679c077b70 2024-03-04T07:43:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f67a0e00640 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f50cd40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6790009c20 tx=0x7f6790011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2d57d12b0ba0799 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f679be00640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f67840a1ea0 0x7f67840a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.192+0000 7f678e000640 1 -- 172.21.15.116:0/648742292 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6790004050 con 0x559c6f508240 2024-03-04T07:43:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f678e000640 1 -- 172.21.15.116:0/648742292 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6790005500 con 0x559c6f508240 2024-03-04T07:43:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f678e000640 1 -- 172.21.15.116:0/648742292 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6790004d50 con 0x559c6f508240 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 -- 172.21.15.116:0/648742292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 msgr2=0x559c6f50cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f679be00640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f67840a1ea0 0x7f67840a4360 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x559c6f508aa0 tx=0x7f6788009880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f50cd40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6790009c20 tx=0x7f6790011810 comp rx=0 tx=0).stop 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 -- 172.21.15.116:0/648742292 shutdown_connections 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f50f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f50cd40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 --2- 172.21.15.116:0/648742292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f507c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 -- 172.21.15.116:0/648742292 >> 172.21.15.116:0/648742292 conn(0x559c6f502dc0 msgr2=0x559c6f5031c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.193+0000 7f67a63f5740 1 -- 172.21.15.116:0/648742292 shutdown_connections 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.194+0000 7f67a63f5740 1 -- 172.21.15.116:0/648742292 wait complete. 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.194+0000 7f67a63f5740 1 -- start start 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.194+0000 7f67a63f5740 1 Processor -- start 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.194+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f511b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f5120c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f519090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a63f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6f510360 con 0x559c6f507850 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a63f5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6f4299b0 con 0x559c6f50d3b0 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a63f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6f5104e0 con 0x559c6f508240 2024-03-04T07:43:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a63f5740 10 client.0 _ll_register_callbacks cb 0x559c6f465340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f511b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a1800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f519090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f511b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56454/0 (socket says 172.21.15.116:56454) 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a0e00640 1 -- 172.21.15.116:0/2650262179 learned_addr learned my addr 172.21.15.116:0/2650262179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f679be00640 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f5120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a1800640 1 -- 172.21.15.116:0/2650262179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 msgr2=0x559c6f5120c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a1800640 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f5120c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a1800640 1 -- 172.21.15.116:0/2650262179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 msgr2=0x559c6f511b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a1800640 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f511b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.195+0000 7f67a1800640 1 -- 172.21.15.116:0/2650262179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c6f182210 con 0x559c6f50d3b0 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a1800640 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f519090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f679402c700 tx=0x7f67940bc3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=535e5023dc46f285 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f6782a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:10.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f6782a00640 1 -- 172.21.15.116:0/2650262179 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940098b0 con 0x559c6f50d3b0 2024-03-04T07:43:10.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f6782a00640 1 -- 172.21.15.116:0/2650262179 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f67940a70b0 con 0x559c6f50d3b0 2024-03-04T07:43:10.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6790005500 con 0x559c6f50d3b0 2024-03-04T07:43:10.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f6782a00640 1 -- 172.21.15.116:0/2650262179 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67940a72b0 con 0x559c6f50d3b0 2024-03-04T07:43:10.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 10 client.25841 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:10.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559c6f24e620 con 0x559c6f50d3b0 2024-03-04T07:43:10.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f6782a00640 1 -- 172.21.15.116:0/2650262179 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6790005500 con 0x559c6f50d3b0 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f6782a00640 1 -- 172.21.15.116:0/2650262179 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f679403c030 con 0x559c6f50d3b0 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 10 client.25841 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 10 client.25841 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 20 client.25841 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f51d960 0x559c6f51fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 4 client.25841 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.196+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c6f47a8b0 con 0x559c6f51d960 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.197+0000 7f67a0e00640 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f51d960 0x559c6f51fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.197+0000 7f67a0e00640 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f51d960 0x559c6f51fdd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54c9a0f9afa9d3d6 server_cookie=63cfbb3626da6e3c in_seq=0 out_seq=0 2024-03-04T07:43:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.197+0000 7f6782a00640 10 client.25841 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f6782a00640 1 -- 172.21.15.116:0/2650262179 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559c6f47a8b0 con 0x559c6f51d960 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f6782a00640 10 client.25841 handle_command_reply: tid=0 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 client.25841 shutdown 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 msgr2=0x559c6f519090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f519090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f679402c700 tx=0x7f67940bc3d0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 shutdown_connections 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f51d960 0x559c6f51fdd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f50d3b0 0x559c6f519090 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f508240 0x559c6f5120c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 --2- 172.21.15.116:0/2650262179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f511b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f6782a00640 0 client.25841 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 >> 172.21.15.116:0/2650262179 conn(0x559c6f502dc0 msgr2=0x559c6f501fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 shutdown_connections 2024-03-04T07:43:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.198+0000 7f67a63f5740 1 -- 172.21.15.116:0/2650262179 wait complete. 2024-03-04T07:43:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.199+0000 7f67a63f5740 20 client.25841 trim_cache size 0 max 16384 2024-03-04T07:43:10.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.213+0000 7f67a63f5740 1 Processor -- start 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.213+0000 7f67a63f5740 1 -- start start 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f519ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f519050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f51a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a63f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1aceb0 con 0x559c6f507850 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a63f5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1af490 con 0x559c6f508240 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a63f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1ad4b0 con 0x559c6f502dc0 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f519050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a1800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f51a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f519050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56456/0 (socket says 172.21.15.116:56456) 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f679be00640 1 -- 172.21.15.116:0/233114942 learned_addr learned my addr 172.21.15.116:0/233114942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a0e00640 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f519ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a1800640 1 -- 172.21.15.116:0/233114942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 msgr2=0x559c6f519ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a1800640 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f519ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a1800640 1 -- 172.21.15.116:0/233114942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 msgr2=0x559c6f519050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a1800640 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f519050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.214+0000 7f67a1800640 1 -- 172.21.15.116:0/233114942 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6f503b30 con 0x559c6f508240 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.215+0000 7f67a1800640 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f51a020 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f67940bab40 tx=0x7f6794005f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d14121e8d153492 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.215+0000 7f678d600640 1 -- 172.21.15.116:0/233114942 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940bc1f0 con 0x559c6f508240 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.215+0000 7f678d600640 1 -- 172.21.15.116:0/233114942 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67940a70e0 con 0x559c6f508240 2024-03-04T07:43:10.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.215+0000 7f678d600640 1 -- 172.21.15.116:0/233114942 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940bde20 con 0x559c6f508240 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.215+0000 7f67a63f5740 1 -- 172.21.15.116:0/233114942 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 msgr2=0x559c6f51a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.215+0000 7f67a63f5740 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f51a020 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f67940bab40 tx=0x7f6794005f60 comp rx=0 tx=0).stop 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 -- 172.21.15.116:0/233114942 shutdown_connections 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f51a020 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f519050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 --2- 172.21.15.116:0/233114942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f519ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 -- 172.21.15.116:0/233114942 >> 172.21.15.116:0/233114942 conn(0x559c6f50ceb0 msgr2=0x559c6f506020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 -- 172.21.15.116:0/233114942 shutdown_connections 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 -- 172.21.15.116:0/233114942 wait complete. 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 -- start start 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.216+0000 7f67a63f5740 1 Processor -- start 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f516dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f517300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a63f5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f518310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a63f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1adb10 con 0x559c6f507850 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a63f5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1aee30 con 0x559c6f508240 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a63f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1ae170 con 0x559c6f502dc0 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a63f5740 10 client.0 _ll_register_callbacks cb 0x7f6794003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f517300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f517300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56466/0 (socket says 172.21.15.116:56466) 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 -- 172.21.15.116:0/172571525 learned_addr learned my addr 172.21.15.116:0/172571525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a1800640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f518310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a0e00640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f516dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 -- 172.21.15.116:0/172571525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 msgr2=0x559c6f516dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f516dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 -- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 msgr2=0x559c6f518310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f518310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f679be00640 1 -- 172.21.15.116:0/172571525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c6f518900 con 0x559c6f507850 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.217+0000 7f67a1800640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f518310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f679be00640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f517300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6788013fd0 tx=0x7f6788005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894063d99fb781b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f6782a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f6782a00640 1 -- 172.21.15.116:0/172571525 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788031030 con 0x559c6f507850 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67940a70e0 con 0x559c6f507850 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f6782a00640 1 -- 172.21.15.116:0/172571525 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f678801c030 con 0x559c6f507850 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f6782a00640 1 -- 172.21.15.116:0/172571525 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6788005020 con 0x559c6f507850 2024-03-04T07:43:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f67a63f5740 10 client.15732 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.218+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559c6f5799d0 con 0x559c6f507850 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f6782a00640 1 -- 172.21.15.116:0/172571525 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67940a70e0 con 0x559c6f507850 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f6782a00640 1 -- 172.21.15.116:0/172571525 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6788036450 con 0x559c6f507850 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a63f5740 10 client.15732 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a63f5740 10 client.15732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a63f5740 20 client.15732 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a63f5740 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f50d3b0 0x559c6f57c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a63f5740 4 client.15732 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559c6f474c00 con 0x559c6f50d3b0 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a0e00640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f50d3b0 0x559c6f57c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f67a0e00640 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f50d3b0 0x559c6f57c470 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9915027200557385 server_cookie=bdae2f0ff6719a61 in_seq=0 out_seq=0 2024-03-04T07:43:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.219+0000 7f6782a00640 10 client.15732 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f6782a00640 1 -- 172.21.15.116:0/172571525 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559c6f474c00 con 0x559c6f50d3b0 2024-03-04T07:43:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f6782a00640 10 client.15732 handle_command_reply: tid=0 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 client.15732 shutdown 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 msgr2=0x559c6f517300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f517300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6788013fd0 tx=0x7f6788005f90 comp rx=0 tx=0).stop 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 shutdown_connections 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559c6f50d3b0 0x559c6f57c470 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559c6f508240 0x559c6f518310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c6f507850 0x559c6f517300 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f6782a00640 0 client.15732 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 --2- 172.21.15.116:0/172571525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c6f502dc0 0x559c6f516dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.233+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 >> 172.21.15.116:0/172571525 conn(0x559c6f50ceb0 msgr2=0x559c6f519ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.234+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 shutdown_connections 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.234+0000 7f67a63f5740 1 -- 172.21.15.116:0/172571525 wait complete. 2024-03-04T07:43:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.234+0000 7f67a63f5740 20 client.15732 trim_cache size 0 max 16384 2024-03-04T07:43:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:43:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:43:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:43:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.234+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f67840a1ea0 msgr2=0x7f67840a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.234+0000 7f67a3800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f67840a1ea0 0x7f67840a4360 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x559c6f508aa0 tx=0x7f6788009880 comp rx=0 tx=0).stop 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.234+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 msgr2=0x7f679c1a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.234+0000 7f67a3800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 0x7f679c1a53c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6794037040 tx=0x7f6794032ea0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 shutdown_connections 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f679c077b70 0x7f679c1a53c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f679c0771d0 0x7f679c19e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f67840a1ea0 0x7f67840a4360 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 --2- 172.21.15.116:0/4088752427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f679c075f90 0x7f679c19de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 >> 172.21.15.116:0/4088752427 conn(0x7f679c0fed90 msgr2=0x7f679c100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 shutdown_connections 2024-03-04T07:43:10.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:10.235+0000 7f67a3800640 1 -- 172.21.15.116:0/4088752427 wait complete. 2024-03-04T07:43:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:10 smithi133 ceph-mon[36505]: pgmap v473: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-03-04T07:43:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:10 smithi116 ceph-mon[32150]: pgmap v473: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-03-04T07:43:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:10 smithi152 ceph-mon[37154]: pgmap v473: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-03-04T07:43:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:12 smithi116 ceph-mon[32150]: pgmap v474: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:43:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:12 smithi152 ceph-mon[37154]: pgmap v474: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:43:13.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:12 smithi133 ceph-mon[36505]: pgmap v474: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:43:15.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:43:15.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:43:15.228 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:14 smithi116 ceph-mon[32150]: pgmap v475: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:43:15.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:14 smithi152 ceph-mon[37154]: pgmap v475: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:43:15.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:14 smithi133 ceph-mon[36505]: pgmap v475: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:43:15.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 -- 172.21.15.116:0/3632441211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 msgr2=0x7f58fc103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 --2- 172.21.15.116:0/3632441211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc103c10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009900 tx=0x7f58ec02bc30 comp rx=0 tx=0).stop 2024-03-04T07:43:15.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 -- 172.21.15.116:0/3632441211 shutdown_connections 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 --2- 172.21.15.116:0/3632441211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58fc1053a0 0x7f58fc10bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 --2- 172.21.15.116:0/3632441211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58fc1049e0 0x7f58fc104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 --2- 172.21.15.116:0/3632441211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc103c10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 -- 172.21.15.116:0/3632441211 >> 172.21.15.116:0/3632441211 conn(0x7f58fc0fed80 msgr2=0x7f58fc1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 -- 172.21.15.116:0/3632441211 shutdown_connections 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.363+0000 7f5902000640 1 -- 172.21.15.116:0/3632441211 wait complete. 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.364+0000 7f5902000640 1 Processor -- start 2024-03-04T07:43:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.364+0000 7f5902000640 1 -- start start 2024-03-04T07:43:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f5902000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f5902000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58fc1049e0 0x7f58fc19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f5902000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58fc1053a0 0x7f58fc1a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f5902000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc10f4b0 con 0x7f58fc1049e0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f5902000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc10f330 con 0x7f58fc1037f0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f5902000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc10f630 con 0x7f58fc1053a0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f58fb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f58fb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38954/0 (socket says 172.21.15.116:38954) 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f58fb400640 1 -- 172.21.15.116:0/3832069833 learned_addr learned my addr 172.21.15.116:0/3832069833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f58f3e00640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58fc1049e0 0x7f58fc19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.365+0000 7f58fbe00640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58fc1053a0 0x7f58fc1a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58fb400640 1 -- 172.21.15.116:0/3832069833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58fc1053a0 msgr2=0x7f58fc1a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58fb400640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58fc1053a0 0x7f58fc1a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58fb400640 1 -- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58fc1049e0 msgr2=0x7f58fc19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58fb400640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58fc1049e0 0x7f58fc19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58fb400640 1 -- 172.21.15.116:0/3832069833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc1a1740 con 0x7f58fc1037f0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58fb400640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc199b20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f58ec000c00 tx=0x7f58ec02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58f8c00640 1 -- 172.21.15.116:0/3832069833 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec043020 con 0x7f58fc1037f0 2024-03-04T07:43:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58f8c00640 1 -- 172.21.15.116:0/3832069833 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58ec0054e0 con 0x7f58fc1037f0 2024-03-04T07:43:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f58f8c00640 1 -- 172.21.15.116:0/3832069833 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec003c40 con 0x7f58fc1037f0 2024-03-04T07:43:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc1a19d0 con 0x7f58fc1037f0 2024-03-04T07:43:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.366+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc1a1ed0 con 0x7f58fc1037f0 2024-03-04T07:43:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.369+0000 7f58f8c00640 1 -- 172.21.15.116:0/3832069833 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f58ec046020 con 0x7f58fc1037f0 2024-03-04T07:43:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.369+0000 7f5904dbf740 1 Processor -- start 2024-03-04T07:43:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f5904dbf740 1 -- start start 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b3bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3c540 0x562592b41040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b416b0 0x562592b43b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f5904dbf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562592a69090 con 0x562592b416b0 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f5904dbf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562592aa0d30 con 0x562592b3bb50 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f5904dbf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562592a49a30 con 0x562592b3c540 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b3bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b3bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38962/0 (socket says 172.21.15.116:38962) 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f58fbe00640 1 -- 172.21.15.116:0/579933174 learned_addr learned my addr 172.21.15.116:0/579933174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f58f3e00640 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b416b0 0x562592b43b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.370+0000 7f58fb400640 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3c540 0x562592b41040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58fbe00640 1 -- 172.21.15.116:0/579933174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3c540 msgr2=0x562592b41040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58fbe00640 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3c540 0x562592b41040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58fbe00640 1 -- 172.21.15.116:0/579933174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b416b0 msgr2=0x562592b43b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58fbe00640 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b416b0 0x562592b43b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58fbe00640 1 -- 172.21.15.116:0/579933174 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259287f7d0 con 0x562592b3bb50 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58fbe00640 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b3bf70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009fd0 tx=0x7f58f40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e3a360506788d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58ebe00640 1 -- 172.21.15.116:0/579933174 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4004d20 con 0x562592b3bb50 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58f8c00640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58e00a1d80 0x7f58e00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58ebe00640 1 -- 172.21.15.116:0/579933174 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58f4004050 con 0x562592b3bb50 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.371+0000 7f58ebe00640 1 -- 172.21.15.116:0/579933174 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4005220 con 0x562592b3bb50 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f58f8c00640 1 -- 172.21.15.116:0/3832069833 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f58ec0efda0 con 0x7f58fc1037f0 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f58fb400640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58e00a1d80 0x7f58e00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 -- 172.21.15.116:0/579933174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 msgr2=0x562592b3bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b3bf70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009fd0 tx=0x7f58f40118d0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 -- 172.21.15.116:0/579933174 shutdown_connections 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b416b0 0x562592b43b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3c540 0x562592b41040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 --2- 172.21.15.116:0/579933174 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b3bf70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 -- 172.21.15.116:0/579933174 >> 172.21.15.116:0/579933174 conn(0x562592b370c0 msgr2=0x562592b374c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 -- 172.21.15.116:0/579933174 shutdown_connections 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f58fb400640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58e00a1d80 0x7f58e00a4240 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f58ec0b9460 tx=0x7f58ec0f37e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.372+0000 7f5904dbf740 1 -- 172.21.15.116:0/579933174 wait complete. 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.373+0000 7f5904dbf740 1 -- start start 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.373+0000 7f5904dbf740 1 Processor -- start 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.373+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3bb50 0x562592b45f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3c540 0x562592b46460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f5904dbf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562592a3f0c0 con 0x562592b3c540 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f5904dbf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562592a9cdd0 con 0x562592b416b0 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f5904dbf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562592b44660 con 0x562592b3bb50 2024-03-04T07:43:15.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f5904dbf740 10 client.0 _ll_register_callbacks cb 0x562592a5c860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3c540 0x562592b46460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3bb50 0x562592b45f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3c540 0x562592b46460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56500/0 (socket says 172.21.15.116:56500) 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 -- 172.21.15.116:0/2605721780 learned_addr learned my addr 172.21.15.116:0/2605721780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58fbe00640 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 -- 172.21.15.116:0/2605721780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3bb50 msgr2=0x562592b45f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3bb50 0x562592b45f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 -- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 msgr2=0x562592b4d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58f3e00640 1 -- 172.21.15.116:0/2605721780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625927b53f0 con 0x562592b3c540 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.374+0000 7f58fbe00640 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:15.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f58f3e00640 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3c540 0x562592b46460 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58e40126e0 tx=0x7f58e4012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d918799c49f8fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f58e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f58e8c00640 1 -- 172.21.15.116:0/2605721780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e400f8c0 con 0x562592b3c540 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f58e8c00640 1 -- 172.21.15.116:0/2605721780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f58e400fa60 con 0x562592b3c540 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58f4004050 con 0x562592b3c540 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f58e8c00640 1 -- 172.21.15.116:0/2605721780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58e400fc60 con 0x562592b3c540 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f5904dbf740 10 client.15744 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5625927750f0 con 0x562592b3c540 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.375+0000 7f58e8c00640 1 -- 172.21.15.116:0/2605721780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58f4004050 con 0x562592b3c540 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f58e8c00640 1 -- 172.21.15.116:0/2605721780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f58e401c4b0 con 0x562592b3c540 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f5904dbf740 10 client.15744 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f5904dbf740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f5904dbf740 20 client.15744 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f5904dbf740 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592b51ab0 0x562592b53ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f5904dbf740 4 client.15744 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:15.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562592aae270 con 0x562592b51ab0 2024-03-04T07:43:15.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.376+0000 7f58fbe00640 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592b51ab0 0x562592b53ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.377+0000 7f58fbe00640 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592b51ab0 0x562592b53ed0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=543a3d7fa64f6259 server_cookie=cdfd69c2b8627d81 in_seq=0 out_seq=0 2024-03-04T07:43:15.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.377+0000 7f58e8c00640 10 client.15744 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:15.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.378+0000 7f58e8c00640 1 -- 172.21.15.116:0/2605721780 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562592aae270 con 0x562592b51ab0 2024-03-04T07:43:15.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.378+0000 7f58e8c00640 10 client.15744 handle_command_reply: tid=0 2024-03-04T07:43:15.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.378+0000 7f5904dbf740 1 client.15744 shutdown 2024-03-04T07:43:15.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3c540 msgr2=0x562592b46460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3c540 0x562592b46460 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58e40126e0 tx=0x7f58e4012ba0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 shutdown_connections 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592b51ab0 0x562592b53ed0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3c540 0x562592b46460 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 --2- 172.21.15.116:0/2605721780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3bb50 0x562592b45f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f58e8c00640 0 client.15744 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 >> 172.21.15.116:0/2605721780 conn(0x562592b370c0 msgr2=0x562592b36360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 shutdown_connections 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 1 -- 172.21.15.116:0/2605721780 wait complete. 2024-03-04T07:43:15.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.379+0000 7f5904dbf740 20 client.15744 trim_cache size 0 max 16384 2024-03-04T07:43:15.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 Processor -- start 2024-03-04T07:43:15.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 -- start start 2024-03-04T07:43:15.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3bb50 0x562592b45540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3aa30 0x562592b58790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc1a8ce0 con 0x562592b3bb50 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc1ab2c0 con 0x562592b416b0 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f5904dbf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc1a92e0 con 0x562592b3aa30 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3bb50 0x562592b45540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f58f3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3bb50 0x562592b45540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56514/0 (socket says 172.21.15.116:56514) 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f58fbe00640 1 -- 172.21.15.116:0/1800188799 learned_addr learned my addr 172.21.15.116:0/1800188799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.397+0000 7f58f3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38970/0 (socket says 172.21.15.116:38970) 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58fbe00640 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3aa30 0x562592b58790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58f3e00640 1 -- 172.21.15.116:0/1800188799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3aa30 msgr2=0x562592b58790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58f3e00640 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3aa30 0x562592b58790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58f3e00640 1 -- 172.21.15.116:0/1800188799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3bb50 msgr2=0x562592b45540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58f3e00640 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3bb50 0x562592b45540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58f3e00640 1 -- 172.21.15.116:0/1800188799 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562592b37e30 con 0x562592b416b0 2024-03-04T07:43:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58fbe00640 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3bb50 0x562592b45540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58f3e00640 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d460 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f58e4011000 tx=0x7f58e4005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32277b77b3988dc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58eb400640 1 -- 172.21.15.116:0/1800188799 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e402b030 con 0x562592b416b0 2024-03-04T07:43:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58eb400640 1 -- 172.21.15.116:0/1800188799 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58e401c4a0 con 0x562592b416b0 2024-03-04T07:43:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.398+0000 7f58eb400640 1 -- 172.21.15.116:0/1800188799 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e40105c0 con 0x562592b416b0 2024-03-04T07:43:15.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 -- 172.21.15.116:0/1800188799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 msgr2=0x562592b4d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d460 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f58e4011000 tx=0x7f58e4005750 comp rx=0 tx=0).stop 2024-03-04T07:43:15.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 -- 172.21.15.116:0/1800188799 shutdown_connections 2024-03-04T07:43:15.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b3aa30 0x562592b58790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b416b0 0x562592b4d460 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 --2- 172.21.15.116:0/1800188799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3bb50 0x562592b45540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 -- 172.21.15.116:0/1800188799 >> 172.21.15.116:0/1800188799 conn(0x562592b370c0 msgr2=0x562592b3a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 -- 172.21.15.116:0/1800188799 shutdown_connections 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.399+0000 7f5904dbf740 1 -- 172.21.15.116:0/1800188799 wait complete. 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 -- start start 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 Processor -- start 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3aa30 0x562592b4bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b4c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b416b0 0x562592b4c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc1a9940 con 0x562592b3aa30 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc1aac60 con 0x562592b3bb50 2024-03-04T07:43:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc1a9fa0 con 0x562592b416b0 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.400+0000 7f5904dbf740 10 client.0 _ll_register_callbacks cb 0x7f58e4009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3aa30 0x562592b4bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3aa30 0x562592b4bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56524/0 (socket says 172.21.15.116:56524) 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58fbe00640 1 -- 172.21.15.116:0/458421065 learned_addr learned my addr 172.21.15.116:0/458421065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58f3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b4c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58fbe00640 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b416b0 0x562592b4c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58f3e00640 1 -- 172.21.15.116:0/458421065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b416b0 msgr2=0x562592b4c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58f3e00640 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b416b0 0x562592b4c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58f3e00640 1 -- 172.21.15.116:0/458421065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3aa30 msgr2=0x562592b4bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58f3e00640 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3aa30 0x562592b4bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58f3e00640 1 -- 172.21.15.116:0/458421065 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562592b37b30 con 0x562592b3bb50 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58fbe00640 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3aa30 0x562592b4bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58f3e00640 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b4c420 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f58e400fb10 tx=0x7f58e400f9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebcb65b2bdd98119 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.401+0000 7f58e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f58e8c00640 1 -- 172.21.15.116:0/458421065 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e402b040 con 0x562592b3bb50 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f58e8c00640 1 -- 172.21.15.116:0/458421065 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f58e401c650 con 0x562592b3bb50 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58e401c4a0 con 0x562592b3bb50 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f58e8c00640 1 -- 172.21.15.116:0/458421065 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58e4005bf0 con 0x562592b3bb50 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 10 client.25865 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562592bae6c0 con 0x562592b3bb50 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f58e8c00640 1 -- 172.21.15.116:0/458421065 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58e401c4a0 con 0x562592b3bb50 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f58e8c00640 1 -- 172.21.15.116:0/458421065 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f58e40036f0 con 0x562592b3bb50 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 10 client.25865 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 10 client.25865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 20 client.25865 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592baef60 0x562592bb1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 4 client.25865 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:43:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.402+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562592b45d10 con 0x562592baef60 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f58fbe00640 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592baef60 0x562592bb1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f58fbe00640 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592baef60 0x562592bb1350 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98ac865d6aa317ad server_cookie=6822d9e5ef95618c in_seq=0 out_seq=0 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f58e8c00640 10 client.25865 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f58e8c00640 1 -- 172.21.15.116:0/458421065 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562592b45d10 con 0x562592baef60 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f58e8c00640 10 client.25865 handle_command_reply: tid=0 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f5904dbf740 1 client.25865 shutdown 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 msgr2=0x562592b4c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.403+0000 7f5904dbf740 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b4c420 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f58e400fb10 tx=0x7f58e400f9a0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 shutdown_connections 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562592baef60 0x562592bb1350 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562592b416b0 0x562592b4c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562592b3bb50 0x562592b4c420 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 --2- 172.21.15.116:0/458421065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562592b3aa30 0x562592b4bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f58e8c00640 0 client.25865 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 >> 172.21.15.116:0/458421065 conn(0x562592b370c0 msgr2=0x562592b45510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 shutdown_connections 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 1 -- 172.21.15.116:0/458421065 wait complete. 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5904dbf740 20 client.25865 trim_cache size 0 max 16384 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:43:15.410 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58e00a1d80 msgr2=0x7f58e00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5902000640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58e00a1d80 0x7f58e00a4240 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f58ec0b9460 tx=0x7f58ec0f37e0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 msgr2=0x7f58fc199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5902000640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc199b20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f58ec000c00 tx=0x7f58ec02c650 comp rx=0 tx=0).stop 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 shutdown_connections 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.404+0000 7f5902000640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f58e00a1d80 0x7f58e00a4240 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.405+0000 7f5902000640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f58fc1053a0 0x7f58fc1a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.405+0000 7f5902000640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58fc1049e0 0x7f58fc19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.405+0000 7f5902000640 1 --2- 172.21.15.116:0/3832069833 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f58fc1037f0 0x7f58fc199b20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.405+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 >> 172.21.15.116:0/3832069833 conn(0x7f58fc0fed80 msgr2=0x7f58fc107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.405+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 shutdown_connections 2024-03-04T07:43:15.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.405+0000 7f5902000640 1 -- 172.21.15.116:0/3832069833 wait complete. 2024-03-04T07:43:15.423 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:43:15.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.621+0000 7f531e800640 1 -- 172.21.15.116:0/3872116677 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318108e00 msgr2=0x7f5318109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.621+0000 7f531e800640 1 --2- 172.21.15.116:0/3872116677 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318108e00 0x7f5318109280 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5304009900 tx=0x7f530402bc30 comp rx=0 tx=0).stop 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.621+0000 7f531e800640 1 -- 172.21.15.116:0/3872116677 shutdown_connections 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.621+0000 7f531e800640 1 --2- 172.21.15.116:0/3872116677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53181097c0 0x7f5318110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.621+0000 7f531e800640 1 --2- 172.21.15.116:0/3872116677 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318108e00 0x7f5318109280 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.621+0000 7f531e800640 1 --2- 172.21.15.116:0/3872116677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5318107c10 0x7f5318108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.621+0000 7f531e800640 1 -- 172.21.15.116:0/3872116677 >> 172.21.15.116:0/3872116677 conn(0x7f5318076be0 msgr2=0x7f5318079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.622+0000 7f531e800640 1 -- 172.21.15.116:0/3872116677 shutdown_connections 2024-03-04T07:43:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.622+0000 7f531e800640 1 -- 172.21.15.116:0/3872116677 wait complete. 2024-03-04T07:43:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.622+0000 7f531e800640 1 Processor -- start 2024-03-04T07:43:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f531e800640 1 -- start start 2024-03-04T07:43:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f531e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318107c10 0x7f531819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f531e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5318108e00 0x7f531819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f531e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53181097c0 0x7f53181a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f531e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53181138c0 con 0x7f53181097c0 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f531e800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318113740 con 0x7f5318107c10 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f531e800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318113a40 con 0x7f5318108e00 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f5317400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318107c10 0x7f531819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f5317400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318107c10 0x7f531819dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:39000/0 (socket says 172.21.15.116:39000) 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f5317400640 1 -- 172.21.15.116:0/2221637211 learned_addr learned my addr 172.21.15.116:0/2221637211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f5317e00640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53181097c0 0x7f53181a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.623+0000 7f5316a00640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5318108e00 0x7f531819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f5317400640 1 -- 172.21.15.116:0/2221637211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5318108e00 msgr2=0x7f531819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f5317400640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5318108e00 0x7f531819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f5317400640 1 -- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53181097c0 msgr2=0x7f53181a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f5317400640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53181097c0 0x7f53181a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f5317400640 1 -- 172.21.15.116:0/2221637211 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53181a5ac0 con 0x7f5318107c10 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f5317400640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318107c10 0x7f531819dea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f530800ebc0 tx=0x7f530800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f530fe00640 1 -- 172.21.15.116:0/2221637211 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308003b70 con 0x7f5318107c10 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f530fe00640 1 -- 172.21.15.116:0/2221637211 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5308003d10 con 0x7f5318107c10 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53181a5db0 con 0x7f5318107c10 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f530fe00640 1 -- 172.21.15.116:0/2221637211 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308019980 con 0x7f5318107c10 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.624+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53181a6270 con 0x7f5318107c10 2024-03-04T07:43:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.625+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531810eed0 con 0x7f5318107c10 2024-03-04T07:43:15.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.627+0000 7f530fe00640 1 -- 172.21.15.116:0/2221637211 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5308004b60 con 0x7f5318107c10 2024-03-04T07:43:15.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.629+0000 7f530fe00640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52fc0a1e00 0x7f52fc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:15.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.629+0000 7f530fe00640 1 -- 172.21.15.116:0/2221637211 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f53080ce540 con 0x7f5318107c10 2024-03-04T07:43:15.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.629+0000 7f5316a00640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52fc0a1e00 0x7f52fc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:15.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.630+0000 7f5316a00640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52fc0a1e00 0x7f52fc0a42c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f530402c240 tx=0x7f530402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:15.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.631+0000 7f530fe00640 1 -- 172.21.15.116:0/2221637211 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f530809b970 con 0x7f5318107c10 2024-03-04T07:43:15.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.871+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f531802d2a0 con 0x7f5318107c10 2024-03-04T07:43:15.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.872+0000 7f530fe00640 1 -- 172.21.15.116:0/2221637211 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f530809b270 con 0x7f5318107c10 2024-03-04T07:43:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:43:15.873 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:43:15.874 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52fc0a1e00 msgr2=0x7f52fc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52fc0a1e00 0x7f52fc0a42c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f530402c240 tx=0x7f530402df90 comp rx=0 tx=0).stop 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318107c10 msgr2=0x7f531819dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318107c10 0x7f531819dea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f530800ebc0 tx=0x7f530800c6a0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 shutdown_connections 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53181097c0 0x7f53181a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52fc0a1e00 0x7f52fc0a42c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5318108e00 0x7f531819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.875+0000 7f531e800640 1 --2- 172.21.15.116:0/2221637211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5318107c10 0x7f531819dea0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.876+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 >> 172.21.15.116:0/2221637211 conn(0x7f5318076be0 msgr2=0x7f531810c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.876+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 shutdown_connections 2024-03-04T07:43:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:15.876+0000 7f531e800640 1 -- 172.21.15.116:0/2221637211 wait complete. 2024-03-04T07:43:15.895 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:43:15.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:43:16.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 -- 172.21.15.116:0/3462794603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10181097a0 msgr2=0x7f1018110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 --2- 172.21.15.116:0/3462794603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10181097a0 0x7f1018110030 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1010009900 tx=0x7f101002bc30 comp rx=0 tx=0).stop 2024-03-04T07:43:16.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 -- 172.21.15.116:0/3462794603 shutdown_connections 2024-03-04T07:43:16.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 --2- 172.21.15.116:0/3462794603 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f10181097a0 0x7f1018110030 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 --2- 172.21.15.116:0/3462794603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1018108de0 0x7f1018109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 --2- 172.21.15.116:0/3462794603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1018107bf0 0x7f1018108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 -- 172.21.15.116:0/3462794603 >> 172.21.15.116:0/3462794603 conn(0x7f1018076b90 msgr2=0x7f1018078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:16.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 -- 172.21.15.116:0/3462794603 shutdown_connections 2024-03-04T07:43:16.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.106+0000 7f101d600640 1 -- 172.21.15.116:0/3462794603 wait complete. 2024-03-04T07:43:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.107+0000 7f101d600640 1 Processor -- start 2024-03-04T07:43:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.107+0000 7f101d600640 1 -- start start 2024-03-04T07:43:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f101d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1018107bf0 0x7f101819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f101d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1018108de0 0x7f101819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f101d600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10181097a0 0x7f10181a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f101d600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181138a0 con 0x7f1018107bf0 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f101d600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1018113720 con 0x7f1018108de0 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f101d600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1018113a20 con 0x7f10181097a0 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1018108de0 0x7f101819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1018108de0 0x7f101819e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45188/0 (socket says 172.21.15.116:45188) 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f1016a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1018107bf0 0x7f101819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 -- 172.21.15.116:0/1085852899 learned_addr learned my addr 172.21.15.116:0/1085852899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f1017400640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10181097a0 0x7f10181a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 -- 172.21.15.116:0/1085852899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10181097a0 msgr2=0x7f10181a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10181097a0 0x7f10181a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 -- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1018107bf0 msgr2=0x7f101819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1018107bf0 0x7f101819de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f100e000640 1 -- 172.21.15.116:0/1085852899 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10181a5aa0 con 0x7f1018108de0 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.108+0000 7f1017400640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10181097a0 0x7f10181a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.109+0000 7f100e000640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1018108de0 0x7f101819e3c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f100000ebc0 tx=0x7f100000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.109+0000 7f100fe00640 1 -- 172.21.15.116:0/1085852899 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000010630 con 0x7f1018108de0 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.109+0000 7f100fe00640 1 -- 172.21.15.116:0/1085852899 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1000010cb0 con 0x7f1018108de0 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.109+0000 7f100fe00640 1 -- 172.21.15.116:0/1085852899 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000018bd0 con 0x7f1018108de0 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.109+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10181a5d90 con 0x7f1018108de0 2024-03-04T07:43:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.109+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10181a6260 con 0x7f1018108de0 2024-03-04T07:43:16.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.112+0000 7f100fe00640 1 -- 172.21.15.116:0/1085852899 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f10000107d0 con 0x7f1018108de0 2024-03-04T07:43:16.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.112+0000 7f102037d740 1 Processor -- start 2024-03-04T07:43:16.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.112+0000 7f102037d740 1 -- start start 2024-03-04T07:43:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f102037d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4280c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f102037d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e42d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f102037d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e42fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f102037d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56108e38cd60 con 0x56108e427ca0 2024-03-04T07:43:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f102037d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56108e335cf0 con 0x56108e428690 2024-03-04T07:43:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f102037d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56108e388e00 con 0x56108e42d800 2024-03-04T07:43:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1017400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4280c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1017400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4280c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35332/0 (socket says 172.21.15.116:35332) 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1016a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e42d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1017400640 1 -- 172.21.15.116:0/4090093157 learned_addr learned my addr 172.21.15.116:0/4090093157 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f100e000640 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e42fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1016a00640 1 -- 172.21.15.116:0/4090093157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 msgr2=0x56108e42fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1016a00640 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e42fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1016a00640 1 -- 172.21.15.116:0/4090093157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 msgr2=0x56108e4280c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1016a00640 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4280c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f1016a00640 1 -- 172.21.15.116:0/4090093157 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56108e0a13f0 con 0x56108e428690 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.113+0000 7f100fe00640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0ffc0a1e00 0x7f0ffc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f1016a00640 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e42d190 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1008009bf0 tx=0x7f1008011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0ed850a0fc04bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f100fe00640 1 -- 172.21.15.116:0/1085852899 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f10000ce420 con 0x7f1018108de0 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f1017400640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0ffc0a1e00 0x7f0ffc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f1007400640 1 -- 172.21.15.116:0/4090093157 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008009da0 con 0x56108e428690 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f1007400640 1 -- 172.21.15.116:0/4090093157 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1008005020 con 0x56108e428690 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f1007400640 1 -- 172.21.15.116:0/4090093157 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008004bc0 con 0x56108e428690 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f102037d740 1 -- 172.21.15.116:0/4090093157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 msgr2=0x56108e42d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.114+0000 7f102037d740 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e42d190 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1008009bf0 tx=0x7f1008011810 comp rx=0 tx=0).stop 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f1017400640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0ffc0a1e00 0x7f0ffc0a42c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1010005c40 tx=0x7f101002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 -- 172.21.15.116:0/4090093157 shutdown_connections 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e42fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e42d190 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 --2- 172.21.15.116:0/4090093157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4280c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 -- 172.21.15.116:0/4090093157 >> 172.21.15.116:0/4090093157 conn(0x56108e423210 msgr2=0x56108e423610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 -- 172.21.15.116:0/4090093157 shutdown_connections 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 -- 172.21.15.116:0/4090093157 wait complete. 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 -- start start 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.115+0000 7f102037d740 1 Processor -- start 2024-03-04T07:43:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f102037d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4321f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f102037d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e432730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f102037d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e439770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f102037d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56108e4307b0 con 0x56108e427ca0 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f102037d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56108e32b220 con 0x56108e428690 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f102037d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56108e430930 con 0x56108e42d800 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f102037d740 10 client.0 _ll_register_callbacks cb 0x56108e0a0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4321f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35348/0 (socket says 172.21.15.116:35348) 2024-03-04T07:43:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 -- 172.21.15.116:0/3083506608 learned_addr learned my addr 172.21.15.116:0/3083506608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f100e000640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e432730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1017400640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e439770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 -- 172.21.15.116:0/3083506608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 msgr2=0x56108e439770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e439770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 -- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 msgr2=0x56108e432730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e432730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f1016a00640 1 -- 172.21.15.116:0/3083506608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56108e16b7d0 con 0x56108e427ca0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.116+0000 7f100e000640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e432730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f1016a00640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4321f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f1008009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7c27ca1841ea06 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f0ffbe00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f0ffbe00640 1 -- 172.21.15.116:0/3083506608 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008029020 con 0x56108e427ca0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f0ffbe00640 1 -- 172.21.15.116:0/3083506608 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1008004b50 con 0x56108e427ca0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1008005020 con 0x56108e427ca0 2024-03-04T07:43:16.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f0ffbe00640 1 -- 172.21.15.116:0/3083506608 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1008005c50 con 0x56108e427ca0 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f102037d740 10 client.15768 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.117+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56108e43cae0 con 0x56108e427ca0 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f0ffbe00640 1 -- 172.21.15.116:0/3083506608 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1008005020 con 0x56108e427ca0 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f0ffbe00640 1 -- 172.21.15.116:0/3083506608 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1008019450 con 0x56108e427ca0 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f102037d740 10 client.15768 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f102037d740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f102037d740 20 client.15768 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f102037d740 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e43df30 0x56108e440360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f102037d740 4 client.15768 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56108e39a2a0 con 0x56108e43df30 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.118+0000 7f100e000640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e43df30 0x56108e440360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.119+0000 7f100e000640 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e43df30 0x56108e440360 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d00bd6dc3287863 server_cookie=f3d5fdeefcd08056 in_seq=0 out_seq=0 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.119+0000 7f0ffbe00640 10 client.15768 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.119+0000 7f0ffbe00640 1 -- 172.21.15.116:0/3083506608 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56108e39a2a0 con 0x56108e43df30 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.119+0000 7f0ffbe00640 10 client.15768 handle_command_reply: tid=0 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.119+0000 7f102037d740 1 client.15768 shutdown 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.119+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 msgr2=0x56108e4321f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.119+0000 7f102037d740 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4321f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f1008009850 comp rx=0 tx=0).stop 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 shutdown_connections 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e43df30 0x56108e440360 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e42d800 0x56108e439770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e428690 0x56108e432730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f0ffbe00640 0 client.15768 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 --2- 172.21.15.116:0/3083506608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e4321f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 >> 172.21.15.116:0/3083506608 conn(0x56108e423210 msgr2=0x56108e4224b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 shutdown_connections 2024-03-04T07:43:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 1 -- 172.21.15.116:0/3083506608 wait complete. 2024-03-04T07:43:16.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.120+0000 7f102037d740 20 client.15768 trim_cache size 0 max 16384 2024-03-04T07:43:16.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.133+0000 7f102037d740 1 Processor -- start 2024-03-04T07:43:16.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.133+0000 7f102037d740 1 -- start start 2024-03-04T07:43:16.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f102037d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e427ca0 0x56108e444c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f102037d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e439730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f102037d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e42d800 0x56108e445160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f102037d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181ad0e0 con 0x56108e42d800 2024-03-04T07:43:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f102037d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181af6c0 con 0x56108e427ca0 2024-03-04T07:43:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f102037d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181ad6e0 con 0x56108e428690 2024-03-04T07:43:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f1016a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e427ca0 0x56108e444c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f1017400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e42d800 0x56108e445160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f1016a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e427ca0 0x56108e444c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45232/0 (socket says 172.21.15.116:45232) 2024-03-04T07:43:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f1016a00640 1 -- 172.21.15.116:0/12740451 learned_addr learned my addr 172.21.15.116:0/12740451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f100e000640 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e439730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.134+0000 7f1016a00640 1 -- 172.21.15.116:0/12740451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 msgr2=0x56108e439730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1016a00640 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e439730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1016a00640 1 -- 172.21.15.116:0/12740451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e42d800 msgr2=0x56108e445160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1016a00640 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e42d800 0x56108e445160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1016a00640 1 -- 172.21.15.116:0/12740451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56108e430a80 con 0x56108e427ca0 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1016a00640 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e427ca0 0x56108e444c20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1008009820 tx=0x7f1008009af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=416398ddde04d232 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1006a00640 1 -- 172.21.15.116:0/12740451 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008019e60 con 0x56108e427ca0 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1006a00640 1 -- 172.21.15.116:0/12740451 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1008019450 con 0x56108e427ca0 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.135+0000 7f1006a00640 1 -- 172.21.15.116:0/12740451 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008029070 con 0x56108e427ca0 2024-03-04T07:43:16.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 -- 172.21.15.116:0/12740451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e427ca0 msgr2=0x56108e444c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e427ca0 0x56108e444c20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1008009820 tx=0x7f1008009af0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 -- 172.21.15.116:0/12740451 shutdown_connections 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e42d800 0x56108e445160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e439730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 --2- 172.21.15.116:0/12740451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e427ca0 0x56108e444c20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 -- 172.21.15.116:0/12740451 >> 172.21.15.116:0/12740451 conn(0x56108e423210 msgr2=0x56108e426d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 -- 172.21.15.116:0/12740451 shutdown_connections 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 -- 172.21.15.116:0/12740451 wait complete. 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 -- start start 2024-03-04T07:43:16.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.136+0000 7f102037d740 1 Processor -- start 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f102037d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e437080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f102037d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e4375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f102037d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e42d800 0x56108e4385d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f102037d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181add40 con 0x56108e427ca0 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f102037d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181af060 con 0x56108e42d800 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f102037d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181ae3a0 con 0x56108e428690 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f102037d740 10 client.0 _ll_register_callbacks cb 0x7f1008005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:16.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f1016a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e437080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f1016a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e437080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35362/0 (socket says 172.21.15.116:35362) 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f1016a00640 1 -- 172.21.15.116:0/2252996190 learned_addr learned my addr 172.21.15.116:0/2252996190 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f1017400640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e42d800 0x56108e4385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.137+0000 7f100e000640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e4375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f1016a00640 1 -- 172.21.15.116:0/2252996190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 msgr2=0x56108e4375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f1016a00640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e4375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f1016a00640 1 -- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e42d800 msgr2=0x56108e4385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f1016a00640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e42d800 0x56108e4385d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f1016a00640 1 -- 172.21.15.116:0/2252996190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56108e438bc0 con 0x56108e427ca0 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f1017400640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e42d800 0x56108e4385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:16.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f1016a00640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e437080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1008019840 tx=0x7f10080054e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8c0b74605bc156 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f0ffbe00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f0ffbe00640 1 -- 172.21.15.116:0/2252996190 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100803d020 con 0x56108e427ca0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f0ffbe00640 1 -- 172.21.15.116:0/2252996190 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1008019a30 con 0x56108e427ca0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1008019450 con 0x56108e427ca0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f0ffbe00640 1 -- 172.21.15.116:0/2252996190 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1008029040 con 0x56108e427ca0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f102037d740 10 client.15774 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.138+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56108e499a40 con 0x56108e427ca0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f0ffbe00640 1 -- 172.21.15.116:0/2252996190 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1008019450 con 0x56108e427ca0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f0ffbe00640 1 -- 172.21.15.116:0/2252996190 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1008025470 con 0x56108e427ca0 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f102037d740 10 client.15774 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f102037d740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f102037d740 20 client.15774 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:16.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f102037d740 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e49a150 0x56108e49c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f102037d740 4 client.15774 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56108e3945f0 con 0x56108e49a150 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.139+0000 7f100e000640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e49a150 0x56108e49c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f0ffbe00640 10 client.15774 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f100e000640 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e49a150 0x56108e49c570 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e662df2eb0cd0d65 server_cookie=dae9a20bd5786c82 in_seq=0 out_seq=0 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f0ffbe00640 1 -- 172.21.15.116:0/2252996190 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56108e3945f0 con 0x56108e49a150 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f0ffbe00640 10 client.15774 handle_command_reply: tid=0 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 client.15774 shutdown 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 msgr2=0x56108e437080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e437080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1008019840 tx=0x7f10080054e0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 shutdown_connections 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56108e49a150 0x56108e49c570 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56108e42d800 0x56108e4385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56108e428690 0x56108e4375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f0ffbe00640 0 client.15774 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:16.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 --2- 172.21.15.116:0/2252996190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56108e427ca0 0x56108e437080 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 >> 172.21.15.116:0/2252996190 conn(0x56108e423210 msgr2=0x56108e421c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:16.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 shutdown_connections 2024-03-04T07:43:16.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 1 -- 172.21.15.116:0/2252996190 wait complete. 2024-03-04T07:43:16.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.140+0000 7f102037d740 20 client.15774 trim_cache size 0 max 16384 2024-03-04T07:43:16.147 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:43:16.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0ffc0a1e00 msgr2=0x7f0ffc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0ffc0a1e00 0x7f0ffc0a42c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1010005c40 tx=0x7f101002df90 comp rx=0 tx=0).stop 2024-03-04T07:43:16.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1018108de0 msgr2=0x7f101819e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1018108de0 0x7f101819e3c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f100000ebc0 tx=0x7f100000c6a0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 shutdown_connections 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0ffc0a1e00 0x7f0ffc0a42c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10181097a0 0x7f10181a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1018108de0 0x7f101819e3c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 --2- 172.21.15.116:0/1085852899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1018107bf0 0x7f101819de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 >> 172.21.15.116:0/1085852899 conn(0x7f1018076b90 msgr2=0x7f101810c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 shutdown_connections 2024-03-04T07:43:16.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:16.141+0000 7f101d600640 1 -- 172.21.15.116:0/1085852899 wait complete. 2024-03-04T07:43:17.159 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:43:17.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:43:17.160 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:43:17.247 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:16 smithi116 ceph-mon[32150]: pgmap v476: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:17.247 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:16 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2221637211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:43:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:16 smithi152 ceph-mon[37154]: pgmap v476: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:16 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2221637211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:43:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:16 smithi133 ceph-mon[36505]: pgmap v476: 97 pgs: 97 active+clean; 1.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:16 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2221637211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.384+0000 7fa287e00640 1 -- 172.21.15.116:0/1979465763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 msgr2=0x7fa280077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.384+0000 7fa287e00640 1 --2- 172.21.15.116:0/1979465763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 0x7fa280077630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009900 tx=0x7fa26c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.385+0000 7fa287e00640 1 -- 172.21.15.116:0/1979465763 shutdown_connections 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.385+0000 7fa287e00640 1 --2- 172.21.15.116:0/1979465763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa280077b70 0x7fa280078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.385+0000 7fa287e00640 1 --2- 172.21.15.116:0/1979465763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 0x7fa280077630 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.385+0000 7fa287e00640 1 --2- 172.21.15.116:0/1979465763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 0x7fa2800763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.385+0000 7fa287e00640 1 -- 172.21.15.116:0/1979465763 >> 172.21.15.116:0/1979465763 conn(0x7fa2800fed80 msgr2=0x7fa2801011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.385+0000 7fa287e00640 1 -- 172.21.15.116:0/1979465763 shutdown_connections 2024-03-04T07:43:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.385+0000 7fa287e00640 1 -- 172.21.15.116:0/1979465763 wait complete. 2024-03-04T07:43:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 Processor -- start 2024-03-04T07:43:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 -- start start 2024-03-04T07:43:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 0x7fa28019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 0x7fa28019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa280077b70 0x7fa2801a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801138e0 con 0x7fa2800771d0 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa280113760 con 0x7fa280077b70 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa287e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa280113a60 con 0x7fa280075f90 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa285400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 0x7fa28019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa284a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 0x7fa28019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.386+0000 7fa285400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 0x7fa28019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:52762/0 (socket says 172.21.15.116:52762) 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285400640 1 -- 172.21.15.116:0/807692292 learned_addr learned my addr 172.21.15.116:0/807692292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa280077b70 0x7fa2801a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285400640 1 -- 172.21.15.116:0/807692292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa280077b70 msgr2=0x7fa2801a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285400640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa280077b70 0x7fa2801a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285400640 1 -- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 msgr2=0x7fa28019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285400640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 0x7fa28019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285400640 1 -- 172.21.15.116:0/807692292 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2801a5b80 con 0x7fa280075f90 2024-03-04T07:43:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa280077b70 0x7fa2801a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa285400640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 0x7fa28019dfb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa27800eb90 tx=0x7fa27800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa27e000640 1 -- 172.21.15.116:0/807692292 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278004d20 con 0x7fa280075f90 2024-03-04T07:43:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa27e000640 1 -- 172.21.15.116:0/807692292 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa27800ee90 con 0x7fa280075f90 2024-03-04T07:43:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.387+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2801a5e70 con 0x7fa280075f90 2024-03-04T07:43:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.388+0000 7fa27e000640 1 -- 172.21.15.116:0/807692292 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278010830 con 0x7fa280075f90 2024-03-04T07:43:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.388+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2801a6280 con 0x7fa280075f90 2024-03-04T07:43:17.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa27e000640 1 -- 172.21.15.116:0/807692292 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa278005020 con 0x7fa280075f90 2024-03-04T07:43:17.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 Processor -- start 2024-03-04T07:43:17.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 -- start start 2024-03-04T07:43:17.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e9c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605c5dafe20 con 0x5605c5e9cac0 2024-03-04T07:43:17.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605c5e01a20 con 0x5605c5e9c0d0 2024-03-04T07:43:17.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.391+0000 7fa28ac00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605c5daab00 con 0x5605c5ea1c30 2024-03-04T07:43:17.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e9c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e9c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45274/0 (socket says 172.21.15.116:45274) 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 -- 172.21.15.116:0/1077634014 learned_addr learned my addr 172.21.15.116:0/1077634014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285400640 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa284a00640 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 -- 172.21.15.116:0/1077634014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 msgr2=0x5605c5ea40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 -- 172.21.15.116:0/1077634014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 msgr2=0x5605c5ea15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 -- 172.21.15.116:0/1077634014 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605c5dbfe10 con 0x5605c5e9c0d0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa285e00640 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e9c4f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa270011540 tx=0x7fa270011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebbcc14d3b7e0b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa276600640 1 -- 172.21.15.116:0/1077634014 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27000f5c0 con 0x5605c5e9c0d0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa276600640 1 -- 172.21.15.116:0/1077634014 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa27000fc40 con 0x5605c5e9c0d0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.392+0000 7fa276600640 1 -- 172.21.15.116:0/1077634014 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270019d20 con 0x5605c5e9c0d0 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa27e000640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2640a1d80 0x7fa2640a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 -- 172.21.15.116:0/1077634014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 msgr2=0x5605c5e9c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e9c4f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa270011540 tx=0x7fa270011a00 comp rx=0 tx=0).stop 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 -- 172.21.15.116:0/1077634014 shutdown_connections 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1077634014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e9c4f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 -- 172.21.15.116:0/1077634014 >> 172.21.15.116:0/1077634014 conn(0x5605c5e97640 msgr2=0x5605c5e97a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa27e000640 1 -- 172.21.15.116:0/807692292 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa2780ce3f0 con 0x7fa280075f90 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa285400640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2640a1d80 0x7fa2640a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.393+0000 7fa28ac00740 1 -- 172.21.15.116:0/1077634014 shutdown_connections 2024-03-04T07:43:17.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.394+0000 7fa28ac00740 1 -- 172.21.15.116:0/1077634014 wait complete. 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.394+0000 7fa285400640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2640a1d80 0x7fa2640a4240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa278097580 tx=0x7fa278005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.394+0000 7fa28ac00740 1 -- start start 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.394+0000 7fa28ac00740 1 Processor -- start 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9c0d0 0x5605c5ea6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5ead9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa28ac00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605c5dbdf40 con 0x5605c5e9cac0 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa28ac00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605c5dfdac0 con 0x5605c5ea1c30 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa28ac00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605c5ea4be0 con 0x5605c5e9c0d0 2024-03-04T07:43:17.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9c0d0 0x5605c5ea6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9c0d0 0x5605c5ea6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:52770/0 (socket says 172.21.15.116:52770) 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 -- 172.21.15.116:0/3915217066 learned_addr learned my addr 172.21.15.116:0/3915217066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa284a00640 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5ead9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa28ac00740 10 client.0 _ll_register_callbacks cb 0x5605c5dfcaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 -- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 msgr2=0x5605c5ead9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5ead9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 -- 172.21.15.116:0/3915217066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 msgr2=0x5605c5ea69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 -- 172.21.15.116:0/3915217066 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605c5bde7e0 con 0x5605c5e9c0d0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa285e00640 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9c0d0 0x5605c5ea6460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa270019710 tx=0x7fa270009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7256ef5a899e3d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.395+0000 7fa26aa00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa26aa00640 1 -- 172.21.15.116:0/3915217066 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27002d060 con 0x5605c5e9c0d0 2024-03-04T07:43:17.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa26aa00640 1 -- 172.21.15.116:0/3915217066 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa27001f040 con 0x5605c5e9c0d0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa27000fc40 con 0x5605c5e9c0d0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa26aa00640 1 -- 172.21.15.116:0/3915217066 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa270019e30 con 0x5605c5e9c0d0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa28ac00740 10 client.25858 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5605c5c3ef70 con 0x5605c5e9c0d0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa26aa00640 1 -- 172.21.15.116:0/3915217066 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa27000fc40 con 0x5605c5e9c0d0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.396+0000 7fa26aa00640 1 -- 172.21.15.116:0/3915217066 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa270032070 con 0x5605c5e9c0d0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa28ac00740 10 client.25858 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa28ac00740 10 client.25858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa28ac00740 20 client.25858 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa28ac00740 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5eb1ff0 0x5605c5eb4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa28ac00740 4 client.25858 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605c5e0ef60 con 0x5605c5eb1ff0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa284a00640 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5eb1ff0 0x5605c5eb4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa284a00640 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5eb1ff0 0x5605c5eb4400 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=886f328f570621d3 server_cookie=e87e2792c2bb2876 in_seq=0 out_seq=0 2024-03-04T07:43:17.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.397+0000 7fa26aa00640 10 client.25858 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.398+0000 7fa26aa00640 1 -- 172.21.15.116:0/3915217066 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5605c5e0ef60 con 0x5605c5eb1ff0 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.398+0000 7fa26aa00640 10 client.25858 handle_command_reply: tid=0 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 client.25858 shutdown 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9c0d0 msgr2=0x5605c5ea6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9c0d0 0x5605c5ea6460 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa270019710 tx=0x7fa270009310 comp rx=0 tx=0).stop 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 shutdown_connections 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5eb1ff0 0x5605c5eb4400 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5ead9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ea69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa26aa00640 0 client.25858 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 --2- 172.21.15.116:0/3915217066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9c0d0 0x5605c5ea6460 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 >> 172.21.15.116:0/3915217066 conn(0x5605c5e97640 msgr2=0x5605c5e98890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 shutdown_connections 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 1 -- 172.21.15.116:0/3915217066 wait complete. 2024-03-04T07:43:17.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.399+0000 7fa28ac00740 20 client.25858 trim_cache size 0 max 16384 2024-03-04T07:43:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.416+0000 7fa28ac00740 1 Processor -- start 2024-03-04T07:43:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.416+0000 7fa28ac00740 1 -- start start 2024-03-04T07:43:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e968e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ead970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa28ac00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801ad070 con 0x5605c5e9cac0 2024-03-04T07:43:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa28ac00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801af650 con 0x5605c5e9c0d0 2024-03-04T07:43:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa28ac00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801ad670 con 0x5605c5ea1c30 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa285e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa285e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45286/0 (socket says 172.21.15.116:45286) 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa285e00640 1 -- 172.21.15.116:0/1564638904 learned_addr learned my addr 172.21.15.116:0/1564638904 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa285e00640 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.417+0000 7fa284a00640 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ead970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa285e00640 1 -- 172.21.15.116:0/1564638904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 msgr2=0x5605c5ea5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa285e00640 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa285e00640 1 -- 172.21.15.116:0/1564638904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 msgr2=0x5605c5ead970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa285e00640 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ead970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa285e00640 1 -- 172.21.15.116:0/1564638904 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605c5ad6170 con 0x5605c5e9c0d0 2024-03-04T07:43:17.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa285e00640 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e968e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa2700059f0 tx=0x7fa270009e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3a4144183d5a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa275c00640 1 -- 172.21.15.116:0/1564638904 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270038040 con 0x5605c5e9c0d0 2024-03-04T07:43:17.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa275c00640 1 -- 172.21.15.116:0/1564638904 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa27000fcb0 con 0x5605c5e9c0d0 2024-03-04T07:43:17.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.418+0000 7fa275c00640 1 -- 172.21.15.116:0/1564638904 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27002d040 con 0x5605c5e9c0d0 2024-03-04T07:43:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 -- 172.21.15.116:0/1564638904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 msgr2=0x5605c5e968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e968e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa2700059f0 tx=0x7fa270009e20 comp rx=0 tx=0).stop 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 -- 172.21.15.116:0/1564638904 shutdown_connections 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5ea1c30 0x5605c5ea5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9cac0 0x5605c5ead970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1564638904 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5e9c0d0 0x5605c5e968e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 -- 172.21.15.116:0/1564638904 >> 172.21.15.116:0/1564638904 conn(0x5605c5e97640 msgr2=0x5605c5e961d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 -- 172.21.15.116:0/1564638904 shutdown_connections 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 -- 172.21.15.116:0/1564638904 wait complete. 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.419+0000 7fa28ac00740 1 -- start start 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 1 Processor -- start 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9c0d0 0x5605c5ead250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9cac0 0x5605c5ead790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5eab860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801adcd0 con 0x5605c5e9c0d0 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801aeff0 con 0x5605c5ea1c30 2024-03-04T07:43:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801ae330 con 0x5605c5e9cac0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa28ac00740 10 client.0 _ll_register_callbacks cb 0x7fa270011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa285e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9c0d0 0x5605c5ead250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa285e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9c0d0 0x5605c5ead250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35424/0 (socket says 172.21.15.116:35424) 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa285e00640 1 -- 172.21.15.116:0/1807588166 learned_addr learned my addr 172.21.15.116:0/1807588166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa284a00640 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9cac0 0x5605c5ead790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.420+0000 7fa285e00640 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5eab860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa284a00640 1 -- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 msgr2=0x5605c5eab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa284a00640 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5eab860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa284a00640 1 -- 172.21.15.116:0/1807588166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9c0d0 msgr2=0x5605c5ead250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa284a00640 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9c0d0 0x5605c5ead250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa284a00640 1 -- 172.21.15.116:0/1807588166 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605c5eabfa0 con 0x5605c5e9cac0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa284a00640 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9cac0 0x5605c5ead790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa26c036040 tx=0x7fa26c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b3ee4e88f538f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa26aa00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:43:17.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa26aa00640 1 -- 172.21.15.116:0/1807588166 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c052070 con 0x5605c5e9cac0 2024-03-04T07:43:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa26aa00640 1 -- 172.21.15.116:0/1807588166 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa26c004240 con 0x5605c5e9cac0 2024-03-04T07:43:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.421+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa27000fcb0 con 0x5605c5e9cac0 2024-03-04T07:43:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa26aa00640 1 -- 172.21.15.116:0/1807588166 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26c05e050 con 0x5605c5e9cac0 2024-03-04T07:43:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 10 client.25864 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5605c5f0e3c0 con 0x5605c5e9cac0 2024-03-04T07:43:17.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa26aa00640 1 -- 172.21.15.116:0/1807588166 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa27000fcb0 con 0x5605c5e9cac0 2024-03-04T07:43:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa26aa00640 1 -- 172.21.15.116:0/1807588166 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa26c048050 con 0x5605c5e9cac0 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 10 client.25864 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 10 client.25864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 20 client.25864 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5f0ed00 0x5605c5f11120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 4 client.25864 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.422+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5605c5e092b0 con 0x5605c5f0ed00 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.423+0000 7fa285e00640 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5f0ed00 0x5605c5f11120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.423+0000 7fa285e00640 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5f0ed00 0x5605c5f11120 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b3bd4afbd224aef server_cookie=d2ddebd503d7935b in_seq=0 out_seq=0 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.423+0000 7fa26aa00640 10 client.25864 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa26aa00640 1 -- 172.21.15.116:0/1807588166 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5605c5e092b0 con 0x5605c5f0ed00 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa26aa00640 10 client.25864 handle_command_reply: tid=0 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 client.25864 shutdown 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9cac0 msgr2=0x5605c5ead790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9cac0 0x5605c5ead790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa26c036040 tx=0x7fa26c000bc0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 shutdown_connections 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5605c5f0ed00 0x5605c5f11120 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5605c5ea1c30 0x5605c5eab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605c5e9cac0 0x5605c5ead790 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 --2- 172.21.15.116:0/1807588166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605c5e9c0d0 0x5605c5ead250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa26aa00640 0 client.25864 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.424+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 >> 172.21.15.116:0/1807588166 conn(0x5605c5e97640 msgr2=0x5605c5e96a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.425+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 shutdown_connections 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.425+0000 7fa28ac00740 1 -- 172.21.15.116:0/1807588166 wait complete. 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.425+0000 7fa28ac00740 20 client.25864 trim_cache size 0 max 16384 2024-03-04T07:43:17.428 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:43:17.429 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:43:17.429 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "2ca33209-5ceb-4c25-afd4-5d61e7292593", 2024-03-04T07:43:17.429 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:43:17.429 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.425+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2640a1d80 msgr2=0x7fa2640a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.425+0000 7fa287e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2640a1d80 0x7fa2640a4240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa278097580 tx=0x7fa278005960 comp rx=0 tx=0).stop 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.425+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 msgr2=0x7fa28019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.425+0000 7fa287e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 0x7fa28019dfb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa27800eb90 tx=0x7fa27800c6a0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 shutdown_connections 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa280077b70 0x7fa2801a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2800771d0 0x7fa28019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa2640a1d80 0x7fa2640a4240 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 --2- 172.21.15.116:0/807692292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa280075f90 0x7fa28019dfb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 >> 172.21.15.116:0/807692292 conn(0x7fa2800fed80 msgr2=0x7fa280100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 shutdown_connections 2024-03-04T07:43:17.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.426+0000 7fa287e00640 1 -- 172.21.15.116:0/807692292 wait complete. 2024-03-04T07:43:17.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:43:17.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:43:17.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 -- 172.21.15.116:0/3182557761 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 msgr2=0x7f291c103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 --2- 172.21.15.116:0/3182557761 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c103c30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f290c009900 tx=0x7f290c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:43:17.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 -- 172.21.15.116:0/3182557761 shutdown_connections 2024-03-04T07:43:17.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 --2- 172.21.15.116:0/3182557761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f291c1053c0 0x7f291c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 --2- 172.21.15.116:0/3182557761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f291c104a00 0x7f291c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 --2- 172.21.15.116:0/3182557761 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c103c30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 -- 172.21.15.116:0/3182557761 >> 172.21.15.116:0/3182557761 conn(0x7f291c0fed80 msgr2=0x7f291c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 -- 172.21.15.116:0/3182557761 shutdown_connections 2024-03-04T07:43:17.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.660+0000 7f2921600640 1 -- 172.21.15.116:0/3182557761 wait complete. 2024-03-04T07:43:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.661+0000 7f2921600640 1 Processor -- start 2024-03-04T07:43:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.661+0000 7f2921600640 1 -- start start 2024-03-04T07:43:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f2921600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f2921600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f291c104a00 0x7f291c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f2921600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f291c1053c0 0x7f291c1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f2921600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c10f460 con 0x7f291c104a00 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f2921600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c10f2e0 con 0x7f291c103810 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f2921600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c10f5e0 con 0x7f291c1053c0 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f291c104a00 0x7f291c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45316/0 (socket says 172.21.15.116:45316) 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 -- 172.21.15.116:0/151674026 learned_addr learned my addr 172.21.15.116:0/151674026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291b400640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f291c1053c0 0x7f291c1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 -- 172.21.15.116:0/151674026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f291c1053c0 msgr2=0x7f291c1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f291c1053c0 0x7f291c1a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 -- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f291c104a00 msgr2=0x7f291c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f291c104a00 0x7f291c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.662+0000 7f291aa00640 1 -- 172.21.15.116:0/151674026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c1a1780 con 0x7f291c103810 2024-03-04T07:43:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.663+0000 7f291aa00640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c199b60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f290c000c00 tx=0x7f290c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.663+0000 7f2913400640 1 -- 172.21.15.116:0/151674026 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c041070 con 0x7f291c103810 2024-03-04T07:43:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.663+0000 7f2913400640 1 -- 172.21.15.116:0/151674026 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f290c0044d0 con 0x7f291c103810 2024-03-04T07:43:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.663+0000 7f2913400640 1 -- 172.21.15.116:0/151674026 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c034440 con 0x7f291c103810 2024-03-04T07:43:17.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.663+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c1a1a10 con 0x7f291c103810 2024-03-04T07:43:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.663+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c1a1eb0 con 0x7f291c103810 2024-03-04T07:43:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.666+0000 7f2913400640 1 -- 172.21.15.116:0/151674026 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f290c044070 con 0x7f291c103810 2024-03-04T07:43:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.666+0000 7f29243db740 1 Processor -- start 2024-03-04T07:43:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.666+0000 7f29243db740 1 -- start start 2024-03-04T07:43:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f29243db740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54795b50 0x55ef54795f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f29243db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54796540 0x55ef5479b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f29243db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef5479db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f29243db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef546c3090 con 0x55ef54796540 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f29243db740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef546fad30 con 0x55ef54795b50 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f29243db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef546a3a30 con 0x55ef5479b6b0 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54795b50 0x55ef54795f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54795b50 0x55ef54795f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45318/0 (socket says 172.21.15.116:45318) 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54796540 0x55ef5479b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291b400640 1 -- 172.21.15.116:0/2634843046 learned_addr learned my addr 172.21.15.116:0/2634843046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291a000640 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef5479db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291b400640 1 -- 172.21.15.116:0/2634843046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 msgr2=0x55ef5479db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291b400640 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef5479db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291b400640 1 -- 172.21.15.116:0/2634843046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54796540 msgr2=0x55ef5479b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.667+0000 7f291b400640 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54796540 0x55ef5479b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f291b400640 1 -- 172.21.15.116:0/2634843046 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef544d97d0 con 0x55ef54795b50 2024-03-04T07:43:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f291b400640 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54795b50 0x55ef54795f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2914005f60 tx=0x7f29140117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70862230aa466f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f2913400640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29000a1e00 0x7f29000a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f290b400640 1 -- 172.21.15.116:0/2634843046 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914009e50 con 0x55ef54795b50 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f290b400640 1 -- 172.21.15.116:0/2634843046 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2914012930 con 0x55ef54795b50 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f290b400640 1 -- 172.21.15.116:0/2634843046 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291400fb40 con 0x55ef54795b50 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f2913400640 1 -- 172.21.15.116:0/151674026 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f290c0ef950 con 0x7f291c103810 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.668+0000 7f291aa00640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29000a1e00 0x7f29000a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 -- 172.21.15.116:0/2634843046 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54795b50 msgr2=0x55ef54795f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54795b50 0x55ef54795f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2914005f60 tx=0x7f29140117b0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f291aa00640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29000a1e00 0x7f29000a42c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f290c0b9010 tx=0x7f290c0f5530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 -- 172.21.15.116:0/2634843046 shutdown_connections 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef5479db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54796540 0x55ef5479b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 --2- 172.21.15.116:0/2634843046 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54795b50 0x55ef54795f70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 -- 172.21.15.116:0/2634843046 >> 172.21.15.116:0/2634843046 conn(0x55ef547910c0 msgr2=0x55ef547914c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 -- 172.21.15.116:0/2634843046 shutdown_connections 2024-03-04T07:43:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 -- 172.21.15.116:0/2634843046 wait complete. 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.669+0000 7f29243db740 1 -- start start 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 1 Processor -- start 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef5479ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef546990c0 con 0x55ef54795b50 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef546f6dd0 con 0x55ef54796540 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef5479e660 con 0x55ef5479b6b0 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f29243db740 10 client.0 _ll_register_callbacks cb 0x55ef546b6860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f291b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef5479ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f291a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f291b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef5479ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35450/0 (socket says 172.21.15.116:35450) 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.670+0000 7f291b400640 1 -- 172.21.15.116:0/895362171 learned_addr learned my addr 172.21.15.116:0/895362171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291b400640 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291b400640 1 -- 172.21.15.116:0/895362171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 msgr2=0x55ef547a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291b400640 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291b400640 1 -- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 msgr2=0x55ef547a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291b400640 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291b400640 1 -- 172.21.15.116:0/895362171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef5440f3f0 con 0x55ef54795b50 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291a000640 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f291b400640 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef5479ff40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2914005e00 tx=0x7f291400f4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbf83ec85e0932a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f28ffe00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f28ffe00640 1 -- 172.21.15.116:0/895362171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914013e00 con 0x55ef54795b50 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f28ffe00640 1 -- 172.21.15.116:0/895362171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f291400f8c0 con 0x55ef54795b50 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.671+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2914012930 con 0x55ef54795b50 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f28ffe00640 1 -- 172.21.15.116:0/895362171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f291400fac0 con 0x55ef54795b50 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 10 client.15804 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:17.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ef543cf0f0 con 0x55ef54795b50 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f28ffe00640 1 -- 172.21.15.116:0/895362171 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2914012930 con 0x55ef54795b50 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f28ffe00640 1 -- 172.21.15.116:0/895362171 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2914031030 con 0x55ef54795b50 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 10 client.15804 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 10 client.15804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 20 client.15804 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef547ab730 0x55ef547adb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 4 client.15804 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.672+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef54708270 con 0x55ef547ab730 2024-03-04T07:43:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.673+0000 7f291a000640 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef547ab730 0x55ef547adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.673+0000 7f291a000640 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef547ab730 0x55ef547adb20 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7397ae50880d5395 server_cookie=d0be6aaec7a5ca32 in_seq=0 out_seq=0 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.673+0000 7f28ffe00640 10 client.15804 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f28ffe00640 1 -- 172.21.15.116:0/895362171 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ef54708270 con 0x55ef547ab730 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f28ffe00640 10 client.15804 handle_command_reply: tid=0 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 client.15804 shutdown 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 msgr2=0x55ef5479ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef5479ff40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2914005e00 tx=0x7f291400f4e0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 shutdown_connections 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef547ab730 0x55ef547adb20 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 --2- 172.21.15.116:0/895362171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef5479ff40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f28ffe00640 0 client.15804 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 >> 172.21.15.116:0/895362171 conn(0x55ef547910c0 msgr2=0x55ef54790360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 shutdown_connections 2024-03-04T07:43:17.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 1 -- 172.21.15.116:0/895362171 wait complete. 2024-03-04T07:43:17.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.674+0000 7f29243db740 20 client.15804 trim_cache size 0 max 16384 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 Processor -- start 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 -- start start 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef54796540 0x55ef547a7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef5479b6b0 0x55ef54794a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1a8d30 con 0x55ef54795b50 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1ab310 con 0x55ef5479b6b0 2024-03-04T07:43:17.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f29243db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1a9330 con 0x55ef54796540 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f291b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f291a000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef54796540 0x55ef547a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f291a000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef54796540 0x55ef547a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:52844/0 (socket says 172.21.15.116:52844) 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.692+0000 7f291a000640 1 -- 172.21.15.116:0/1979277326 learned_addr learned my addr 172.21.15.116:0/1979277326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291b400640 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef5479b6b0 0x55ef54794a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291a000640 1 -- 172.21.15.116:0/1979277326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef5479b6b0 msgr2=0x55ef54794a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291a000640 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef5479b6b0 0x55ef54794a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291a000640 1 -- 172.21.15.116:0/1979277326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 msgr2=0x55ef547a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291a000640 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291a000640 1 -- 172.21.15.116:0/1979277326 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef547adf80 con 0x55ef54796540 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291b400640 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f291a000640 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef54796540 0x55ef547a7450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55ef54792030 tx=0x7f2904009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c343da7a2a78b9e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f290aa00640 1 -- 172.21.15.116:0/1979277326 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290401f0f0 con 0x55ef54796540 2024-03-04T07:43:17.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f290aa00640 1 -- 172.21.15.116:0/1979277326 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2904023020 con 0x55ef54796540 2024-03-04T07:43:17.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.693+0000 7f290aa00640 1 -- 172.21.15.116:0/1979277326 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290401daa0 con 0x55ef54796540 2024-03-04T07:43:17.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 -- 172.21.15.116:0/1979277326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef54796540 msgr2=0x55ef547a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef54796540 0x55ef547a7450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55ef54792030 tx=0x7f2904009760 comp rx=0 tx=0).stop 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 -- 172.21.15.116:0/1979277326 shutdown_connections 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef5479b6b0 0x55ef54794a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef54796540 0x55ef547a7450 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 --2- 172.21.15.116:0/1979277326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 -- 172.21.15.116:0/1979277326 >> 172.21.15.116:0/1979277326 conn(0x55ef547910c0 msgr2=0x55ef54794320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 -- 172.21.15.116:0/1979277326 shutdown_connections 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 -- 172.21.15.116:0/1979277326 wait complete. 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.694+0000 7f29243db740 1 -- start start 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 1 Processor -- start 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1a9990 con 0x55ef54795b50 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1aacb0 con 0x55ef54796540 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1a9ff0 con 0x55ef5479b6b0 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f29243db740 10 client.0 _ll_register_callbacks cb 0x7f2914009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f291a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f291a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45356/0 (socket says 172.21.15.116:45356) 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f291b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.695+0000 7f291a000640 1 -- 172.21.15.116:0/2826506797 learned_addr learned my addr 172.21.15.116:0/2826506797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291b400640 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291a000640 1 -- 172.21.15.116:0/2826506797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 msgr2=0x55ef547a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291a000640 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291a000640 1 -- 172.21.15.116:0/2826506797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 msgr2=0x55ef547a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291a000640 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291a000640 1 -- 172.21.15.116:0/2826506797 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef547a6c40 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291b400640 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f291a000640 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a5640 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2904000f80 tx=0x7f290401d500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7436e619af9f90f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f28ffe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f28ffe00640 1 -- 172.21.15.116:0/2826506797 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904004c40 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f28ffe00640 1 -- 172.21.15.116:0/2826506797 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2904004de0 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2904023020 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f28ffe00640 1 -- 172.21.15.116:0/2826506797 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f290401f300 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f29243db740 10 client.25925 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.696+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ef548073d0 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f28ffe00640 1 -- 172.21.15.116:0/2826506797 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2904023020 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f28ffe00640 1 -- 172.21.15.116:0/2826506797 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2904005040 con 0x55ef54796540 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f29243db740 10 client.25925 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f29243db740 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:17.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f29243db740 20 client.25925 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f29243db740 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef54807cf0 0x55ef5480a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f29243db740 4 client.25925 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ef547025c0 con 0x55ef54807cf0 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.697+0000 7f291b400640 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef54807cf0 0x55ef5480a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f291b400640 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef54807cf0 0x55ef5480a110 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7db3af88c8c5a1c9 server_cookie=aa31841088e925a in_seq=0 out_seq=0 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f28ffe00640 10 client.25925 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f28ffe00640 1 -- 172.21.15.116:0/2826506797 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ef547025c0 con 0x55ef54807cf0 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f28ffe00640 10 client.25925 handle_command_reply: tid=0 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 client.25925 shutdown 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 msgr2=0x55ef547a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a5640 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2904000f80 tx=0x7f290401d500 comp rx=0 tx=0).stop 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 shutdown_connections 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ef54807cf0 0x55ef5480a110 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ef5479b6b0 0x55ef547a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ef54796540 0x55ef547a5640 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f28ffe00640 0 client.25925 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 --2- 172.21.15.116:0/2826506797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef54795b50 0x55ef547a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 >> 172.21.15.116:0/2826506797 conn(0x55ef547910c0 msgr2=0x55ef54794bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 shutdown_connections 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 1 -- 172.21.15.116:0/2826506797 wait complete. 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.698+0000 7f29243db740 20 client.25925 trim_cache size 0 max 16384 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:43:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "2ca33209-5ceb-4c25-afd4-5d61e7292593": { 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "2ca33209-5ceb-4c25-afd4-5d61e7292593", 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:43:17.705 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29000a1e00 msgr2=0x7f29000a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29000a1e00 0x7f29000a42c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f290c0b9010 tx=0x7f290c0f5530 comp rx=0 tx=0).stop 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 msgr2=0x7f291c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c199b60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f290c000c00 tx=0x7f290c005d50 comp rx=0 tx=0).stop 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 shutdown_connections 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29000a1e00 0x7f29000a42c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f291c1053c0 0x7f291c1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f291c104a00 0x7f291c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 --2- 172.21.15.116:0/151674026 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f291c103810 0x7f291c199b60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 >> 172.21.15.116:0/151674026 conn(0x7f291c0fed80 msgr2=0x7f291c107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 shutdown_connections 2024-03-04T07:43:17.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:17.699+0000 7f2921600640 1 -- 172.21.15.116:0/151674026 wait complete. 2024-03-04T07:43:17.717 INFO:tasks.cephfs.filesystem:scrub status for tag:2ca33209-5ceb-4c25-afd4-5d61e7292593 - {'path': '/', 'tag': '2ca33209-5ceb-4c25-afd4-5d61e7292593', 'options': 'recursive,force'} 2024-03-04T07:43:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:17 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:43:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:17 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:17 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:43:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:17 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:17 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:43:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:17 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:43:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:17 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:17 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:43:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:17 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:17 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:43:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:17 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:43:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:17 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:18.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:17 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:43:18.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:17 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:18.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:17 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:43:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:18 smithi116 ceph-mon[32150]: pgmap v477: 97 pgs: 97 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:18 smithi152 ceph-mon[37154]: pgmap v477: 97 pgs: 97 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:19.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:18 smithi133 ceph-mon[36505]: pgmap v477: 97 pgs: 97 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:21.139 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:20 smithi133 ceph-mon[36505]: pgmap v478: 97 pgs: 97 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:20 smithi116 ceph-mon[32150]: pgmap v478: 97 pgs: 97 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:20 smithi152 ceph-mon[37154]: pgmap v478: 97 pgs: 97 active+clean; 1.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:22 smithi116 ceph-mon[32150]: pgmap v479: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:43:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:22 smithi152 ceph-mon[37154]: pgmap v479: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:43:23.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:22 smithi133 ceph-mon[36505]: pgmap v479: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:43:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 -- 172.21.15.116:0/642181243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce8103830 msgr2=0x7f4ce8103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 --2- 172.21.15.116:0/642181243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce8103830 0x7f4ce8103c50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009900 tx=0x7f4cd802bc50 comp rx=0 tx=0).stop 2024-03-04T07:43:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 -- 172.21.15.116:0/642181243 shutdown_connections 2024-03-04T07:43:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 --2- 172.21.15.116:0/642181243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce81053e0 0x7f4ce810bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 --2- 172.21.15.116:0/642181243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ce8104a20 0x7f4ce8104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 --2- 172.21.15.116:0/642181243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce8103830 0x7f4ce8103c50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 -- 172.21.15.116:0/642181243 >> 172.21.15.116:0/642181243 conn(0x7f4ce80fedc0 msgr2=0x7f4ce81011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 -- 172.21.15.116:0/642181243 shutdown_connections 2024-03-04T07:43:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.376+0000 7f4cecc00640 1 -- 172.21.15.116:0/642181243 wait complete. 2024-03-04T07:43:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 Processor -- start 2024-03-04T07:43:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 -- start start 2024-03-04T07:43:23.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce8103830 0x7f4ce8199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ce8104a20 0x7f4ce8199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce81053e0 0x7f4ce81a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce810f4d0 con 0x7f4ce8103830 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce810f350 con 0x7f4ce81053e0 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.377+0000 7f4cecc00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce810f650 con 0x7f4ce8104a20 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce8103830 0x7f4ce8199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce8103830 0x7f4ce8199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35500/0 (socket says 172.21.15.116:35500) 2024-03-04T07:43:23.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce5e00640 1 -- 172.21.15.116:0/1912513127 learned_addr learned my addr 172.21.15.116:0/1912513127 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce6800640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce81053e0 0x7f4ce81a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce5400640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ce8104a20 0x7f4ce8199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce6800640 1 -- 172.21.15.116:0/1912513127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ce8104a20 msgr2=0x7f4ce8199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce6800640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ce8104a20 0x7f4ce8199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce6800640 1 -- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce8103830 msgr2=0x7f4ce8199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce6800640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce8103830 0x7f4ce8199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce6800640 1 -- 172.21.15.116:0/1912513127 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce81a16d0 con 0x7f4ce81053e0 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.378+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce8103830 0x7f4ce8199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.379+0000 7f4ce6800640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce81053e0 0x7f4ce81a1060 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000d6b0 tx=0x7f4ce000db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.379+0000 7f4cdea00640 1 -- 172.21.15.116:0/1912513127 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0017070 con 0x7f4ce81053e0 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.379+0000 7f4cdea00640 1 -- 172.21.15.116:0/1912513127 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ce0009d00 con 0x7f4ce81053e0 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.379+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce81a19c0 con 0x7f4ce81053e0 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.379+0000 7f4cdea00640 1 -- 172.21.15.116:0/1912513127 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0003d60 con 0x7f4ce81053e0 2024-03-04T07:43:23.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.379+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce81a1f10 con 0x7f4ce81053e0 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.382+0000 7f4cdea00640 1 -- 172.21.15.116:0/1912513127 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4ce0009820 con 0x7f4ce81053e0 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.382+0000 7f4cef976740 1 Processor -- start 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.382+0000 7f4cef976740 1 -- start start 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4cef976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c11c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4cef976740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c16d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4cef976740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c19840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4cef976740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560632b77370 con 0x560632c11850 2024-03-04T07:43:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4cef976740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560632b20550 con 0x560632c173b0 2024-03-04T07:43:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4cef976740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560632b73410 con 0x560632c12240 2024-03-04T07:43:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c11c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c16d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c11c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35506/0 (socket says 172.21.15.116:35506) 2024-03-04T07:43:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5400640 1 -- 172.21.15.116:0/3712226303 learned_addr learned my addr 172.21.15.116:0/3712226303 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5400640 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c19840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5e00640 1 -- 172.21.15.116:0/3712226303 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 msgr2=0x560632c19840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c19840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5e00640 1 -- 172.21.15.116:0/3712226303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 msgr2=0x560632c11c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.383+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c11c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4ce5e00640 1 -- 172.21.15.116:0/3712226303 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560632b343c0 con 0x560632c12240 2024-03-04T07:43:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4cdea00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4cc80a1ea0 0x7f4cc80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4cdea00640 1 -- 172.21.15.116:0/1912513127 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4ce00cd550 con 0x7f4ce81053e0 2024-03-04T07:43:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4ce5400640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4cc80a1ea0 0x7f4cc80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c16d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8005ab0 tx=0x7f4cd8005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2240926d7198c0f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4cd7000640 1 -- 172.21.15.116:0/3712226303 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8042070 con 0x560632c12240 2024-03-04T07:43:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4cd7000640 1 -- 172.21.15.116:0/3712226303 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cd8004ba0 con 0x560632c12240 2024-03-04T07:43:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.384+0000 7f4cd7000640 1 -- 172.21.15.116:0/3712226303 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80053c0 con 0x560632c12240 2024-03-04T07:43:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 -- 172.21.15.116:0/3712226303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 msgr2=0x560632c16d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c16d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8005ab0 tx=0x7f4cd8005dd0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4ce5400640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4cc80a1ea0 0x7f4cc80a4360 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x560632c12aa0 tx=0x7f4cd0009ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 -- 172.21.15.116:0/3712226303 shutdown_connections 2024-03-04T07:43:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c19840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c16d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 --2- 172.21.15.116:0/3712226303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c11c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 -- 172.21.15.116:0/3712226303 >> 172.21.15.116:0/3712226303 conn(0x560632c0cdc0 msgr2=0x560632c0d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 -- 172.21.15.116:0/3712226303 shutdown_connections 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.385+0000 7f4cef976740 1 -- 172.21.15.116:0/3712226303 wait complete. 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 -- start start 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 Processor -- start 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c12240 0x560632c1c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c23070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560632c1a300 con 0x560632c12240 2024-03-04T07:43:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560632b339b0 con 0x560632c173b0 2024-03-04T07:43:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.386+0000 7f4cef976740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560632c1a480 con 0x560632c11850 2024-03-04T07:43:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4cef976740 10 client.0 _ll_register_callbacks cb 0x560632b6f340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c12240 0x560632c1c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c23070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c23070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45404/0 (socket says 172.21.15.116:45404) 2024-03-04T07:43:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 -- 172.21.15.116:0/3590438427 learned_addr learned my addr 172.21.15.116:0/3590438427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c12240 0x560632c1c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35522/0 (socket says 172.21.15.116:35522) 2024-03-04T07:43:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 -- 172.21.15.116:0/3590438427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c11850 msgr2=0x560632c1bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 -- 172.21.15.116:0/3590438427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c12240 msgr2=0x560632c1c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c12240 0x560632c1c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 -- 172.21.15.116:0/3590438427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56063288c210 con 0x560632c173b0 2024-03-04T07:43:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ce6800640 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c23070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4ce009a280 tx=0x7f4ce0096690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=989bc8fd54e439dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ccf400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ccf400640 1 -- 172.21.15.116:0/3590438427 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0020070 con 0x560632c173b0 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4ccf400640 1 -- 172.21.15.116:0/3590438427 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4ce009e120 con 0x560632c173b0 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.387+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4cd8004ba0 con 0x560632c173b0 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4ccf400640 1 -- 172.21.15.116:0/3590438427 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ce0094e60 con 0x560632c173b0 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 10 client.25937 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560632958620 con 0x560632c173b0 2024-03-04T07:43:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4ccf400640 1 -- 172.21.15.116:0/3590438427 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cd8004ba0 con 0x560632c173b0 2024-03-04T07:43:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4ccf400640 1 -- 172.21.15.116:0/3590438427 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4ce00cd070 con 0x560632c173b0 2024-03-04T07:43:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 10 client.25937 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 10 client.25937 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 20 client.25937 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c27910 0x560632c29d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 4 client.25937 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560632b848b0 con 0x560632c27910 2024-03-04T07:43:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.388+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c27910 0x560632c29d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.389+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c27910 0x560632c29d00 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd380784de0e0aba server_cookie=4767a73e2e8c61c3 in_seq=0 out_seq=0 2024-03-04T07:43:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.389+0000 7f4ccf400640 10 client.25937 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4ccf400640 1 -- 172.21.15.116:0/3590438427 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560632b848b0 con 0x560632c27910 2024-03-04T07:43:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4ccf400640 10 client.25937 handle_command_reply: tid=0 2024-03-04T07:43:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 client.25937 shutdown 2024-03-04T07:43:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 msgr2=0x560632c23070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c23070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4ce009a280 tx=0x7f4ce0096690 comp rx=0 tx=0).stop 2024-03-04T07:43:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 shutdown_connections 2024-03-04T07:43:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c27910 0x560632c29d00 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c23070 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4ccf400640 0 client.25937 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c12240 0x560632c1c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 --2- 172.21.15.116:0/3590438427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 >> 172.21.15.116:0/3590438427 conn(0x560632c0cdc0 msgr2=0x560632c0bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 shutdown_connections 2024-03-04T07:43:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.390+0000 7f4cef976740 1 -- 172.21.15.116:0/3590438427 wait complete. 2024-03-04T07:43:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.391+0000 7f4cef976740 20 client.25937 trim_cache size 0 max 16384 2024-03-04T07:43:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.405+0000 7f4cef976740 1 Processor -- start 2024-03-04T07:43:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4cef976740 1 -- start start 2024-03-04T07:43:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4cef976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4cef976740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c23030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4cef976740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c1c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4cef976740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81a8b80 con 0x560632c11850 2024-03-04T07:43:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4cef976740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81ab160 con 0x560632c173b0 2024-03-04T07:43:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4cef976740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81a9180 con 0x560632c12240 2024-03-04T07:43:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4ce5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4ce5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35528/0 (socket says 172.21.15.116:35528) 2024-03-04T07:43:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4ce5e00640 1 -- 172.21.15.116:0/307315109 learned_addr learned my addr 172.21.15.116:0/307315109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4ce6800640 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c1c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4ce5400640 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c23030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.406+0000 7f4ce5e00640 1 -- 172.21.15.116:0/307315109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 msgr2=0x560632c23030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c23030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4ce5e00640 1 -- 172.21.15.116:0/307315109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 msgr2=0x560632c1c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c1c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4ce5e00640 1 -- 172.21.15.116:0/307315109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560632c0db30 con 0x560632c11850 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4ce6800640 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c1c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c1bb60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4cd80096c0 tx=0x7f4cd802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd4f0b4ccbc9fbc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4cd6600640 1 -- 172.21.15.116:0/307315109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8042040 con 0x560632c11850 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4cd6600640 1 -- 172.21.15.116:0/307315109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cd802cd50 con 0x560632c11850 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.407+0000 7f4cd6600640 1 -- 172.21.15.116:0/307315109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80054d0 con 0x560632c11850 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 -- 172.21.15.116:0/307315109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 msgr2=0x560632c1bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c1bb60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4cd80096c0 tx=0x7f4cd802e040 comp rx=0 tx=0).stop 2024-03-04T07:43:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 -- 172.21.15.116:0/307315109 shutdown_connections 2024-03-04T07:43:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c1c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c23030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 --2- 172.21.15.116:0/307315109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c11850 0x560632c1bb60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 -- 172.21.15.116:0/307315109 >> 172.21.15.116:0/307315109 conn(0x560632c0cdc0 msgr2=0x560632c0d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 -- 172.21.15.116:0/307315109 shutdown_connections 2024-03-04T07:43:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 -- 172.21.15.116:0/307315109 wait complete. 2024-03-04T07:43:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.408+0000 7f4cef976740 1 -- start start 2024-03-04T07:43:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 1 Processor -- start 2024-03-04T07:43:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c208b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c20df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c21d10 0x560632c22190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81a97e0 con 0x560632c21d10 2024-03-04T07:43:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81aab00 con 0x560632c173b0 2024-03-04T07:43:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81a9e40 con 0x560632c12240 2024-03-04T07:43:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4cef976740 10 client.0 _ll_register_callbacks cb 0x7f4ce0003700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4ce5400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c20df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4ce6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c21d10 0x560632c22190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4ce5400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c20df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45424/0 (socket says 172.21.15.116:45424) 2024-03-04T07:43:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4ce5400640 1 -- 172.21.15.116:0/2423387123 learned_addr learned my addr 172.21.15.116:0/2423387123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.409+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c208b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce5400640 1 -- 172.21.15.116:0/2423387123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 msgr2=0x560632c208b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce5400640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c208b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce5400640 1 -- 172.21.15.116:0/2423387123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c21d10 msgr2=0x560632c22190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce5400640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c21d10 0x560632c22190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce5400640 1 -- 172.21.15.116:0/2423387123 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560632c226d0 con 0x560632c173b0 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c208b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce6800640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c21d10 0x560632c22190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ce5400640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c20df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000ebe0 tx=0x7f4cd0009760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6039f171f74e8a41 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ccf400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ccf400640 1 -- 172.21.15.116:0/2423387123 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0030060 con 0x560632c173b0 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4ccf400640 1 -- 172.21.15.116:0/2423387123 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4cd0010d00 con 0x560632c173b0 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.410+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4cd802cd50 con 0x560632c173b0 2024-03-04T07:43:23.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4ccf400640 1 -- 172.21.15.116:0/2423387123 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cd001f480 con 0x560632c173b0 2024-03-04T07:43:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 10 client.25943 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560632c836a0 con 0x560632c173b0 2024-03-04T07:43:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4ccf400640 1 -- 172.21.15.116:0/2423387123 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cd802cd50 con 0x560632c173b0 2024-03-04T07:43:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4ccf400640 1 -- 172.21.15.116:0/2423387123 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4cd0035070 con 0x560632c173b0 2024-03-04T07:43:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 10 client.25943 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 10 client.25943 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:23.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 20 client.25943 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c84060 0x560632c86480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 4 client.25943 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:43:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560632b7ec00 con 0x560632c84060 2024-03-04T07:43:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.411+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c84060 0x560632c86480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.412+0000 7f4ce5e00640 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c84060 0x560632c86480 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c183763e3587cf7 server_cookie=6feff94aa7f0ea19 in_seq=0 out_seq=0 2024-03-04T07:43:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.412+0000 7f4ccf400640 10 client.25943 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4ccf400640 1 -- 172.21.15.116:0/2423387123 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560632b7ec00 con 0x560632c84060 2024-03-04T07:43:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4ccf400640 10 client.25943 handle_command_reply: tid=0 2024-03-04T07:43:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 client.25943 shutdown 2024-03-04T07:43:23.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 msgr2=0x560632c20df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c20df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000ebe0 tx=0x7f4cd0009760 comp rx=0 tx=0).stop 2024-03-04T07:43:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 shutdown_connections 2024-03-04T07:43:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560632c84060 0x560632c86480 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560632c21d10 0x560632c22190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560632c173b0 0x560632c20df0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 --2- 172.21.15.116:0/2423387123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560632c12240 0x560632c208b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4ccf400640 0 client.25943 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 >> 172.21.15.116:0/2423387123 conn(0x560632c0cdc0 msgr2=0x560632c1bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 shutdown_connections 2024-03-04T07:43:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 1 -- 172.21.15.116:0/2423387123 wait complete. 2024-03-04T07:43:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.427+0000 7f4cef976740 20 client.25943 trim_cache size 0 max 16384 2024-03-04T07:43:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:43:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:43:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:43:23.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:43:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4cc80a1ea0 msgr2=0x7f4cc80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4cc80a1ea0 0x7f4cc80a4360 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x560632c12aa0 tx=0x7f4cd0009ba0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce81053e0 msgr2=0x7f4ce81a1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce81053e0 0x7f4ce81a1060 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000d6b0 tx=0x7f4ce000db70 comp rx=0 tx=0).stop 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 shutdown_connections 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ce81053e0 0x7f4ce81a1060 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4cc80a1ea0 0x7f4cc80a4360 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ce8104a20 0x7f4ce8199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 --2- 172.21.15.116:0/1912513127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ce8103830 0x7f4ce8199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.428+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 >> 172.21.15.116:0/1912513127 conn(0x7f4ce80fedc0 msgr2=0x7f4ce8107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.429+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 shutdown_connections 2024-03-04T07:43:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:23.429+0000 7f4cecc00640 1 -- 172.21.15.116:0/1912513127 wait complete. 2024-03-04T07:43:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:23 smithi116 ceph-mon[32150]: pgmap v480: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:43:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:23 smithi152 ceph-mon[37154]: pgmap v480: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:43:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:23 smithi133 ceph-mon[36505]: pgmap v480: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:43:27.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:26 smithi116 ceph-mon[32150]: pgmap v481: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:43:27.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:26 smithi152 ceph-mon[37154]: pgmap v481: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:43:27.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:26 smithi133 ceph-mon[36505]: pgmap v481: 97 pgs: 97 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:43:29.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:28 smithi116 ceph-mon[32150]: pgmap v482: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:43:29.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:28 smithi152 ceph-mon[37154]: pgmap v482: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:43:29.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:28 smithi133 ceph-mon[36505]: pgmap v482: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:43:31.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:30 smithi116 ceph-mon[32150]: pgmap v483: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:43:31.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:30 smithi152 ceph-mon[37154]: pgmap v483: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:43:31.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:30 smithi133 ceph-mon[36505]: pgmap v483: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:43:33.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:32 smithi116 ceph-mon[32150]: pgmap v484: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:43:33.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:32 smithi152 ceph-mon[37154]: pgmap v484: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:43:33.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:32 smithi133 ceph-mon[36505]: pgmap v484: 97 pgs: 97 active+clean; 1.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:43:35.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:34 smithi116 ceph-mon[32150]: pgmap v485: 97 pgs: 97 active+clean; 1.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:35.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:34 smithi152 ceph-mon[37154]: pgmap v485: 97 pgs: 97 active+clean; 1.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:34 smithi133 ceph-mon[36505]: pgmap v485: 97 pgs: 97 active+clean; 1.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 -- 172.21.15.116:0/3325996027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8075f90 msgr2=0x7f5aa80763b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 --2- 172.21.15.116:0/3325996027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8075f90 0x7f5aa80763b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009860 tx=0x7f5a9802bb70 comp rx=0 tx=0).stop 2024-03-04T07:43:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 -- 172.21.15.116:0/3325996027 shutdown_connections 2024-03-04T07:43:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 --2- 172.21.15.116:0/3325996027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5aa8077b70 0x7f5aa8078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 --2- 172.21.15.116:0/3325996027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 0x7f5aa8077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 --2- 172.21.15.116:0/3325996027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8075f90 0x7f5aa80763b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 -- 172.21.15.116:0/3325996027 >> 172.21.15.116:0/3325996027 conn(0x7f5aa80fedc0 msgr2=0x7f5aa81011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 -- 172.21.15.116:0/3325996027 shutdown_connections 2024-03-04T07:43:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.569+0000 7f5aaf200640 1 -- 172.21.15.116:0/3325996027 wait complete. 2024-03-04T07:43:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.570+0000 7f5aaf200640 1 Processor -- start 2024-03-04T07:43:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.570+0000 7f5aaf200640 1 -- start start 2024-03-04T07:43:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aaf200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5aa8075f90 0x7f5aa819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aaf200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 0x7f5aa819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aaf200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8077b70 0x7f5aa81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aaf200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81138c0 con 0x7f5aa8077b70 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aaf200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8113740 con 0x7f5aa8075f90 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aaf200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8113a40 con 0x7f5aa80771d0 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aa7400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 0x7f5aa819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5aa8075f90 0x7f5aa819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aa7400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 0x7f5aa819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58506/0 (socket says 172.21.15.116:58506) 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aa7400640 1 -- 172.21.15.116:0/2339736676 learned_addr learned my addr 172.21.15.116:0/2339736676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aad200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8077b70 0x7f5aa81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aa7e00640 1 -- 172.21.15.116:0/2339736676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 msgr2=0x7f5aa819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.571+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 0x7f5aa819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa7e00640 1 -- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8077b70 msgr2=0x7f5aa81a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8077b70 0x7f5aa81a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa7e00640 1 -- 172.21.15.116:0/2339736676 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa81a5ac0 con 0x7f5aa8075f90 2024-03-04T07:43:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa7400640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 0x7f5aa819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5aa8075f90 0x7f5aa819dea0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5a9802c0b0 tx=0x7f5a98032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa4c00640 1 -- 172.21.15.116:0/2339736676 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98041020 con 0x7f5aa8075f90 2024-03-04T07:43:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa4c00640 1 -- 172.21.15.116:0/2339736676 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a9802c9f0 con 0x7f5aa8075f90 2024-03-04T07:43:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aa4c00640 1 -- 172.21.15.116:0/2339736676 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98039040 con 0x7f5aa8075f90 2024-03-04T07:43:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa81a5d50 con 0x7f5aa8075f90 2024-03-04T07:43:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.572+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa81a61c0 con 0x7f5aa8075f90 2024-03-04T07:43:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.575+0000 7f5aa4c00640 1 -- 172.21.15.116:0/2339736676 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5a98044020 con 0x7f5aa8075f90 2024-03-04T07:43:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.575+0000 7f5ab1e80740 1 Processor -- start 2024-03-04T07:43:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5ab1e80740 1 -- start start 2024-03-04T07:43:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916cf240 0x5555916d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916d6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555591634370 con 0x5555916d43b0 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555915dd550 con 0x5555916cf240 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555591630410 con 0x5555916ce850 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916cf240 0x5555916d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916cf240 0x5555916d3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38770/0 (socket says 172.21.15.116:38770) 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5aa7e00640 1 -- 172.21.15.116:0/205779773 learned_addr learned my addr 172.21.15.116:0/205779773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5aad200640 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.576+0000 7f5aa7400640 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916d6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa7e00640 1 -- 172.21.15.116:0/205779773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 msgr2=0x5555916cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa7e00640 1 -- 172.21.15.116:0/205779773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 msgr2=0x5555916d6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916d6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa7e00640 1 -- 172.21.15.116:0/205779773 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555915f13c0 con 0x5555916cf240 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aad200640 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa7400640 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916d6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:36.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916cf240 0x5555916d3d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5a980f5540 tx=0x7f5a980f5630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760759e86123b805 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa4c00640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a8c0a1e20 0x7f5a8c0a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aa4c00640 1 -- 172.21.15.116:0/2339736676 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5a980ef5d0 con 0x7f5aa8075f90 2024-03-04T07:43:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5aad200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a8c0a1e20 0x7f5a8c0a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.577+0000 7f5a9d600640 1 -- 172.21.15.116:0/205779773 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980f5e60 con 0x5555916cf240 2024-03-04T07:43:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5a9d600640 1 -- 172.21.15.116:0/205779773 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a980fc030 con 0x5555916cf240 2024-03-04T07:43:36.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5a9d600640 1 -- 172.21.15.116:0/205779773 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980bbc20 con 0x5555916cf240 2024-03-04T07:43:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5ab1e80740 1 -- 172.21.15.116:0/205779773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916cf240 msgr2=0x5555916d3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5aad200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a8c0a1e20 0x7f5a8c0a42e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5555916cfaa0 tx=0x7f5aa0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916cf240 0x5555916d3d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5a980f5540 tx=0x7f5a980f5630 comp rx=0 tx=0).stop 2024-03-04T07:43:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5ab1e80740 1 -- 172.21.15.116:0/205779773 shutdown_connections 2024-03-04T07:43:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916d6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916cf240 0x5555916d3d40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/205779773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.578+0000 7f5ab1e80740 1 -- 172.21.15.116:0/205779773 >> 172.21.15.116:0/205779773 conn(0x5555916c9dc0 msgr2=0x5555916ca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.579+0000 7f5ab1e80740 1 -- 172.21.15.116:0/205779773 shutdown_connections 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.579+0000 7f5ab1e80740 1 -- 172.21.15.116:0/205779773 wait complete. 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.579+0000 7f5ab1e80740 1 -- start start 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.579+0000 7f5ab1e80740 1 Processor -- start 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916d88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916d8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916d43b0 0x5555916dfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555916d7120 con 0x5555916cf240 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555915f09b0 con 0x5555916d43b0 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555916d72a0 con 0x5555916ce850 2024-03-04T07:43:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5ab1e80740 10 client.0 _ll_register_callbacks cb 0x55559162c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aad200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916d43b0 0x5555916dfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916d8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916d88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916d8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41816/0 (socket says 172.21.15.116:41816) 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aad200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916d43b0 0x5555916dfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38786/0 (socket says 172.21.15.116:38786) 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916d88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58538/0 (socket says 172.21.15.116:58538) 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7400640 1 -- 172.21.15.116:0/819497274 learned_addr learned my addr 172.21.15.116:0/819497274 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7e00640 1 -- 172.21.15.116:0/819497274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916d43b0 msgr2=0x5555916dfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916d43b0 0x5555916dfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7e00640 1 -- 172.21.15.116:0/819497274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 msgr2=0x5555916d8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916d8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.580+0000 7f5aa7e00640 1 -- 172.21.15.116:0/819497274 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555591349210 con 0x5555916ce850 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916d88d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x5555916cfcb0 tx=0x7f5a980bcb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61aa4b55dfc833a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5a92000640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5a92000640 1 -- 172.21.15.116:0/819497274 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980bbe00 con 0x5555916ce850 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5a92000640 1 -- 172.21.15.116:0/819497274 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5a980ba360 con 0x5555916ce850 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a980fc030 con 0x5555916ce850 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5a92000640 1 -- 172.21.15.116:0/819497274 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a980bc090 con 0x5555916ce850 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5ab1e80740 10 client.25909 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555591415620 con 0x5555916ce850 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.581+0000 7f5a92000640 1 -- 172.21.15.116:0/819497274 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a980fc030 con 0x5555916ce850 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5a92000640 1 -- 172.21.15.116:0/819497274 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5a9803c030 con 0x5555916ce850 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5ab1e80740 10 client.25909 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5ab1e80740 10 client.25909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5ab1e80740 20 client.25909 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5555916e4340 0x5555916e6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5ab1e80740 4 client.25909 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5555916418b0 con 0x5555916e4340 2024-03-04T07:43:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.582+0000 7f5aa7400640 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5555916e4340 0x5555916e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.583+0000 7f5aa7400640 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5555916e4340 0x5555916e6730 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85b1af6892218263 server_cookie=889a97d69868130 in_seq=0 out_seq=0 2024-03-04T07:43:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.583+0000 7f5a92000640 10 client.25909 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5a92000640 1 -- 172.21.15.116:0/819497274 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5555916418b0 con 0x5555916e4340 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5a92000640 10 client.25909 handle_command_reply: tid=0 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 client.25909 shutdown 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 msgr2=0x5555916d88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916d88d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x5555916cfcb0 tx=0x7f5a980bcb30 comp rx=0 tx=0).stop 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 shutdown_connections 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5555916e4340 0x5555916e6730 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916d43b0 0x5555916dfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916d8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/819497274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916ce850 0x5555916d88d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 >> 172.21.15.116:0/819497274 conn(0x5555916c9dc0 msgr2=0x5555916c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5a92000640 0 client.25909 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 shutdown_connections 2024-03-04T07:43:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 1 -- 172.21.15.116:0/819497274 wait complete. 2024-03-04T07:43:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.584+0000 7f5ab1e80740 20 client.25909 trim_cache size 0 max 16384 2024-03-04T07:43:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 Processor -- start 2024-03-04T07:43:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 -- start start 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916dfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916d43b0 0x5555916cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81acff0 con 0x5555916cf240 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81af5d0 con 0x5555916ce850 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81ad5f0 con 0x5555916d43b0 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916d8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38790/0 (socket says 172.21.15.116:38790) 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5aa7e00640 1 -- 172.21.15.116:0/483376011 learned_addr learned my addr 172.21.15.116:0/483376011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5aa7400640 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916dfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.600+0000 7f5aad200640 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916d43b0 0x5555916cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5aa7e00640 1 -- 172.21.15.116:0/483376011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916d43b0 msgr2=0x5555916cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916d43b0 0x5555916cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5aa7e00640 1 -- 172.21.15.116:0/483376011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 msgr2=0x5555916dfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916dfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5aa7e00640 1 -- 172.21.15.116:0/483376011 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555916c5610 con 0x5555916ce850 2024-03-04T07:43:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5aad200640 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916d43b0 0x5555916cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916d8d70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005d20 tx=0x7f5a98005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa648232afc1649 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5a9cc00640 1 -- 172.21.15.116:0/483376011 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980b8560 con 0x5555916ce850 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5a9cc00640 1 -- 172.21.15.116:0/483376011 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a980fc070 con 0x5555916ce850 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.601+0000 7f5a9cc00640 1 -- 172.21.15.116:0/483376011 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980f5390 con 0x5555916ce850 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 -- 172.21.15.116:0/483376011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 msgr2=0x5555916d8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916d8d70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005d20 tx=0x7f5a98005780 comp rx=0 tx=0).stop 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 -- 172.21.15.116:0/483376011 shutdown_connections 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916d43b0 0x5555916cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916cf240 0x5555916dfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/483376011 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916d8d70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 -- 172.21.15.116:0/483376011 >> 172.21.15.116:0/483376011 conn(0x5555916c9dc0 msgr2=0x5555916cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 -- 172.21.15.116:0/483376011 shutdown_connections 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.602+0000 7f5ab1e80740 1 -- 172.21.15.116:0/483376011 wait complete. 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 -- start start 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 Processor -- start 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916dd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916cf240 0x5555916dda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916de9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81adc50 con 0x5555916d43b0 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81aef70 con 0x5555916ce850 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81ae2b0 con 0x5555916cf240 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5ab1e80740 10 client.0 _ll_register_callbacks cb 0x7f5aa0009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5aa7400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916cf240 0x5555916dda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5aa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916dd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38796/0 (socket says 172.21.15.116:38796) 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.603+0000 7f5aa7e00640 1 -- 172.21.15.116:0/2146137164 learned_addr learned my addr 172.21.15.116:0/2146137164 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aad200640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916de9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aa7e00640 1 -- 172.21.15.116:0/2146137164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916cf240 msgr2=0x5555916dda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916cf240 0x5555916dda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aa7e00640 1 -- 172.21.15.116:0/2146137164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 msgr2=0x5555916de9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916de9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aa7e00640 1 -- 172.21.15.116:0/2146137164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555916defc0 con 0x5555916ce850 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aad200640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916de9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aa7400640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916cf240 0x5555916dda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5aa7e00640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916dd520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5a980c4530 tx=0x7f5a98005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=788e3e0fa2e1e488 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5a92000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5a92000640 1 -- 172.21.15.116:0/2146137164 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980b8560 con 0x5555916ce850 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5a92000640 1 -- 172.21.15.116:0/2146137164 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5a98032640 con 0x5555916ce850 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.604+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a980fc070 con 0x5555916ce850 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5a92000640 1 -- 172.21.15.116:0/2146137164 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a980bbe00 con 0x5555916ce850 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 10 client.25961 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5555916defc0 con 0x5555916ce850 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5a92000640 1 -- 172.21.15.116:0/2146137164 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a980fc070 con 0x5555916ce850 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5a92000640 1 -- 172.21.15.116:0/2146137164 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5a98081ce0 con 0x5555916ce850 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 10 client.25961 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 10 client.25961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 20 client.25961 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55559173feb0 0x5555917422a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 4 client.25961 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55559163bc00 con 0x55559173feb0 2024-03-04T07:43:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.605+0000 7f5aa7400640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55559173feb0 0x5555917422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.606+0000 7f5aa7400640 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55559173feb0 0x5555917422a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5435e730ece44424 server_cookie=2a16d342d8b508d7 in_seq=0 out_seq=0 2024-03-04T07:43:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.606+0000 7f5a92000640 10 client.25961 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.621+0000 7f5a92000640 1 -- 172.21.15.116:0/2146137164 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55559163bc00 con 0x55559173feb0 2024-03-04T07:43:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.621+0000 7f5a92000640 10 client.25961 handle_command_reply: tid=0 2024-03-04T07:43:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.621+0000 7f5ab1e80740 1 client.25961 shutdown 2024-03-04T07:43:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.621+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 msgr2=0x5555916dd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.621+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916dd520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5a980c4530 tx=0x7f5a98005f90 comp rx=0 tx=0).stop 2024-03-04T07:43:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 shutdown_connections 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55559173feb0 0x5555917422a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5555916d43b0 0x5555916de9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555916cf240 0x5555916dda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5a92000640 0 client.25961 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 --2- 172.21.15.116:0/2146137164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5555916ce850 0x5555916dd520 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 >> 172.21.15.116:0/2146137164 conn(0x5555916c9dc0 msgr2=0x5555916cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 shutdown_connections 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 1 -- 172.21.15.116:0/2146137164 wait complete. 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.622+0000 7f5ab1e80740 20 client.25961 trim_cache size 0 max 16384 2024-03-04T07:43:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:43:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:43:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:43:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:43:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a8c0a1e20 msgr2=0x7f5a8c0a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a8c0a1e20 0x7f5a8c0a42e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5555916cfaa0 tx=0x7f5aa0009500 comp rx=0 tx=0).stop 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5aa8075f90 msgr2=0x7f5aa819dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5aa8075f90 0x7f5aa819dea0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5a9802c0b0 tx=0x7f5a98032430 comp rx=0 tx=0).stop 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 shutdown_connections 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5aa8077b70 0x7f5aa81a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a8c0a1e20 0x7f5a8c0a42e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5aa80771d0 0x7f5aa819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 --2- 172.21.15.116:0/2339736676 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5aa8075f90 0x7f5aa819dea0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 >> 172.21.15.116:0/2339736676 conn(0x7f5aa80fedc0 msgr2=0x7f5aa81008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 shutdown_connections 2024-03-04T07:43:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:36.623+0000 7f5aaf200640 1 -- 172.21.15.116:0/2339736676 wait complete. 2024-03-04T07:43:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:36 smithi116 ceph-mon[32150]: pgmap v486: 97 pgs: 97 active+clean; 1.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:37.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:36 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:43:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:36 smithi152 ceph-mon[37154]: pgmap v486: 97 pgs: 97 active+clean; 1.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:37.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:36 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:43:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:36 smithi133 ceph-mon[36505]: pgmap v486: 97 pgs: 97 active+clean; 1.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:43:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:36 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:43:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:43:38.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:43:38.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:43:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:43:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:43:38.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:43:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:43:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:43:38.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:43:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:38 smithi116 ceph-mon[32150]: pgmap v487: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:43:39.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:38 smithi152 ceph-mon[37154]: pgmap v487: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:43:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:38 smithi133 ceph-mon[36505]: pgmap v487: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:43:41.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:40 smithi116 ceph-mon[32150]: pgmap v488: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:43:41.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:40 smithi152 ceph-mon[37154]: pgmap v488: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:43:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:40 smithi133 ceph-mon[36505]: pgmap v488: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:43:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:42 smithi116 ceph-mon[32150]: pgmap v489: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:42 smithi152 ceph-mon[37154]: pgmap v489: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:42 smithi133 ceph-mon[36505]: pgmap v489: 97 pgs: 97 active+clean; 2.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:45.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:44 smithi116 ceph-mon[32150]: pgmap v490: 97 pgs: 97 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:44 smithi152 ceph-mon[37154]: pgmap v490: 97 pgs: 97 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:45.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:44 smithi133 ceph-mon[36505]: pgmap v490: 97 pgs: 97 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:46 smithi116 ceph-mon[32150]: pgmap v491: 97 pgs: 97 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:47.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:46 smithi152 ceph-mon[37154]: pgmap v491: 97 pgs: 97 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:46 smithi133 ceph-mon[36505]: pgmap v491: 97 pgs: 97 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:43:47.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:43:47.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:43:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.943+0000 7f789f600640 1 -- 172.21.15.116:0/3910926437 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7898104a00 msgr2=0x7f7898104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.943+0000 7f789f600640 1 --2- 172.21.15.116:0/3910926437 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7898104a00 0x7f7898104e80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7884009900 tx=0x7f788402bc30 comp rx=0 tx=0).stop 2024-03-04T07:43:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.943+0000 7f789f600640 1 -- 172.21.15.116:0/3910926437 shutdown_connections 2024-03-04T07:43:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.944+0000 7f789f600640 1 --2- 172.21.15.116:0/3910926437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78981053c0 0x7f789810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.944+0000 7f789f600640 1 --2- 172.21.15.116:0/3910926437 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7898104a00 0x7f7898104e80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.944+0000 7f789f600640 1 --2- 172.21.15.116:0/3910926437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 0x7f7898103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.944+0000 7f789f600640 1 -- 172.21.15.116:0/3910926437 >> 172.21.15.116:0/3910926437 conn(0x7f78980fed80 msgr2=0x7f78981011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.944+0000 7f789f600640 1 -- 172.21.15.116:0/3910926437 shutdown_connections 2024-03-04T07:43:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.944+0000 7f789f600640 1 -- 172.21.15.116:0/3910926437 wait complete. 2024-03-04T07:43:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 Processor -- start 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 -- start start 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 0x7f7898199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7898104a00 0x7f789819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78981053c0 0x7f78981a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789810f4e0 con 0x7f7898103810 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789810f360 con 0x7f78981053c0 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.945+0000 7f789f600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789810f660 con 0x7f7898104a00 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 0x7f7898199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 0x7f7898199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58304/0 (socket says 172.21.15.116:58304) 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 -- 172.21.15.116:0/3195535203 learned_addr learned my addr 172.21.15.116:0/3195535203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789d600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78981053c0 0x7f78981a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f7897e00640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7898104a00 0x7f789819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 -- 172.21.15.116:0/3195535203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7898104a00 msgr2=0x7f789819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7898104a00 0x7f789819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 -- 172.21.15.116:0/3195535203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78981053c0 msgr2=0x7f78981a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78981053c0 0x7f78981a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 -- 172.21.15.116:0/3195535203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78981a1740 con 0x7f7898103810 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789d600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78981053c0 0x7f78981a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.946+0000 7f789cc00640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 0x7f7898199b20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f788c00e9c0 tx=0x7f788c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.947+0000 7f7895600640 1 -- 172.21.15.116:0/3195535203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c00cd80 con 0x7f7898103810 2024-03-04T07:43:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.947+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78981a1a30 con 0x7f7898103810 2024-03-04T07:43:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.947+0000 7f7895600640 1 -- 172.21.15.116:0/3195535203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f788c0035f0 con 0x7f7898103810 2024-03-04T07:43:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.947+0000 7f7895600640 1 -- 172.21.15.116:0/3195535203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c010690 con 0x7f7898103810 2024-03-04T07:43:47.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.947+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78981a1fa0 con 0x7f7898103810 2024-03-04T07:43:47.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.950+0000 7f78a21f2740 1 Processor -- start 2024-03-04T07:43:47.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.950+0000 7f78a21f2740 1 -- start start 2024-03-04T07:43:47.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b6b50 0x55de4b4b6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b7540 0x55de4b4bc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4beb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f78a21f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b3e4090 con 0x55de4b4bc6b0 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f78a21f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b41bd30 con 0x55de4b4b6b50 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f78a21f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b3c4a30 con 0x55de4b4b7540 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b6b50 0x55de4b4b6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b6b50 0x55de4b4b6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59946/0 (socket says 172.21.15.116:59946) 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 -- 172.21.15.116:0/3286498744 learned_addr learned my addr 172.21.15.116:0/3286498744 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f7897e00640 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4beb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f7895600640 1 -- 172.21.15.116:0/3195535203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f788c0037a0 con 0x7f7898103810 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 -- 172.21.15.116:0/3286498744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b7540 msgr2=0x55de4b4bc040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b7540 0x55de4b4bc040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 -- 172.21.15.116:0/3286498744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 msgr2=0x55de4b4beb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4beb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f789d600640 1 -- 172.21.15.116:0/3286498744 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de4b1fa7d0 con 0x55de4b4b6b50 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.951+0000 7f7897e00640 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4beb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.952+0000 7f789d600640 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b6b50 0x55de4b4b6f70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f78900117e0 tx=0x7f7890011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aabe59d2c30550a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.952+0000 7f7889c00640 1 -- 172.21.15.116:0/3286498744 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890011310 con 0x55de4b4b6b50 2024-03-04T07:43:47.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.952+0000 7f7889c00640 1 -- 172.21.15.116:0/3286498744 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7890003e50 con 0x55de4b4b6b50 2024-03-04T07:43:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.952+0000 7f7889c00640 1 -- 172.21.15.116:0/3286498744 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890004600 con 0x55de4b4b6b50 2024-03-04T07:43:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.952+0000 7f7895600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78800a1d80 0x7f78800a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 -- 172.21.15.116:0/3286498744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b6b50 msgr2=0x55de4b4b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b6b50 0x55de4b4b6f70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f78900117e0 tx=0x7f7890011ca0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f7895600640 1 -- 172.21.15.116:0/3195535203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f788c0ce200 con 0x7f7898103810 2024-03-04T07:43:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f789cc00640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78800a1d80 0x7f78800a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 -- 172.21.15.116:0/3286498744 shutdown_connections 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4beb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b7540 0x55de4b4bc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3286498744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b6b50 0x55de4b4b6f70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 -- 172.21.15.116:0/3286498744 >> 172.21.15.116:0/3286498744 conn(0x55de4b4b20c0 msgr2=0x55de4b4b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 -- 172.21.15.116:0/3286498744 shutdown_connections 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 -- 172.21.15.116:0/3286498744 wait complete. 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f78a21f2740 1 -- start start 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.953+0000 7f789cc00640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78800a1d80 0x7f78800a4240 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f788c09aec0 tx=0x7f788c0ce480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 1 Processor -- start 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b3ba0c0 con 0x55de4b4bc6b0 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b417dd0 con 0x55de4b4b7540 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de4b4bf660 con 0x55de4b4b6b50 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f78a21f2740 10 client.0 _ll_register_callbacks cb 0x55de4b3d7860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f7897e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f789d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.954+0000 7f7897e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59954/0 (socket says 172.21.15.116:59954) 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f7897e00640 1 -- 172.21.15.116:0/138415575 learned_addr learned my addr 172.21.15.116:0/138415575 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f789d600640 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f7897e00640 1 -- 172.21.15.116:0/138415575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 msgr2=0x55de4b4c0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f7897e00640 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f7897e00640 1 -- 172.21.15.116:0/138415575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 msgr2=0x55de4b4c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f7897e00640 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f7897e00640 1 -- 172.21.15.116:0/138415575 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de4b1303f0 con 0x55de4b4b7540 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f7897e00640 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c1490 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7884006010 tx=0x7f7884004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91bbb10e2bddb913 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f787e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f787e000640 1 -- 172.21.15.116:0/138415575 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884005d50 con 0x55de4b4b7540 2024-03-04T07:43:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.955+0000 7f787e000640 1 -- 172.21.15.116:0/138415575 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7884005230 con 0x55de4b4b7540 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7890003e50 con 0x55de4b4b7540 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f787e000640 1 -- 172.21.15.116:0/138415575 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7884005430 con 0x55de4b4b7540 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 10 client.25973 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55de4b0f00f0 con 0x55de4b4b7540 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f787e000640 1 -- 172.21.15.116:0/138415575 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7890003e50 con 0x55de4b4b7540 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f787e000640 1 -- 172.21.15.116:0/138415575 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f78840039c0 con 0x55de4b4b7540 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 10 client.25973 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 10 client.25973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 20 client.25973 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b4cca60 0x55de4b4cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 4 client.25973 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.956+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de4b429270 con 0x55de4b4cca60 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.957+0000 7f789d600640 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b4cca60 0x55de4b4cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.957+0000 7f789d600640 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b4cca60 0x55de4b4cee50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a75ef74a7a0153e8 server_cookie=e8b44888053e5025 in_seq=0 out_seq=0 2024-03-04T07:43:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.957+0000 7f787e000640 10 client.25973 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f787e000640 1 -- 172.21.15.116:0/138415575 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55de4b429270 con 0x55de4b4cca60 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f787e000640 10 client.25973 handle_command_reply: tid=0 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 client.25973 shutdown 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 msgr2=0x55de4b4c1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c1490 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7884006010 tx=0x7f7884004270 comp rx=0 tx=0).stop 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 shutdown_connections 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b4cca60 0x55de4b4cee50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c1490 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f787e000640 0 client.25973 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 --2- 172.21.15.116:0/138415575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 >> 172.21.15.116:0/138415575 conn(0x55de4b4b20c0 msgr2=0x55de4b4b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 shutdown_connections 2024-03-04T07:43:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 1 -- 172.21.15.116:0/138415575 wait complete. 2024-03-04T07:43:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.958+0000 7f78a21f2740 20 client.25973 trim_cache size 0 max 16384 2024-03-04T07:43:47.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 Processor -- start 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 -- start start 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b7540 0x55de4b4c8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4bc6b0 0x55de4b4b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981a8da0 con 0x55de4b4b7540 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981ab380 con 0x55de4b4bc6b0 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f78a21f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981a93a0 con 0x55de4b4b6b50 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f7897e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b7540 0x55de4b4c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f789d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4bc6b0 0x55de4b4b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f7897e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b7540 0x55de4b4c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58336/0 (socket says 172.21.15.116:58336) 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f789d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4bc6b0 0x55de4b4b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59960/0 (socket says 172.21.15.116:59960) 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f7897e00640 1 -- 172.21.15.116:0/2838579532 learned_addr learned my addr 172.21.15.116:0/2838579532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:47.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.976+0000 7f789d600640 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7897e00640 1 -- 172.21.15.116:0/2838579532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 msgr2=0x55de4b4c0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:47.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7897e00640 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7897e00640 1 -- 172.21.15.116:0/2838579532 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4bc6b0 msgr2=0x55de4b4b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:47.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7897e00640 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4bc6b0 0x55de4b4b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7897e00640 1 -- 172.21.15.116:0/2838579532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de4b4ac770 con 0x55de4b4b7540 2024-03-04T07:43:47.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7897e00640 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b7540 0x55de4b4c8490 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7884003e30 tx=0x7f788402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbd194af09b12e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7889200640 1 -- 172.21.15.116:0/2838579532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884000e60 con 0x55de4b4b7540 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7889200640 1 -- 172.21.15.116:0/2838579532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7884009520 con 0x55de4b4b7540 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.977+0000 7f7889200640 1 -- 172.21.15.116:0/2838579532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884052410 con 0x55de4b4b7540 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 -- 172.21.15.116:0/2838579532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b7540 msgr2=0x55de4b4c8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b7540 0x55de4b4c8490 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7884003e30 tx=0x7f788402bc30 comp rx=0 tx=0).stop 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 -- 172.21.15.116:0/2838579532 shutdown_connections 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4bc6b0 0x55de4b4b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b7540 0x55de4b4c8490 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 --2- 172.21.15.116:0/2838579532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4b6b50 0x55de4b4c0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 -- 172.21.15.116:0/2838579532 >> 172.21.15.116:0/2838579532 conn(0x55de4b4b20c0 msgr2=0x55de4b4b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 -- 172.21.15.116:0/2838579532 shutdown_connections 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 -- 172.21.15.116:0/2838579532 wait complete. 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.978+0000 7f78a21f2740 1 -- start start 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 1 Processor -- start 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b6b50 0x55de4b4c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981a9a00 con 0x55de4b4b6b50 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981aad20 con 0x55de4b4b7540 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981aa060 con 0x55de4b4bc6b0 2024-03-04T07:43:47.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f78a21f2740 10 client.0 _ll_register_callbacks cb 0x7f7884009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f7897e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f7897e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59966/0 (socket says 172.21.15.116:59966) 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f7897e00640 1 -- 172.21.15.116:0/3931323892 learned_addr learned my addr 172.21.15.116:0/3931323892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.979+0000 7f789d600640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b6b50 0x55de4b4c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f789d600640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f7897e00640 1 -- 172.21.15.116:0/3931323892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4bc6b0 msgr2=0x55de4b4c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f7897e00640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f7897e00640 1 -- 172.21.15.116:0/3931323892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b6b50 msgr2=0x55de4b4c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f7897e00640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b6b50 0x55de4b4c81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f7897e00640 1 -- 172.21.15.116:0/3931323892 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de4b4c7400 con 0x55de4b4b7540 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f789d600640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f7897e00640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c6850 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f788402c170 tx=0x7f788402cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33fe1ea0112172e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f787e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f787e000640 1 -- 172.21.15.116:0/3931323892 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884005d50 con 0x55de4b4b7540 2024-03-04T07:43:47.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7884009520 con 0x55de4b4b7540 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.980+0000 7f787e000640 1 -- 172.21.15.116:0/3931323892 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f78840037f0 con 0x55de4b4b7540 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f787e000640 1 -- 172.21.15.116:0/3931323892 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78840039f0 con 0x55de4b4b7540 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 10 client.25979 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55de4b5291a0 con 0x55de4b4b7540 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f787e000640 1 -- 172.21.15.116:0/3931323892 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7884009520 con 0x55de4b4b7540 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f787e000640 1 -- 172.21.15.116:0/3931323892 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7884052410 con 0x55de4b4b7540 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 10 client.25979 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 10 client.25979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 20 client.25979 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b529aa0 0x55de4b52bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 4 client.25979 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.981+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55de4b4235c0 con 0x55de4b529aa0 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f789d600640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b529aa0 0x55de4b52bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:47.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f789d600640 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b529aa0 0x55de4b52bec0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd776d448173068c server_cookie=3e8c3457d350ece5 in_seq=0 out_seq=0 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f787e000640 10 client.25979 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f787e000640 1 -- 172.21.15.116:0/3931323892 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55de4b4235c0 con 0x55de4b529aa0 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f787e000640 10 client.25979 handle_command_reply: tid=0 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f78a21f2740 1 client.25979 shutdown 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 msgr2=0x55de4b4c6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.982+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c6850 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f788402c170 tx=0x7f788402cdb0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 shutdown_connections 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55de4b529aa0 0x55de4b52bec0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55de4b4bc6b0 0x55de4b4c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f787e000640 0 client.25979 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55de4b4b7540 0x55de4b4c6850 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 --2- 172.21.15.116:0/3931323892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de4b4b6b50 0x55de4b4c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 >> 172.21.15.116:0/3931323892 conn(0x55de4b4b20c0 msgr2=0x55de4b4b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 shutdown_connections 2024-03-04T07:43:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 1 -- 172.21.15.116:0/3931323892 wait complete. 2024-03-04T07:43:47.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f78a21f2740 20 client.25979 trim_cache size 0 max 16384 2024-03-04T07:43:47.988 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:43:47.988 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:43:47.988 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:43:47.988 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:43:47.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78800a1d80 msgr2=0x7f78800a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f789f600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78800a1d80 0x7f78800a4240 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f788c09aec0 tx=0x7f788c0ce480 comp rx=0 tx=0).stop 2024-03-04T07:43:47.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 msgr2=0x7f7898199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:47.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.983+0000 7f789f600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 0x7f7898199b20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f788c00e9c0 tx=0x7f788c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 shutdown_connections 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78981053c0 0x7f78981a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78800a1d80 0x7f78800a4240 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7898104a00 0x7f789819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 --2- 172.21.15.116:0/3195535203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7898103810 0x7f7898199b20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 >> 172.21.15.116:0/3195535203 conn(0x7f78980fed80 msgr2=0x7f7898107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 shutdown_connections 2024-03-04T07:43:47.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:47.984+0000 7f789f600640 1 -- 172.21.15.116:0/3195535203 wait complete. 2024-03-04T07:43:48.003 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:43:48.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:43:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 -- 172.21.15.116:0/2176134873 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84104810 msgr2=0x7f8b84104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 --2- 172.21.15.116:0/2176134873 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84104810 0x7f8b84104c90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009900 tx=0x7f8b7002bc30 comp rx=0 tx=0).stop 2024-03-04T07:43:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 -- 172.21.15.116:0/2176134873 shutdown_connections 2024-03-04T07:43:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 --2- 172.21.15.116:0/2176134873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b841051d0 0x7f8b8410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 --2- 172.21.15.116:0/2176134873 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84104810 0x7f8b84104c90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 --2- 172.21.15.116:0/2176134873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b84103620 0x7f8b84103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 -- 172.21.15.116:0/2176134873 >> 172.21.15.116:0/2176134873 conn(0x7f8b840febb0 msgr2=0x7f8b84100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 -- 172.21.15.116:0/2176134873 shutdown_connections 2024-03-04T07:43:48.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.215+0000 7f8b83200640 1 -- 172.21.15.116:0/2176134873 wait complete. 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.216+0000 7f8b83200640 1 Processor -- start 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.216+0000 7f8b83200640 1 -- start start 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b83200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84103620 0x7f8b841999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b83200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b84104810 0x7f8b84199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b83200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b841051d0 0x7f8b841a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b83200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8410f300 con 0x7f8b841051d0 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b83200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8410f180 con 0x7f8b84103620 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b83200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8410f480 con 0x7f8b84104810 2024-03-04T07:43:48.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84103620 0x7f8b841999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84103620 0x7f8b841999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59978/0 (socket says 172.21.15.116:59978) 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 -- 172.21.15.116:0/589566301 learned_addr learned my addr 172.21.15.116:0/589566301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81400640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b84104810 0x7f8b84199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b82800640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b841051d0 0x7f8b841a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 -- 172.21.15.116:0/589566301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b84104810 msgr2=0x7f8b84199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b84104810 0x7f8b84199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 -- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b841051d0 msgr2=0x7f8b841a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b841051d0 0x7f8b841a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.217+0000 7f8b81e00640 1 -- 172.21.15.116:0/589566301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b841a15d0 con 0x7f8b84103620 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.218+0000 7f8b81e00640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84103620 0x7f8b841999b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00eb70 tx=0x7f8b7c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.218+0000 7f8b7aa00640 1 -- 172.21.15.116:0/589566301 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c003b70 con 0x7f8b84103620 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.218+0000 7f8b7aa00640 1 -- 172.21.15.116:0/589566301 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b7c00ee70 con 0x7f8b84103620 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.218+0000 7f8b7aa00640 1 -- 172.21.15.116:0/589566301 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c010830 con 0x7f8b84103620 2024-03-04T07:43:48.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.218+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b841a18c0 con 0x7f8b84103620 2024-03-04T07:43:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.218+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b841a1de0 con 0x7f8b84103620 2024-03-04T07:43:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.219+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8410a8e0 con 0x7f8b84103620 2024-03-04T07:43:48.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.221+0000 7f8b7aa00640 1 -- 172.21.15.116:0/589566301 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8b7c004020 con 0x7f8b84103620 2024-03-04T07:43:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.222+0000 7f8b7aa00640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8b680a1e00 0x7f8b680a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.222+0000 7f8b7aa00640 1 -- 172.21.15.116:0/589566301 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8b7c0ce3f0 con 0x7f8b84103620 2024-03-04T07:43:48.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.222+0000 7f8b81400640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8b680a1e00 0x7f8b680a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.223+0000 7f8b81400640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8b680a1e00 0x7f8b680a42c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b70000c00 tx=0x7f8b7002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.226+0000 7f8b7aa00640 1 -- 172.21.15.116:0/589566301 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8b7c09b7e0 con 0x7f8b84103620 2024-03-04T07:43:48.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.463+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b8402cff0 con 0x7f8b84103620 2024-03-04T07:43:48.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.465+0000 7f8b7aa00640 1 -- 172.21.15.116:0/589566301 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f8b7c09b0e0 con 0x7f8b84103620 2024-03-04T07:43:48.465 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:43:48.465 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:43:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8b680a1e00 msgr2=0x7f8b680a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8b680a1e00 0x7f8b680a42c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b70000c00 tx=0x7f8b7002df90 comp rx=0 tx=0).stop 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84103620 msgr2=0x7f8b841999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84103620 0x7f8b841999b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00eb70 tx=0x7f8b7c00c6a0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 shutdown_connections 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b841051d0 0x7f8b841a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8b680a1e00 0x7f8b680a42c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b84104810 0x7f8b84199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 --2- 172.21.15.116:0/589566301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8b84103620 0x7f8b841999b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 >> 172.21.15.116:0/589566301 conn(0x7f8b840febb0 msgr2=0x7f8b84107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 shutdown_connections 2024-03-04T07:43:48.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.468+0000 7f8b83200640 1 -- 172.21.15.116:0/589566301 wait complete. 2024-03-04T07:43:48.488 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:43:48.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.699+0000 7f3023200640 1 -- 172.21.15.116:0/1530501679 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 msgr2=0x7f3024073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.699+0000 7f3023200640 1 --2- 172.21.15.116:0/1530501679 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f3024073a60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3014009cd0 tx=0x7f301402c0f0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.700+0000 7f3023200640 1 -- 172.21.15.116:0/1530501679 shutdown_connections 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.700+0000 7f3023200640 1 --2- 172.21.15.116:0/1530501679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f302410daa0 0x7f302410fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.700+0000 7f3023200640 1 --2- 172.21.15.116:0/1530501679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3024074030 0x7f30240744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.700+0000 7f3023200640 1 --2- 172.21.15.116:0/1530501679 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f3024073a60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.700+0000 7f3023200640 1 -- 172.21.15.116:0/1530501679 >> 172.21.15.116:0/1530501679 conn(0x7f30240fca40 msgr2=0x7f30240fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.700+0000 7f3023200640 1 -- 172.21.15.116:0/1530501679 shutdown_connections 2024-03-04T07:43:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.700+0000 7f3023200640 1 -- 172.21.15.116:0/1530501679 wait complete. 2024-03-04T07:43:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 Processor -- start 2024-03-04T07:43:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 -- start start 2024-03-04T07:43:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3024074030 0x7f302419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f302419e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f302410daa0 0x7f30241a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3024113740 con 0x7f3024074030 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241135c0 con 0x7f3024075610 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.701+0000 7f3023200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241138c0 con 0x7f302410daa0 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3021e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3024074030 0x7f302419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f302419e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3021e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3024074030 0x7f302419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58364/0 (socket says 172.21.15.116:58364) 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f302419e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60002/0 (socket says 172.21.15.116:60002) 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3021e00640 1 -- 172.21.15.116:0/3655789862 learned_addr learned my addr 172.21.15.116:0/3655789862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3022800640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f302410daa0 0x7f30241a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 -- 172.21.15.116:0/3655789862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f302410daa0 msgr2=0x7f30241a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f302410daa0 0x7f30241a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 -- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3024074030 msgr2=0x7f302419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3024074030 0x7f302419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 -- 172.21.15.116:0/3655789862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30241a5a30 con 0x7f3024075610 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.702+0000 7f3019400640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f302419e350 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f300c00e9f0 tx=0x7f300c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.703+0000 7f301b400640 1 -- 172.21.15.116:0/3655789862 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c00ccc0 con 0x7f3024075610 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.703+0000 7f301b400640 1 -- 172.21.15.116:0/3655789862 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f300c010910 con 0x7f3024075610 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.703+0000 7f301b400640 1 -- 172.21.15.116:0/3655789862 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c018960 con 0x7f3024075610 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.703+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30241a5d20 con 0x7f3024075610 2024-03-04T07:43:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.703+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30241a61c0 con 0x7f3024075610 2024-03-04T07:43:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.706+0000 7f301b400640 1 -- 172.21.15.116:0/3655789862 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f300c010ac0 con 0x7f3024075610 2024-03-04T07:43:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.706+0000 7f302bfd1740 1 Processor -- start 2024-03-04T07:43:48.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.706+0000 7f302bfd1740 1 -- start start 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578612290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e578617360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e578619e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f302bfd1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e578576d60 con 0x55e578611e70 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f302bfd1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e57851fcf0 con 0x55e578612860 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f302bfd1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e578572e00 con 0x55e5786179d0 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3022800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578612290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3021e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e578617360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3022800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578612290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58374/0 (socket says 172.21.15.116:58374) 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3022800640 1 -- 172.21.15.116:0/2732322920 learned_addr learned my addr 172.21.15.116:0/2732322920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3019400640 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e578619e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3021e00640 1 -- 172.21.15.116:0/2732322920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 msgr2=0x55e578619e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3021e00640 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e578619e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3021e00640 1 -- 172.21.15.116:0/2732322920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 msgr2=0x55e578612290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f301b400640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30080a1e00 0x7f30080a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3021e00640 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578612290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3021e00640 1 -- 172.21.15.116:0/2732322920 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e57828b3f0 con 0x55e578612860 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3019400640 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e578619e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.707+0000 7f3022800640 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578612290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.708+0000 7f301b400640 1 -- 172.21.15.116:0/3655789862 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f300c0ce0f0 con 0x7f3024075610 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.708+0000 7f3021e00640 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e578617360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3014000f80 tx=0x7f3014005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=600c8bf22f40c05 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.708+0000 7f3022800640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30080a1e00 0x7f30080a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.708+0000 7f3013400640 1 -- 172.21.15.116:0/2732322920 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014043070 con 0x55e578612860 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.708+0000 7f3013400640 1 -- 172.21.15.116:0/2732322920 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3014032c90 con 0x55e578612860 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.708+0000 7f3013400640 1 -- 172.21.15.116:0/2732322920 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014039040 con 0x55e578612860 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 -- 172.21.15.116:0/2732322920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 msgr2=0x55e578617360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f3022800640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30080a1e00 0x7f30080a42c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55e5786130c0 tx=0x7f301c0046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e578617360 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3014000f80 tx=0x7f3014005ef0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 -- 172.21.15.116:0/2732322920 shutdown_connections 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e578619e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e578617360 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2732322920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578612290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 -- 172.21.15.116:0/2732322920 >> 172.21.15.116:0/2732322920 conn(0x55e57860d3e0 msgr2=0x55e57860d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 -- 172.21.15.116:0/2732322920 shutdown_connections 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 -- 172.21.15.116:0/2732322920 wait complete. 2024-03-04T07:43:48.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.709+0000 7f302bfd1740 1 -- start start 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 1 Processor -- start 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e57861c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e57861c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e5786238c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e57861a920 con 0x55e578611e70 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e578515220 con 0x55e578612860 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e57861aaa0 con 0x55e5786179d0 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f302bfd1740 10 client.0 _ll_register_callbacks cb 0x55e57828abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f3021e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e57861c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f3019400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e57861c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f3019400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e57861c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60028/0 (socket says 172.21.15.116:60028) 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f3022800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e5786238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f3021e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e57861c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58378/0 (socket says 172.21.15.116:58378) 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.710+0000 7f3019400640 1 -- 172.21.15.116:0/2759629690 learned_addr learned my addr 172.21.15.116:0/2759629690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3019400640 1 -- 172.21.15.116:0/2759629690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 msgr2=0x55e5786238c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3019400640 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e5786238c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3019400640 1 -- 172.21.15.116:0/2759629690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 msgr2=0x55e57861c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3019400640 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e57861c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3019400640 1 -- 172.21.15.116:0/2759629690 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5783557d0 con 0x55e578612860 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3019400640 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e57861c880 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f300c0972f0 tx=0x7f300c09a9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fda789600dcc4b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3007e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3007e00640 1 -- 172.21.15.116:0/2759629690 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c0d02e0 con 0x55e578612860 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3007e00640 1 -- 172.21.15.116:0/2759629690 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f300c09a2e0 con 0x55e578612860 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3014032c90 con 0x55e578612860 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f3007e00640 1 -- 172.21.15.116:0/2759629690 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f300c09ef30 con 0x55e578612860 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f302bfd1740 10 client.25997 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:48.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.711+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e578626ca0 con 0x55e578612860 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f3007e00640 1 -- 172.21.15.116:0/2759629690 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3014032c90 con 0x55e578612860 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f3007e00640 1 -- 172.21.15.116:0/2759629690 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f300c018b00 con 0x55e578612860 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f302bfd1740 10 client.25997 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f302bfd1740 10 client.25997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f302bfd1740 20 client.25997 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5786282f0 0x55e57862a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f302bfd1740 4 client.25997 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5785842a0 con 0x55e5786282f0 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.712+0000 7f3021e00640 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5786282f0 0x55e57862a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.713+0000 7f3021e00640 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5786282f0 0x55e57862a790 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c35d66cbc49d3235 server_cookie=32e6139bf28b369a in_seq=0 out_seq=0 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.713+0000 7f3007e00640 10 client.25997 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.713+0000 7f3007e00640 1 -- 172.21.15.116:0/2759629690 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e5785842a0 con 0x55e5786282f0 2024-03-04T07:43:48.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.713+0000 7f3007e00640 10 client.25997 handle_command_reply: tid=0 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.713+0000 7f302bfd1740 1 client.25997 shutdown 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 msgr2=0x55e57861c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e57861c880 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f300c0972f0 tx=0x7f300c09a9d0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 shutdown_connections 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5786282f0 0x55e57862a790 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5786179d0 0x55e5786238c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e578612860 0x55e57861c880 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 --2- 172.21.15.116:0/2759629690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e57861c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f3007e00640 0 client.25997 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 >> 172.21.15.116:0/2759629690 conn(0x55e57860d3e0 msgr2=0x55e57860c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 shutdown_connections 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 1 -- 172.21.15.116:0/2759629690 wait complete. 2024-03-04T07:43:48.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.714+0000 7f302bfd1740 20 client.25997 trim_cache size 0 max 16384 2024-03-04T07:43:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.728+0000 7f302bfd1740 1 Processor -- start 2024-03-04T07:43:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.728+0000 7f302bfd1740 1 -- start start 2024-03-04T07:43:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578610d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578612860 0x55e578623880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e57862f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f302bfd1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241acfa0 con 0x55e578611e70 2024-03-04T07:43:48.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f302bfd1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241af580 con 0x55e5786179d0 2024-03-04T07:43:48.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f302bfd1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241ad5a0 con 0x55e578612860 2024-03-04T07:43:48.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e57862f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3021e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578610d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e57862f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60036/0 (socket says 172.21.15.116:60036) 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 -- 172.21.15.116:0/3163147117 learned_addr learned my addr 172.21.15.116:0/3163147117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3019400640 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578612860 0x55e578623880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 -- 172.21.15.116:0/3163147117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578612860 msgr2=0x55e578623880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578612860 0x55e578623880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 -- 172.21.15.116:0/3163147117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 msgr2=0x55e578610d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578610d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3022800640 1 -- 172.21.15.116:0/3163147117 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e57861b750 con 0x55e5786179d0 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.729+0000 7f3021e00640 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578610d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.730+0000 7f3022800640 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e57862f050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f301c00f7d0 tx=0x7f301c013030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1566883ba78a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.730+0000 7f3012a00640 1 -- 172.21.15.116:0/3163147117 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c02d070 con 0x55e5786179d0 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.730+0000 7f3012a00640 1 -- 172.21.15.116:0/3163147117 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f301c01ce30 con 0x55e5786179d0 2024-03-04T07:43:48.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.730+0000 7f3012a00640 1 -- 172.21.15.116:0/3163147117 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0273e0 con 0x55e5786179d0 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 -- 172.21.15.116:0/3163147117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 msgr2=0x55e57862f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e57862f050 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f301c00f7d0 tx=0x7f301c013030 comp rx=0 tx=0).stop 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 -- 172.21.15.116:0/3163147117 shutdown_connections 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e57862f050 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578612860 0x55e578623880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3163147117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578611e70 0x55e578610d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 -- 172.21.15.116:0/3163147117 >> 172.21.15.116:0/3163147117 conn(0x55e57860d3e0 msgr2=0x55e578610640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 -- 172.21.15.116:0/3163147117 shutdown_connections 2024-03-04T07:43:48.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 -- 172.21.15.116:0/3163147117 wait complete. 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 -- start start 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.731+0000 7f302bfd1740 1 Processor -- start 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578611e70 0x55e578621830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578612860 0x55e578621d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f302bfd1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e578622d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f302bfd1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241adc00 con 0x55e578612860 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f302bfd1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241aef20 con 0x55e5786179d0 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f302bfd1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241ae260 con 0x55e578611e70 2024-03-04T07:43:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f302bfd1740 10 client.0 _ll_register_callbacks cb 0x7f300c00ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3019400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578612860 0x55e578621d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3019400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578612860 0x55e578621d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58388/0 (socket says 172.21.15.116:58388) 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3019400640 1 -- 172.21.15.116:0/3057612130 learned_addr learned my addr 172.21.15.116:0/3057612130 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3021e00640 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578611e70 0x55e578621830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3022800640 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e578622d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3019400640 1 -- 172.21.15.116:0/3057612130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578611e70 msgr2=0x55e578621830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3019400640 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578611e70 0x55e578621830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3019400640 1 -- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 msgr2=0x55e578622d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.732+0000 7f3019400640 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e578622d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f3019400640 1 -- 172.21.15.116:0/3057612130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e578623370 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f3019400640 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578612860 0x55e578621d70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f300c097860 tx=0x7f300c0b3ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422991bce598e073 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f3007e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f3007e00640 1 -- 172.21.15.116:0/3057612130 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c09a4e0 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f301c01ce30 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f3007e00640 1 -- 172.21.15.116:0/3057612130 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f300c010430 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f3007e00640 1 -- 172.21.15.116:0/3057612130 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f300c0a3580 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f302bfd1740 10 client.15888 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e5786844e0 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.733+0000 7f3007e00640 1 -- 172.21.15.116:0/3057612130 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f301c01ce30 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f3007e00640 1 -- 172.21.15.116:0/3057612130 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f300c09fe50 con 0x55e578612860 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f302bfd1740 10 client.15888 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f302bfd1740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:48.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f302bfd1740 20 client.15888 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e578684b70 0x55e578686f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f302bfd1740 4 client.15888 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e57857e5f0 con 0x55e578684b70 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f3021e00640 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e578684b70 0x55e578686f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f3021e00640 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e578684b70 0x55e578686f90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eef19c9334fa42b9 server_cookie=5e4b4763c74d3ce in_seq=0 out_seq=0 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.734+0000 7f3007e00640 10 client.15888 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f3007e00640 1 -- 172.21.15.116:0/3057612130 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e57857e5f0 con 0x55e578684b70 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f3007e00640 10 client.15888 handle_command_reply: tid=0 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 client.15888 shutdown 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578612860 msgr2=0x55e578621d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578612860 0x55e578621d70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f300c097860 tx=0x7f300c0b3ea0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 shutdown_connections 2024-03-04T07:43:48.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e578684b70 0x55e578686f90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5786179d0 0x55e578622d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e578612860 0x55e578621d70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 --2- 172.21.15.116:0/3057612130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e578611e70 0x55e578621830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f3007e00640 0 client.15888 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 >> 172.21.15.116:0/3057612130 conn(0x55e57860d3e0 msgr2=0x55e578610ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 shutdown_connections 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 1 -- 172.21.15.116:0/3057612130 wait complete. 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.735+0000 7f302bfd1740 20 client.15888 trim_cache size 0 max 16384 2024-03-04T07:43:48.740 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30080a1e00 msgr2=0x7f30080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30080a1e00 0x7f30080a42c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55e5786130c0 tx=0x7f301c0046c0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 msgr2=0x7f302419e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f302419e350 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f300c00e9f0 tx=0x7f300c00eeb0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 shutdown_connections 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30080a1e00 0x7f30080a42c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f302410daa0 0x7f30241a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3024075610 0x7f302419e350 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 --2- 172.21.15.116:0/3655789862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3024074030 0x7f302419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 >> 172.21.15.116:0/3655789862 conn(0x7f30240fca40 msgr2=0x7f3024100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 shutdown_connections 2024-03-04T07:43:48.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:48.736+0000 7f3023200640 1 -- 172.21.15.116:0/3655789862 wait complete. 2024-03-04T07:43:48.750 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:48 smithi116 ceph-mon[32150]: pgmap v492: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:43:48.750 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:48 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/589566301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:43:49.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:48 smithi152 ceph-mon[37154]: pgmap v492: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:43:49.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:48 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/589566301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:43:49.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:48 smithi133 ceph-mon[36505]: pgmap v492: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:43:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:48 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/589566301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:43:49.754 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:43:49.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:43:49.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:43:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 -- 172.21.15.116:0/525597947 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f030010da10 msgr2=0x7f030010fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 --2- 172.21.15.116:0/525597947 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f030010da10 0x7f030010fe00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009cd0 tx=0x7f02f802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 -- 172.21.15.116:0/525597947 shutdown_connections 2024-03-04T07:43:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 --2- 172.21.15.116:0/525597947 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f030010da10 0x7f030010fe00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 --2- 172.21.15.116:0/525597947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 0x7f03000744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 --2- 172.21.15.116:0/525597947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0300075610 0x7f0300073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 -- 172.21.15.116:0/525597947 >> 172.21.15.116:0/525597947 conn(0x7f03000fc9b0 msgr2=0x7f03000fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 -- 172.21.15.116:0/525597947 shutdown_connections 2024-03-04T07:43:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.766+0000 7f0306800640 1 -- 172.21.15.116:0/525597947 wait complete. 2024-03-04T07:43:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.767+0000 7f0306800640 1 Processor -- start 2024-03-04T07:43:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.767+0000 7f0306800640 1 -- start start 2024-03-04T07:43:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0306800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 0x7f030019dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0306800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0300075610 0x7f030019e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0306800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f030010da10 0x7f03001a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0306800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0300113690 con 0x7f0300074030 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0306800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0300113510 con 0x7f0300075610 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0306800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0300113810 con 0x7f030010da10 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 0x7f030019dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0304a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0300075610 0x7f030019e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 0x7f030019dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58400/0 (socket says 172.21.15.116:58400) 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 -- 172.21.15.116:0/3337943496 learned_addr learned my addr 172.21.15.116:0/3337943496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305e00640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f030010da10 0x7f03001a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 -- 172.21.15.116:0/3337943496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f030010da10 msgr2=0x7f03001a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f030010da10 0x7f03001a5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 -- 172.21.15.116:0/3337943496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0300075610 msgr2=0x7f030019e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0300075610 0x7f030019e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0305400640 1 -- 172.21.15.116:0/3337943496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03001a5900 con 0x7f0300074030 2024-03-04T07:43:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.768+0000 7f0304a00640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0300075610 0x7f030019e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.769+0000 7f0305400640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 0x7f030019dcf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f02f400d950 tx=0x7f02f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.769+0000 7f02fe000640 1 -- 172.21.15.116:0/3337943496 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4017070 con 0x7f0300074030 2024-03-04T07:43:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.769+0000 7f02fe000640 1 -- 172.21.15.116:0/3337943496 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f4009c30 con 0x7f0300074030 2024-03-04T07:43:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.769+0000 7f02fe000640 1 -- 172.21.15.116:0/3337943496 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f400f6b0 con 0x7f0300074030 2024-03-04T07:43:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.769+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03001a5bf0 con 0x7f0300074030 2024-03-04T07:43:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.769+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03001a60f0 con 0x7f0300074030 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.772+0000 7f030ac0c740 1 Processor -- start 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.772+0000 7f02fe000640 1 -- 172.21.15.116:0/3337943496 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f02f40109d0 con 0x7f0300074030 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f030ac0c740 1 -- start start 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919eae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919eff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919f0570 0x563c919f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f030ac0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c91950370 con 0x563c919eaa10 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f030ac0c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c918f9550 con 0x563c919eb400 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f030ac0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c9194c410 con 0x563c919f0570 2024-03-04T07:43:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f0305400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f0305e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f0305400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919eff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60080/0 (socket says 172.21.15.116:60080) 2024-03-04T07:43:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.773+0000 7f0305400640 1 -- 172.21.15.116:0/83999855 learned_addr learned my addr 172.21.15.116:0/83999855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0304a00640 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919f0570 0x563c919f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305e00640 1 -- 172.21.15.116:0/83999855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919f0570 msgr2=0x563c919f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305e00640 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919f0570 0x563c919f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305e00640 1 -- 172.21.15.116:0/83999855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 msgr2=0x563c919eff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305e00640 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919eff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305e00640 1 -- 172.21.15.116:0/83999855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c9190d3c0 con 0x563c919eaa10 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305400640 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919eff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f02fe000640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e80a1d70 0x7f02e80a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f02fe000640 1 -- 172.21.15.116:0/3337943496 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f02f40cda30 con 0x7f0300074030 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305e00640 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919eae30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009bf0 tx=0x7f02f80093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c82ae4efcfb2b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.774+0000 7f0305400640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e80a1d70 0x7f02e80a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.775+0000 7f02f2600640 1 -- 172.21.15.116:0/83999855 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8043070 con 0x563c919eaa10 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.775+0000 7f02f2600640 1 -- 172.21.15.116:0/83999855 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f80098c0 con 0x563c919eaa10 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.775+0000 7f02f2600640 1 -- 172.21.15.116:0/83999855 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8039040 con 0x563c919eaa10 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.775+0000 7f030ac0c740 1 -- 172.21.15.116:0/83999855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 msgr2=0x563c919eae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.775+0000 7f030ac0c740 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919eae30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009bf0 tx=0x7f02f80093e0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.775+0000 7f0305400640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e80a1d70 0x7f02e80a4230 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f02f4097150 tx=0x7f02f40d3510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 -- 172.21.15.116:0/83999855 shutdown_connections 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919f0570 0x563c919f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919eff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 --2- 172.21.15.116:0/83999855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919eae30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 -- 172.21.15.116:0/83999855 >> 172.21.15.116:0/83999855 conn(0x563c919e5f80 msgr2=0x563c919e6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 -- 172.21.15.116:0/83999855 shutdown_connections 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 -- 172.21.15.116:0/83999855 wait complete. 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 -- start start 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.776+0000 7f030ac0c740 1 Processor -- start 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eaa10 0x563c919f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919f5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c919fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f030ac0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c919f34c0 con 0x563c919f0570 2024-03-04T07:43:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f030ac0c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c9190c9b0 con 0x563c919eaa10 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f030ac0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c919f3640 con 0x563c919eb400 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f030ac0c740 10 client.0 _ll_register_callbacks cb 0x563c91948340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f0304a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f0304a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919f5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41622/0 (socket says 172.21.15.116:41622) 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f0304a00640 1 -- 172.21.15.116:0/4129904326 learned_addr learned my addr 172.21.15.116:0/4129904326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f0305e00640 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eaa10 0x563c919f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.777+0000 7f0305e00640 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c919fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f0304a00640 1 -- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eaa10 msgr2=0x563c919f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f0304a00640 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eaa10 0x563c919f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f0304a00640 1 -- 172.21.15.116:0/4129904326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 msgr2=0x563c919fc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f0304a00640 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c919fc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f0304a00640 1 -- 172.21.15.116:0/4129904326 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c91665210 con 0x563c919eb400 2024-03-04T07:43:49.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f0304a00640 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919f5220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0129e0 tx=0x7f02ec012ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6517c3fce22a8c6d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f02e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f02e6a00640 1 -- 172.21.15.116:0/4129904326 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec004050 con 0x563c919eb400 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f02e6a00640 1 -- 172.21.15.116:0/4129904326 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f02ec0041f0 con 0x563c919eb400 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.778+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02f80098c0 con 0x563c919eb400 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f02e6a00640 1 -- 172.21.15.116:0/4129904326 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02ec0043f0 con 0x563c919eb400 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 10 client.25963 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563c91731620 con 0x563c919eb400 2024-03-04T07:43:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f02e6a00640 1 -- 172.21.15.116:0/4129904326 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02f80098c0 con 0x563c919eb400 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f02e6a00640 1 -- 172.21.15.116:0/4129904326 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f02ec003c70 con 0x563c919eb400 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 10 client.25963 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 10 client.25963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 20 client.25963 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a00bf0 0x563c91a02fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 4 client.25963 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.779+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c9195d8b0 con 0x563c91a00bf0 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.780+0000 7f0305e00640 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a00bf0 0x563c91a02fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.780+0000 7f02e6a00640 10 client.25963 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.780+0000 7f0305e00640 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a00bf0 0x563c91a02fe0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12f7062f63926d56 server_cookie=9d4612b5d1d6f77a in_seq=0 out_seq=0 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.781+0000 7f02e6a00640 1 -- 172.21.15.116:0/4129904326 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563c9195d8b0 con 0x563c91a00bf0 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.781+0000 7f02e6a00640 10 client.25963 handle_command_reply: tid=0 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.781+0000 7f030ac0c740 1 client.25963 shutdown 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.781+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 msgr2=0x563c919f5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.781+0000 7f030ac0c740 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919f5220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0129e0 tx=0x7f02ec012ea0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 shutdown_connections 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a00bf0 0x563c91a02fe0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c919fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919f5220 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f02e6a00640 0 client.25963 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 --2- 172.21.15.116:0/4129904326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eaa10 0x563c919f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 >> 172.21.15.116:0/4129904326 conn(0x563c919e5f80 msgr2=0x563c919e5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 shutdown_connections 2024-03-04T07:43:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 1 -- 172.21.15.116:0/4129904326 wait complete. 2024-03-04T07:43:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.782+0000 7f030ac0c740 20 client.25963 trim_cache size 0 max 16384 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 Processor -- start 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 -- start start 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eaa10 0x563c919e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919fc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c91a078e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001aceb0 con 0x563c919f0570 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001af490 con 0x563c919eb400 2024-03-04T07:43:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.797+0000 7f030ac0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001ad4b0 con 0x563c919eaa10 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60098/0 (socket says 172.21.15.116:60098) 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 -- 172.21.15.116:0/3945032623 learned_addr learned my addr 172.21.15.116:0/3945032623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0305e00640 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eaa10 0x563c919e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0305e00640 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c91a078e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 -- 172.21.15.116:0/3945032623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eaa10 msgr2=0x563c919e98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eaa10 0x563c919e98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 -- 172.21.15.116:0/3945032623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 msgr2=0x563c91a078e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c91a078e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 -- 172.21.15.116:0/3945032623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c919ebc90 con 0x563c919eb400 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0305e00640 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c91a078e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f0304a00640 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919fc220 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009730 tx=0x7f02ec004290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994e1fef4af1c31 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f02f1c00640 1 -- 172.21.15.116:0/3945032623 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec00f950 con 0x563c919eb400 2024-03-04T07:43:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f02f1c00640 1 -- 172.21.15.116:0/3945032623 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02ec005070 con 0x563c919eb400 2024-03-04T07:43:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.798+0000 7f02f1c00640 1 -- 172.21.15.116:0/3945032623 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec010930 con 0x563c919eb400 2024-03-04T07:43:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.799+0000 7f030ac0c740 1 -- 172.21.15.116:0/3945032623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 msgr2=0x563c919fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.799+0000 7f030ac0c740 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919fc220 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009730 tx=0x7f02ec004290 comp rx=0 tx=0).stop 2024-03-04T07:43:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 -- 172.21.15.116:0/3945032623 shutdown_connections 2024-03-04T07:43:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919f0570 0x563c91a078e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919eb400 0x563c919fc220 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 --2- 172.21.15.116:0/3945032623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eaa10 0x563c919e98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 -- 172.21.15.116:0/3945032623 >> 172.21.15.116:0/3945032623 conn(0x563c919e5f80 msgr2=0x563c919e91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 -- 172.21.15.116:0/3945032623 shutdown_connections 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 -- 172.21.15.116:0/3945032623 wait complete. 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 -- start start 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.800+0000 7f030ac0c740 1 Processor -- start 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919fbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f030ac0c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919f0570 0x563c919fa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f030ac0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001adb10 con 0x563c919eaa10 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f030ac0c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001aee30 con 0x563c919f0570 2024-03-04T07:43:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f030ac0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001ae170 con 0x563c919eb400 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f030ac0c740 10 client.0 _ll_register_callbacks cb 0x7f02ec009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f0305e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f0305e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919fb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58446/0 (socket says 172.21.15.116:58446) 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f0305e00640 1 -- 172.21.15.116:0/176409122 learned_addr learned my addr 172.21.15.116:0/176409122 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f0304a00640 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919fbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.801+0000 7f0305e00640 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919f0570 0x563c919fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f0304a00640 1 -- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919f0570 msgr2=0x563c919fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f0304a00640 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919f0570 0x563c919fa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f0304a00640 1 -- 172.21.15.116:0/176409122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 msgr2=0x563c919fb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f0304a00640 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919fb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f0304a00640 1 -- 172.21.15.116:0/176409122 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c919fa6d0 con 0x563c919eb400 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f0304a00640 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919fbf00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02ec010480 tx=0x7f02ec00fb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4da3760916dbdce server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f02e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:43:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f02e6a00640 1 -- 172.21.15.116:0/176409122 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0106a0 con 0x563c919eb400 2024-03-04T07:43:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f02e6a00640 1 -- 172.21.15.116:0/176409122 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f02ec010840 con 0x563c919eb400 2024-03-04T07:43:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.802+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02ec005070 con 0x563c919eb400 2024-03-04T07:43:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f02e6a00640 1 -- 172.21.15.116:0/176409122 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02ec010a40 con 0x563c919eb400 2024-03-04T07:43:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 10 client.25969 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563c91a5cd30 con 0x563c919eb400 2024-03-04T07:43:49.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f02e6a00640 1 -- 172.21.15.116:0/176409122 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02ec005070 con 0x563c919eb400 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f02e6a00640 1 -- 172.21.15.116:0/176409122 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f02ec030070 con 0x563c919eb400 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 10 client.25969 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 20 client.25969 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a5d620 0x563c91a5fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 4 client.25969 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563c91957c00 con 0x563c91a5d620 2024-03-04T07:43:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.803+0000 7f0305e00640 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a5d620 0x563c91a5fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.804+0000 7f0305e00640 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a5d620 0x563c91a5fa40 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edbe15e7fe2bd310 server_cookie=e6b3dd88ba5fa86 in_seq=0 out_seq=0 2024-03-04T07:43:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.804+0000 7f02e6a00640 10 client.25969 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f02e6a00640 1 -- 172.21.15.116:0/176409122 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563c91957c00 con 0x563c91a5d620 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f02e6a00640 10 client.25969 handle_command_reply: tid=0 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 client.25969 shutdown 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 msgr2=0x563c919fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919fbf00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02ec010480 tx=0x7f02ec00fb80 comp rx=0 tx=0).stop 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 shutdown_connections 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c91a5d620 0x563c91a5fa40 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c919f0570 0x563c919fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c919eb400 0x563c919fbf00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 --2- 172.21.15.116:0/176409122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c919eaa10 0x563c919fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f02e6a00640 0 client.25969 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 >> 172.21.15.116:0/176409122 conn(0x563c919e5f80 msgr2=0x563c919e91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 shutdown_connections 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 1 -- 172.21.15.116:0/176409122 wait complete. 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.818+0000 7f030ac0c740 20 client.25969 trim_cache size 0 max 16384 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:43:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:43:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:43:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e80a1d70 msgr2=0x7f02e80a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e80a1d70 0x7f02e80a4230 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f02f4097150 tx=0x7f02f40d3510 comp rx=0 tx=0).stop 2024-03-04T07:43:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 msgr2=0x7f030019dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 0x7f030019dcf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f02f400d950 tx=0x7f02f400de10 comp rx=0 tx=0).stop 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 shutdown_connections 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f02e80a1d70 0x7f02e80a4230 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f030010da10 0x7f03001a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0300075610 0x7f030019e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 --2- 172.21.15.116:0/3337943496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0300074030 0x7f030019dcf0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 >> 172.21.15.116:0/3337943496 conn(0x7f03000fc9b0 msgr2=0x7f0300100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 shutdown_connections 2024-03-04T07:43:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:43:49.819+0000 7f0306800640 1 -- 172.21.15.116:0/3337943496 wait complete. 2024-03-04T07:43:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 -- 172.21.15.116:0/926731410 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 msgr2=0x7efef8110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 --2- 172.21.15.116:0/926731410 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef8110030 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efef0009900 tx=0x7efef002bc50 comp rx=0 tx=0).stop 2024-03-04T07:43:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 -- 172.21.15.116:0/926731410 shutdown_connections 2024-03-04T07:43:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 --2- 172.21.15.116:0/926731410 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef8110030 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 --2- 172.21.15.116:0/926731410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efef8108de0 0x7efef8109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 --2- 172.21.15.116:0/926731410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef8107bf0 0x7efef8108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 -- 172.21.15.116:0/926731410 >> 172.21.15.116:0/926731410 conn(0x7efef8076b90 msgr2=0x7efef8078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 -- 172.21.15.116:0/926731410 shutdown_connections 2024-03-04T07:43:49.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.949+0000 7efefea00640 1 -- 172.21.15.116:0/926731410 wait complete. 2024-03-04T07:43:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.950+0000 7efefea00640 1 Processor -- start 2024-03-04T07:43:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.950+0000 7efefea00640 1 -- start start 2024-03-04T07:43:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef8107bf0 0x7efef819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efef8108de0 0x7efef819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef8113850 con 0x7efef8108de0 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefea00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81136d0 con 0x7efef81097a0 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefea00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81139d0 con 0x7efef8107bf0 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efeef400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efef8108de0 0x7efef819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efeef400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efef8108de0 0x7efef819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58462/0 (socket says 172.21.15.116:58462) 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef81a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60136/0 (socket says 172.21.15.116:60136) 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efeef400640 1 -- 172.21.15.116:0/664859715 learned_addr learned my addr 172.21.15.116:0/664859715 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efef7e00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef8107bf0 0x7efef819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefca00640 1 -- 172.21.15.116:0/664859715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef8107bf0 msgr2=0x7efef819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefca00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef8107bf0 0x7efef819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefca00640 1 -- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efef8108de0 msgr2=0x7efef819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.951+0000 7efefca00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efef8108de0 0x7efef819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.952+0000 7efefca00640 1 -- 172.21.15.116:0/664859715 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef81a5ac0 con 0x7efef81097a0 2024-03-04T07:43:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.952+0000 7efefca00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef81a5450 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efef002c190 tx=0x7efef00040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.952+0000 7efef5600640 1 -- 172.21.15.116:0/664859715 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0042020 con 0x7efef81097a0 2024-03-04T07:43:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.952+0000 7efef5600640 1 -- 172.21.15.116:0/664859715 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efef00044c0 con 0x7efef81097a0 2024-03-04T07:43:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.952+0000 7efef5600640 1 -- 172.21.15.116:0/664859715 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef003e460 con 0x7efef81097a0 2024-03-04T07:43:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.952+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef81a5d50 con 0x7efef81097a0 2024-03-04T07:43:49.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.952+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef81a61c0 con 0x7efef81097a0 2024-03-04T07:43:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.955+0000 7efef5600640 1 -- 172.21.15.116:0/664859715 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7efef0045020 con 0x7efef81097a0 2024-03-04T07:43:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.955+0000 7eff0177c740 1 Processor -- start 2024-03-04T07:43:49.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7eff0177c740 1 -- start start 2024-03-04T07:43:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7eff0177c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b38034f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7eff0177c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7eff0177c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b380b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7eff0177c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590b3716e20 con 0x5590b3803ac0 2024-03-04T07:43:49.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7eff0177c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590b3768a20 con 0x5590b38030d0 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7eff0177c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590b3711b00 con 0x5590b3808c30 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7efef7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7efef7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58476/0 (socket says 172.21.15.116:58476) 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7efef7e00640 1 -- 172.21.15.116:0/1272404164 learned_addr learned my addr 172.21.15.116:0/1272404164 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7efeef400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b38034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.956+0000 7efeef400640 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b380b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef7e00640 1 -- 172.21.15.116:0/1272404164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 msgr2=0x5590b380b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef7e00640 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b380b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef7e00640 1 -- 172.21.15.116:0/1272404164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 msgr2=0x5590b38034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef7e00640 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b38034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef7e00640 1 -- 172.21.15.116:0/1272404164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590b3726e10 con 0x5590b3803ac0 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef5600640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efedc0a1d80 0x7efedc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef7e00640 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38085c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efee8010420 tx=0x7efee80108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392edd0f02ab514a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efef5600640 1 -- 172.21.15.116:0/664859715 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7efef00efee0 con 0x7efef81097a0 2024-03-04T07:43:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efeef400640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efedc0a1d80 0x7efedc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efeed200640 1 -- 172.21.15.116:0/1272404164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8016070 con 0x5590b3803ac0 2024-03-04T07:43:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efeed200640 1 -- 172.21.15.116:0/1272404164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efee8012bf0 con 0x5590b3803ac0 2024-03-04T07:43:49.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.957+0000 7efeed200640 1 -- 172.21.15.116:0/1272404164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee801bd90 con 0x5590b3803ac0 2024-03-04T07:43:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 -- 172.21.15.116:0/1272404164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 msgr2=0x5590b38085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7efeef400640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efedc0a1d80 0x7efedc0a4240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5590b3804320 tx=0x7efee0013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38085c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efee8010420 tx=0x7efee80108e0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 -- 172.21.15.116:0/1272404164 shutdown_connections 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b380b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38085c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 --2- 172.21.15.116:0/1272404164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b38034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 -- 172.21.15.116:0/1272404164 >> 172.21.15.116:0/1272404164 conn(0x5590b37fe640 msgr2=0x5590b37fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 -- 172.21.15.116:0/1272404164 shutdown_connections 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.958+0000 7eff0177c740 1 -- 172.21.15.116:0/1272404164 wait complete. 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 -- start start 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 Processor -- start 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b380d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3803ac0 0x5590b380d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3808c30 0x5590b38149f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590b3724f40 con 0x5590b38030d0 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590b3764ac0 con 0x5590b3808c30 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590b380bbe0 con 0x5590b3803ac0 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.959+0000 7eff0177c740 10 client.0 _ll_register_callbacks cb 0x5590b3763aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efef7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b380d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3808c30 0x5590b38149f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3808c30 0x5590b38149f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60150/0 (socket says 172.21.15.116:60150) 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 -- 172.21.15.116:0/2125715376 learned_addr learned my addr 172.21.15.116:0/2125715376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efeef400640 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3803ac0 0x5590b380d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 -- 172.21.15.116:0/2125715376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3803ac0 msgr2=0x5590b380d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3803ac0 0x5590b380d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 -- 172.21.15.116:0/2125715376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 msgr2=0x5590b380d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b380d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 -- 172.21.15.116:0/2125715376 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590b35457e0 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efef7e00640 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b380d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efefca00640 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3808c30 0x5590b38149f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efef00b95a0 tx=0x7efef00bc5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b56bce3a9d9941 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efee5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.960+0000 7efee5600640 1 -- 172.21.15.116:0/2125715376 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef00d5060 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efee8012bf0 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7efee5600640 1 -- 172.21.15.116:0/2125715376 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7efef0004c80 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7efee5600640 1 -- 172.21.15.116:0/2125715376 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efef00ede60 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 10 client.26033 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5590b35a5f70 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7efee5600640 1 -- 172.21.15.116:0/2125715376 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efee8012bf0 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7efee5600640 1 -- 172.21.15.116:0/2125715376 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efef00c4ee0 con 0x5590b3808c30 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 10 client.26033 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 10 client.26033 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:49.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 20 client.26033 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b38192f0 0x5590b381b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 4 client.26033 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.961+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590b3775f60 con 0x5590b38192f0 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.962+0000 7efef7e00640 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b38192f0 0x5590b381b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.962+0000 7efef7e00640 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b38192f0 0x5590b381b790 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e866b4ae3c2904d1 server_cookie=a7795819812b3cd9 in_seq=0 out_seq=0 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.962+0000 7efee5600640 10 client.26033 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7efee5600640 1 -- 172.21.15.116:0/2125715376 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5590b3775f60 con 0x5590b38192f0 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7efee5600640 10 client.26033 handle_command_reply: tid=0 2024-03-04T07:43:49.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 client.26033 shutdown 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3808c30 msgr2=0x5590b38149f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3808c30 0x5590b38149f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efef00b95a0 tx=0x7efef00bc5f0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 shutdown_connections 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b38192f0 0x5590b381b790 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3808c30 0x5590b38149f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3803ac0 0x5590b380d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 --2- 172.21.15.116:0/2125715376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b380d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7efee5600640 0 client.26033 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 >> 172.21.15.116:0/2125715376 conn(0x5590b37fe640 msgr2=0x5590b37ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 shutdown_connections 2024-03-04T07:43:49.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.963+0000 7eff0177c740 1 -- 172.21.15.116:0/2125715376 wait complete. 2024-03-04T07:43:49.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.964+0000 7eff0177c740 20 client.26033 trim_cache size 0 max 16384 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 Processor -- start 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 -- start start 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b37fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38149b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b37fd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81acf30 con 0x5590b3803ac0 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81af510 con 0x5590b38030d0 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7eff0177c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81ad530 con 0x5590b3808c30 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7efeef400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7efef7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b37fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7efef7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b37fd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60160/0 (socket says 172.21.15.116:60160) 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.981+0000 7efef7e00640 1 -- 172.21.15.116:0/45876663 learned_addr learned my addr 172.21.15.116:0/45876663 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efefca00640 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b37fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efef7e00640 1 -- 172.21.15.116:0/45876663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 msgr2=0x5590b37fd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efef7e00640 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b37fd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efef7e00640 1 -- 172.21.15.116:0/45876663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 msgr2=0x5590b38149b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efef7e00640 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38149b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efef7e00640 1 -- 172.21.15.116:0/45876663 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590b343d170 con 0x5590b38030d0 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efefca00640 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b37fd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efef7e00640 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b37fd1d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efee8010de0 tx=0x7efee8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d735f0c74fd11d69 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efee7e00640 1 -- 172.21.15.116:0/45876663 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee802f070 con 0x5590b38030d0 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efee7e00640 1 -- 172.21.15.116:0/45876663 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efee801cba0 con 0x5590b38030d0 2024-03-04T07:43:49.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.982+0000 7efee7e00640 1 -- 172.21.15.116:0/45876663 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee802a070 con 0x5590b38030d0 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 -- 172.21.15.116:0/45876663 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 msgr2=0x5590b37fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b37fd1d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efee8010de0 tx=0x7efee8005fd0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 -- 172.21.15.116:0/45876663 shutdown_connections 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b37fd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b3803ac0 0x5590b38149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 --2- 172.21.15.116:0/45876663 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b38030d0 0x5590b37fd1d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 -- 172.21.15.116:0/45876663 >> 172.21.15.116:0/45876663 conn(0x5590b37fe640 msgr2=0x5590b37fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 -- 172.21.15.116:0/45876663 shutdown_connections 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 -- 172.21.15.116:0/45876663 wait complete. 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 -- start start 2024-03-04T07:43:49.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.983+0000 7eff0177c740 1 Processor -- start 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7eff0177c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b3814150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7eff0177c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3803ac0 0x5590b3814690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7eff0177c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b3812760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7eff0177c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81adb90 con 0x5590b38030d0 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7eff0177c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81aeeb0 con 0x5590b3803ac0 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7eff0177c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81ae1f0 con 0x5590b3808c30 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7eff0177c740 10 client.0 _ll_register_callbacks cb 0x7efef00035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7efef7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b3814150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7efeef400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3803ac0 0x5590b3814690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7efeef400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3803ac0 0x5590b3814690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60172/0 (socket says 172.21.15.116:60172) 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7efeef400640 1 -- 172.21.15.116:0/4089837024 learned_addr learned my addr 172.21.15.116:0/4089837024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7efef7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b3814150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58504/0 (socket says 172.21.15.116:58504) 2024-03-04T07:43:49.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.984+0000 7efefca00640 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b3812760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efeef400640 1 -- 172.21.15.116:0/4089837024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 msgr2=0x5590b3812760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efeef400640 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b3812760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efeef400640 1 -- 172.21.15.116:0/4089837024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 msgr2=0x5590b3814150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efeef400640 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b3814150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efeef400640 1 -- 172.21.15.116:0/4089837024 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590b3812da0 con 0x5590b3803ac0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efef7e00640 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b3814150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efeef400640 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3803ac0 0x5590b3814690 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efee0021db0 tx=0x7efee0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6b5e6a1659c17ca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efee5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efee5600640 1 -- 172.21.15.116:0/4089837024 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0005020 con 0x5590b3803ac0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efee801cba0 con 0x5590b3803ac0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efee5600640 1 -- 172.21.15.116:0/4089837024 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7efee00051c0 con 0x5590b3803ac0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7efee5600640 1 -- 172.21.15.116:0/4089837024 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7efee00053c0 con 0x5590b3803ac0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7eff0177c740 10 client.26039 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.985+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5590b38755c0 con 0x5590b3803ac0 2024-03-04T07:43:49.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7efee5600640 1 -- 172.21.15.116:0/4089837024 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efee801cba0 con 0x5590b3803ac0 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7efee5600640 1 -- 172.21.15.116:0/4089837024 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7efee0029070 con 0x5590b3803ac0 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7eff0177c740 10 client.26039 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7eff0177c740 10 client.26039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7eff0177c740 20 client.26039 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7eff0177c740 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b3875f80 0x5590b38783a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7eff0177c740 4 client.26039 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5590b37702b0 con 0x5590b3875f80 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.986+0000 7efef7e00640 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b3875f80 0x5590b38783a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.987+0000 7efee5600640 10 client.26039 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.987+0000 7efef7e00640 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b3875f80 0x5590b38783a0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1903f07e3f349b61 server_cookie=d67ad8d1dc798f32 in_seq=0 out_seq=0 2024-03-04T07:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.987+0000 7efee5600640 1 -- 172.21.15.116:0/4089837024 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5590b37702b0 con 0x5590b3875f80 2024-03-04T07:43:49.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.987+0000 7efee5600640 10 client.26039 handle_command_reply: tid=0 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.987+0000 7eff0177c740 1 client.26039 shutdown 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3803ac0 msgr2=0x5590b3814690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3803ac0 0x5590b3814690 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efee0021db0 tx=0x7efee0004730 comp rx=0 tx=0).stop 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 shutdown_connections 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5590b3875f80 0x5590b38783a0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5590b3808c30 0x5590b3812760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7efee5600640 0 client.26039 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5590b3803ac0 0x5590b3814690 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 --2- 172.21.15.116:0/4089837024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590b38030d0 0x5590b3814150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 >> 172.21.15.116:0/4089837024 conn(0x5590b37fe640 msgr2=0x5590b37fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 shutdown_connections 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 1 -- 172.21.15.116:0/4089837024 wait complete. 2024-03-04T07:43:49.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.988+0000 7eff0177c740 20 client.26039 trim_cache size 0 max 16384 2024-03-04T07:43:49.993 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:43:49.993 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:43:49.993 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "1f491a6b-1a4f-4657-b161-72166a9d2bbe", 2024-03-04T07:43:49.993 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:43:49.993 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efedc0a1d80 msgr2=0x7efedc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efedc0a1d80 0x7efedc0a4240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5590b3804320 tx=0x7efee0013040 comp rx=0 tx=0).stop 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 msgr2=0x7efef81a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef81a5450 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efef002c190 tx=0x7efef00040c0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 shutdown_connections 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efef81097a0 0x7efef81a5450 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efef8108de0 0x7efef819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7efedc0a1d80 0x7efedc0a4240 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 --2- 172.21.15.116:0/664859715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef8107bf0 0x7efef819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 >> 172.21.15.116:0/664859715 conn(0x7efef8076b90 msgr2=0x7efef810c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 shutdown_connections 2024-03-04T07:43:49.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:49.989+0000 7efefea00640 1 -- 172.21.15.116:0/664859715 wait complete. 2024-03-04T07:43:50.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:43:50.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 -- 172.21.15.116:0/2691688280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 msgr2=0x7fcf98108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 --2- 172.21.15.116:0/2691688280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf98108010 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009900 tx=0x7fcf8802bc30 comp rx=0 tx=0).stop 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 -- 172.21.15.116:0/2691688280 shutdown_connections 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 --2- 172.21.15.116:0/2691688280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf981097a0 0x7fcf98110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 --2- 172.21.15.116:0/2691688280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf98108de0 0x7fcf98109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 --2- 172.21.15.116:0/2691688280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf98108010 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 -- 172.21.15.116:0/2691688280 >> 172.21.15.116:0/2691688280 conn(0x7fcf98076b90 msgr2=0x7fcf98078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 -- 172.21.15.116:0/2691688280 shutdown_connections 2024-03-04T07:43:50.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.267+0000 7fcf9d600640 1 -- 172.21.15.116:0/2691688280 wait complete. 2024-03-04T07:43:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 Processor -- start 2024-03-04T07:43:50.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 -- start start 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf9819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf98108de0 0x7fcf9819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf981097a0 0x7fcf981a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981138b0 con 0x7fcf981097a0 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98113730 con 0x7fcf98107bf0 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf9d600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98113a30 con 0x7fcf98108de0 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf9819df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf9819df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60178/0 (socket says 172.21.15.116:60178) 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf981097a0 0x7fcf981a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf96a00640 1 -- 172.21.15.116:0/1699043639 learned_addr learned my addr 172.21.15.116:0/1699043639 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.268+0000 7fcf8e000640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf98108de0 0x7fcf9819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf96a00640 1 -- 172.21.15.116:0/1699043639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf98108de0 msgr2=0x7fcf9819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf96a00640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf98108de0 0x7fcf9819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf96a00640 1 -- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf981097a0 msgr2=0x7fcf981a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf96a00640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf981097a0 0x7fcf981a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf96a00640 1 -- 172.21.15.116:0/1699043639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf981a5b40 con 0x7fcf98107bf0 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf96a00640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf9819df20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcf88000c00 tx=0x7fcf8802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf8fe00640 1 -- 172.21.15.116:0/1699043639 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88043020 con 0x7fcf98107bf0 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf8fe00640 1 -- 172.21.15.116:0/1699043639 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf880054e0 con 0x7fcf98107bf0 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.269+0000 7fcf8fe00640 1 -- 172.21.15.116:0/1699043639 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88003c40 con 0x7fcf98107bf0 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.270+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf981a5dd0 con 0x7fcf98107bf0 2024-03-04T07:43:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.270+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf981a61e0 con 0x7fcf98107bf0 2024-03-04T07:43:50.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 Processor -- start 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcf8fe00640 1 -- 172.21.15.116:0/1699043639 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fcf88046020 con 0x7fcf98107bf0 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 -- start start 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516d4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6515fc090 con 0x55c6516ceb50 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c651633d30 con 0x55c6516d46b0 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6515dca30 con 0x55c6516cf540 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcf97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.273+0000 7fcf8e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf97400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58530/0 (socket says 172.21.15.116:58530) 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516d4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf97400640 1 -- 172.21.15.116:0/1385628631 learned_addr learned my addr 172.21.15.116:0/1385628631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8e000640 1 -- 172.21.15.116:0/1385628631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 msgr2=0x55c6516d4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8e000640 1 --2- 172.21.15.116:0/1385628631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516d4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8e000640 1 -- 172.21.15.116:0/1385628631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 msgr2=0x55c6516cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8e000640 1 --2- 172.21.15.116:0/1385628631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8e000640 1 -- 172.21.15.116:0/1385628631 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6514127d0 con 0x55c6516d46b0 2024-03-04T07:43:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8e000640 1 --2- 172.21.15.116:0/1385628631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d6b40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009bd0 tx=0x7fcf80011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e651a25d59b24444 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:50.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8fe00640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf7c0a1e00 0x7fcf7c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf8fe00640 1 -- 172.21.15.116:0/1699043639 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcf880efce0 con 0x7fcf98107bf0 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf87400640 1 -- 172.21.15.116:0/1385628631 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80004230 con 0x55c6516d46b0 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf87400640 1 -- 172.21.15.116:0/1385628631 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf80009e40 con 0x55c6516d46b0 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.274+0000 7fcf87400640 1 -- 172.21.15.116:0/1385628631 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80004db0 con 0x55c6516d46b0 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcf97400640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf7c0a1e00 0x7fcf7c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcfa01f4740 1 -- 172.21.15.116:0/1385628631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 msgr2=0x55c6516d6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/1385628631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d6b40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009bd0 tx=0x7fcf80011810 comp rx=0 tx=0).stop 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcf97400640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf7c0a1e00 0x7fcf7c0a42c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55c6516cfda0 tx=0x7fcf90009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcfa01f4740 1 -- 172.21.15.116:0/1385628631 shutdown_connections 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/1385628631 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d6b40 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/1385628631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516d4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/1385628631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.275+0000 7fcfa01f4740 1 -- 172.21.15.116:0/1385628631 >> 172.21.15.116:0/1385628631 conn(0x55c6516ca0c0 msgr2=0x55c6516ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:50.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.276+0000 7fcfa01f4740 1 -- 172.21.15.116:0/1385628631 shutdown_connections 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.276+0000 7fcfa01f4740 1 -- 172.21.15.116:0/1385628631 wait complete. 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.276+0000 7fcfa01f4740 1 -- start start 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.276+0000 7fcfa01f4740 1 Processor -- start 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.276+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516ceb50 0x55c6516d8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516cf540 0x55c6516d9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516d46b0 0x55c6516e0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6515d20c0 con 0x55c6516cf540 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c65162fdd0 con 0x55c6516ceb50 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6516d7660 con 0x55c6516d46b0 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcfa01f4740 10 client.0 _ll_register_callbacks cb 0x55c6515ef860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516ceb50 0x55c6516d8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf8e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516cf540 0x55c6516d9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516ceb50 0x55c6516d8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60198/0 (socket says 172.21.15.116:60198) 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 -- 172.21.15.116:0/888655548 learned_addr learned my addr 172.21.15.116:0/888655548 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf8e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516cf540 0x55c6516d9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58538/0 (socket says 172.21.15.116:58538) 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf97400640 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516d46b0 0x55c6516e0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 -- 172.21.15.116:0/888655548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516d46b0 msgr2=0x55c6516e0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516d46b0 0x55c6516e0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 -- 172.21.15.116:0/888655548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516cf540 msgr2=0x55c6516d9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516cf540 0x55c6516d9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 -- 172.21.15.116:0/888655548 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6513483f0 con 0x55c6516ceb50 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf8e000640 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516cf540 0x55c6516d9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf96a00640 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516ceb50 0x55c6516d8f10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcf8802ceb0 tx=0x7fcf880b8cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd5cc6b256822a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.277+0000 7fcf7be00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:50.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcf7be00640 1 -- 172.21.15.116:0/888655548 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88004250 con 0x55c6516ceb50 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcf7be00640 1 -- 172.21.15.116:0/888655548 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcf880ed8f0 con 0x55c6516ceb50 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c651641270 con 0x55c6516ceb50 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcf7be00640 1 -- 172.21.15.116:0/888655548 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf880035f0 con 0x55c6516ceb50 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 10 client.26051 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c6513080f0 con 0x55c6516ceb50 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcf7be00640 1 -- 172.21.15.116:0/888655548 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c651641270 con 0x55c6516ceb50 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcf7be00640 1 -- 172.21.15.116:0/888655548 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcf880bcc70 con 0x55c6516ceb50 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 10 client.26051 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 10 client.26051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 20 client.26051 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c6516e4a20 0x55c6516e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 4 client.26051 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:43:50.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.278+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c65163b5c0 con 0x55c6516e4a20 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.279+0000 7fcf8e000640 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c6516e4a20 0x55c6516e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.279+0000 7fcf8e000640 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c6516e4a20 0x55c6516e6e70 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3acbeef6da3c3802 server_cookie=5b932a21471ab6b7 in_seq=0 out_seq=0 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.279+0000 7fcf7be00640 10 client.26051 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcf7be00640 1 -- 172.21.15.116:0/888655548 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c65163b5c0 con 0x55c6516e4a20 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcf7be00640 10 client.26051 handle_command_reply: tid=0 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 client.26051 shutdown 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516ceb50 msgr2=0x55c6516d8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516ceb50 0x55c6516d8f10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcf8802ceb0 tx=0x7fcf880b8cb0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 shutdown_connections 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c6516e4a20 0x55c6516e6e70 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516d46b0 0x55c6516e0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516cf540 0x55c6516d9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcf7be00640 0 client.26051 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:50.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/888655548 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516ceb50 0x55c6516d8f10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 >> 172.21.15.116:0/888655548 conn(0x55c6516ca0c0 msgr2=0x55c6516c9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 shutdown_connections 2024-03-04T07:43:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 1 -- 172.21.15.116:0/888655548 wait complete. 2024-03-04T07:43:50.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.280+0000 7fcfa01f4740 20 client.26051 trim_cache size 0 max 16384 2024-03-04T07:43:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 Processor -- start 2024-03-04T07:43:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 -- start start 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516d8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981acfd0 con 0x55c6516ceb50 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981af5b0 con 0x55c6516d46b0 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.298+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981ad5d0 con 0x55c6516cf540 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516d8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf97400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516d8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58548/0 (socket says 172.21.15.116:58548) 2024-03-04T07:43:50.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf96a00640 1 -- 172.21.15.116:0/2958602311 learned_addr learned my addr 172.21.15.116:0/2958602311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:50.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf8e000640 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf97400640 1 -- 172.21.15.116:0/2958602311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 msgr2=0x55c6516e0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf97400640 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf97400640 1 -- 172.21.15.116:0/2958602311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 msgr2=0x55c6516d8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf97400640 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516d8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf97400640 1 -- 172.21.15.116:0/2958602311 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6516d3ce0 con 0x55c6516d46b0 2024-03-04T07:43:50.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf8e000640 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.299+0000 7fcf96a00640 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516d8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcf97400640 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d8a70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcf90005fd0 tx=0x7fcf90016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7d2c07fbb7092df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcf86a00640 1 -- 172.21.15.116:0/2958602311 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90010430 con 0x55c6516d46b0 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcf86a00640 1 -- 172.21.15.116:0/2958602311 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf90010ab0 con 0x55c6516d46b0 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcf86a00640 1 -- 172.21.15.116:0/2958602311 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9001e9b0 con 0x55c6516d46b0 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcfa01f4740 1 -- 172.21.15.116:0/2958602311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 msgr2=0x55c6516d8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d8a70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcf90005fd0 tx=0x7fcf90016000 comp rx=0 tx=0).stop 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcfa01f4740 1 -- 172.21.15.116:0/2958602311 shutdown_connections 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516d8a70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/2958602311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516d8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.300+0000 7fcfa01f4740 1 -- 172.21.15.116:0/2958602311 >> 172.21.15.116:0/2958602311 conn(0x55c6516ca0c0 msgr2=0x55c6516cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:50.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.301+0000 7fcfa01f4740 1 -- 172.21.15.116:0/2958602311 shutdown_connections 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.301+0000 7fcfa01f4740 1 -- 172.21.15.116:0/2958602311 wait complete. 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.301+0000 7fcfa01f4740 1 -- start start 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.301+0000 7fcfa01f4740 1 Processor -- start 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.301+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516dfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcfa01f4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516de300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981adc30 con 0x55c6516ceb50 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981aef50 con 0x55c6516d46b0 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcfa01f4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981ae290 con 0x55c6516cf540 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcfa01f4740 10 client.0 _ll_register_callbacks cb 0x7fcf880f3450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516dfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516de300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516de300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60212/0 (socket says 172.21.15.116:60212) 2024-03-04T07:43:50.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf96a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516dfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58558/0 (socket says 172.21.15.116:58558) 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 -- 172.21.15.116:0/393652941 learned_addr learned my addr 172.21.15.116:0/393652941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf8e000640 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 -- 172.21.15.116:0/393652941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 msgr2=0x55c6516e01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 -- 172.21.15.116:0/393652941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 msgr2=0x55c6516dfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516dfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 -- 172.21.15.116:0/393652941 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6516de970 con 0x55c6516d46b0 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf97400640 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516de300 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcf90018040 tx=0x7fcf9001e5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92f752c60adba187 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.302+0000 7fcf7be00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcf7be00640 1 -- 172.21.15.116:0/393652941 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90004d10 con 0x55c6516d46b0 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcf7be00640 1 -- 172.21.15.116:0/393652941 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcf90010430 con 0x55c6516d46b0 2024-03-04T07:43:50.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf90010ab0 con 0x55c6516d46b0 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcf7be00640 1 -- 172.21.15.116:0/393652941 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf90010630 con 0x55c6516d46b0 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 10 client.26057 fetch_fsmap learned FSMap version 19 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c651740cb0 con 0x55c6516d46b0 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcf7be00640 1 -- 172.21.15.116:0/393652941 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf90010ab0 con 0x55c6516d46b0 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcf7be00640 1 -- 172.21.15.116:0/393652941 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcf90003cb0 con 0x55c6516d46b0 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 10 client.26057 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 10 client.26057 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 20 client.26057 populate_metadata read hostname 'smithi116' 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c651741710 0x55c651743b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 4 client.26057 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.303+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c6515b93a0 con 0x55c651741710 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcf96a00640 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c651741710 0x55c651743b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcf96a00640 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c651741710 0x55c651743b00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4719b5c8b57574fc server_cookie=34a0d9c085ed2c6d in_seq=0 out_seq=0 2024-03-04T07:43:50.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcf7be00640 10 client.26057 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcf7be00640 1 -- 172.21.15.116:0/393652941 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c6515b93a0 con 0x55c651741710 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcf7be00640 10 client.26057 handle_command_reply: tid=0 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcfa01f4740 1 client.26057 shutdown 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 msgr2=0x55c6516de300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.304+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516de300 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcf90018040 tx=0x7fcf9001e5d0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 shutdown_connections 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c651741710 0x55c651743b00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c6516d46b0 0x55c6516de300 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6516cf540 0x55c6516e01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf7be00640 0 client.26057 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 --2- 172.21.15.116:0/393652941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c6516ceb50 0x55c6516dfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 >> 172.21.15.116:0/393652941 conn(0x55c6516ca0c0 msgr2=0x55c6516c8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 shutdown_connections 2024-03-04T07:43:50.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 1 -- 172.21.15.116:0/393652941 wait complete. 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcfa01f4740 20 client.26057 trim_cache size 0 max 16384 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "1f491a6b-1a4f-4657-b161-72166a9d2bbe": { 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "1f491a6b-1a4f-4657-b161-72166a9d2bbe", 2024-03-04T07:43:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:43:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:43:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:43:50.312 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:43:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf7c0a1e00 msgr2=0x7fcf7c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf7c0a1e00 0x7fcf7c0a42c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55c6516cfda0 tx=0x7fcf90009500 comp rx=0 tx=0).stop 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 msgr2=0x7fcf9819df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf9819df20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcf88000c00 tx=0x7fcf8802c650 comp rx=0 tx=0).stop 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 shutdown_connections 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf981097a0 0x7fcf981a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf7c0a1e00 0x7fcf7c0a42c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf98108de0 0x7fcf9819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 --2- 172.21.15.116:0/1699043639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcf98107bf0 0x7fcf9819df20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.305+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 >> 172.21.15.116:0/1699043639 conn(0x7fcf98076b90 msgr2=0x7fcf9810c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.306+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 shutdown_connections 2024-03-04T07:43:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:43:50.306+0000 7fcf9d600640 1 -- 172.21.15.116:0/1699043639 wait complete. 2024-03-04T07:43:50.324 INFO:tasks.cephfs.filesystem:scrub status for tag:1f491a6b-1a4f-4657-b161-72166a9d2bbe - {'path': '/', 'tag': '1f491a6b-1a4f-4657-b161-72166a9d2bbe', 'options': 'recursive,force'} 2024-03-04T07:43:51.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:50 smithi116 ceph-mon[32150]: pgmap v493: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:43:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:50 smithi152 ceph-mon[37154]: pgmap v493: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:43:51.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:50 smithi133 ceph-mon[36505]: pgmap v493: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:43:51.988 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:51 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:43:51.988 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:51 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:51.988 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:51 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:43:51.988 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:51 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:52.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:51 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:43:52.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:51 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:52.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:51 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:43:52.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:51 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:52.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:51 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:43:52.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:51 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:52.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:51 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:43:52.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:51 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:43:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:52 smithi116 ceph-mon[32150]: pgmap v494: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:43:53.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:52 smithi152 ceph-mon[37154]: pgmap v494: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:43:53.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:52 smithi133 ceph-mon[36505]: pgmap v494: 97 pgs: 97 active+clean; 2.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:43:54.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:53 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:43:54.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:53 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:43:54.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:53 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:43:55.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:54 smithi116 ceph-mon[32150]: pgmap v495: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:55.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:54 smithi152 ceph-mon[37154]: pgmap v495: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:55.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:54 smithi133 ceph-mon[36505]: pgmap v495: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:43:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:56 smithi116 ceph-mon[32150]: pgmap v496: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:43:57.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:56 smithi152 ceph-mon[37154]: pgmap v496: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:43:57.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:56 smithi133 ceph-mon[36505]: pgmap v496: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:43:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:43:58 smithi152 ceph-mon[37154]: pgmap v497: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:43:59.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:43:58 smithi133 ceph-mon[36505]: pgmap v497: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:43:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:43:58 smithi116 ceph-mon[32150]: pgmap v497: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:44:01.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:00 smithi152 ceph-mon[37154]: pgmap v498: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:44:01.061 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:00 smithi116 ceph-mon[32150]: pgmap v498: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:44:01.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:00 smithi133 ceph-mon[36505]: pgmap v498: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:44:02.721 INFO:tasks.workunit.client.0.smithi116.stdout:64/600: open fd = 3 2024-03-04T07:44:02.721 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.721 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.721 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.721 INFO:tasks.workunit.client.0.smithi116.stdout:65/600: open fd = 3 2024-03-04T07:44:02.721 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.721 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:66/600: open fd = 3 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:67/600: open fd = 3 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:68/600: open fd = 3 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.722 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:69/600: open fd = 3 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:70/600: open fd = 3 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:71/600: open fd = 3 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.723 INFO:tasks.workunit.client.0.smithi116.stdout:72/600: open fd = 3 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:73/600: open fd = 3 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:74/600: open fd = 3 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:75/600: open fd = 3 2024-03-04T07:44:02.724 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:76/600: open fd = 3 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:77/600: open fd = 3 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:78/600: open fd = 3 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.725 INFO:tasks.workunit.client.0.smithi116.stdout:79/600: open fd = 3 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:80/600: open fd = 3 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:81/600: open fd = 3 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:82/600: open fd = 3 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.726 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:83/600: open fd = 3 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:84/600: open fd = 3 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:85/600: open fd = 3 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:86/600: open fd = 3 2024-03-04T07:44:02.727 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:87/600: open fd = 3 2024-03-04T07:44:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.728 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:88/600: open fd = 3 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:89/600: open fd = 3 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.729 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:90/600: open fd = 3 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:91/600: open fd = 3 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.730 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:92/600: open fd = 3 2024-03-04T07:44:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:93/600: open fd = 3 2024-03-04T07:44:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.732 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.732 INFO:tasks.workunit.client.0.smithi116.stdout:94/600: open fd = 3 2024-03-04T07:44:02.732 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.732 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.732 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.732 INFO:tasks.workunit.client.0.smithi116.stdout:95/600: open fd = 3 2024-03-04T07:44:02.733 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.733 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.733 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.733 INFO:tasks.workunit.client.0.smithi116.stdout:96/600: open fd = 3 2024-03-04T07:44:02.733 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.733 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.734 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.734 INFO:tasks.workunit.client.0.smithi116.stdout:97/600: open fd = 3 2024-03-04T07:44:02.734 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.734 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.734 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.734 INFO:tasks.workunit.client.0.smithi116.stdout:98/600: open fd = 3 2024-03-04T07:44:02.734 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.735 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.735 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.735 INFO:tasks.workunit.client.0.smithi116.stdout:99/600: open fd = 3 2024-03-04T07:44:02.735 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.735 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.735 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.735 INFO:tasks.workunit.client.0.smithi116.stdout:100/600: open fd = 3 2024-03-04T07:44:02.736 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.736 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.736 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.736 INFO:tasks.workunit.client.0.smithi116.stdout:101/600: open fd = 3 2024-03-04T07:44:02.736 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.736 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.736 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.737 INFO:tasks.workunit.client.0.smithi116.stdout:102/600: open fd = 3 2024-03-04T07:44:02.737 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.737 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.737 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.737 INFO:tasks.workunit.client.0.smithi116.stdout:103/600: open fd = 3 2024-03-04T07:44:02.738 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.738 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.738 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.738 INFO:tasks.workunit.client.0.smithi116.stdout:104/600: open fd = 3 2024-03-04T07:44:02.738 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.738 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:105/600: open fd = 3 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:106/600: open fd = 3 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.739 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:107/600: open fd = 3 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:108/600: open fd = 3 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:109/600: open fd = 3 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.740 INFO:tasks.workunit.client.0.smithi116.stdout:110/600: open fd = 3 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:111/600: open fd = 3 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:112/600: open fd = 3 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:113/600: open fd = 3 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.741 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:114/600: open fd = 3 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:115/600: open fd = 3 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:116/600: open fd = 3 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:117/600: open fd = 3 2024-03-04T07:44:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:118/600: open fd = 3 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:119/600: open fd = 3 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:120/600: open fd = 3 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.743 INFO:tasks.workunit.client.0.smithi116.stdout:121/600: open fd = 3 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:122/600: open fd = 3 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:123/600: open fd = 3 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.744 INFO:tasks.workunit.client.0.smithi116.stdout:124/600: open fd = 3 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:125/600: open fd = 3 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:126/600: open fd = 3 2024-03-04T07:44:02.745 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.957+0000 7f4639c00640 1 -- 172.21.15.116:0/3757103989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 msgr2=0x7f4634110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.957+0000 7f4639c00640 1 --2- 172.21.15.116:0/3757103989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 0x7f4634110070 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f462c009ca0 tx=0x7f462c02c100 comp rx=0 tx=0).stop 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.957+0000 7f4639c00640 1 -- 172.21.15.116:0/3757103989 shutdown_connections 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.957+0000 7f4639c00640 1 --2- 172.21.15.116:0/3757103989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 0x7f4634110070 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.957+0000 7f4639c00640 1 --2- 172.21.15.116:0/3757103989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4634108e20 0x7f46341092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.957+0000 7f4639c00640 1 --2- 172.21.15.116:0/3757103989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4634107c30 0x7f4634108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.957+0000 7f4639c00640 1 -- 172.21.15.116:0/3757103989 >> 172.21.15.116:0/3757103989 conn(0x7f4634076be0 msgr2=0x7f4634079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.958+0000 7f4639c00640 1 -- 172.21.15.116:0/3757103989 shutdown_connections 2024-03-04T07:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.958+0000 7f4639c00640 1 -- 172.21.15.116:0/3757103989 wait complete. 2024-03-04T07:44:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.958+0000 7f4639c00640 1 Processor -- start 2024-03-04T07:44:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4639c00640 1 -- start start 2024-03-04T07:44:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4639c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4634107c30 0x7f463419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4639c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4634108e20 0x7f463419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4639c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 0x7f46341a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4639c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341138b0 con 0x7f4634107c30 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4639c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4634113730 con 0x7f46341097e0 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4639c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4634113a30 con 0x7f4634108e20 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4632a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4634107c30 0x7f463419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4633400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 0x7f46341a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4632a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4634107c30 0x7f463419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38642/0 (socket says 172.21.15.116:38642) 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f4632a00640 1 -- 172.21.15.116:0/3901829690 learned_addr learned my addr 172.21.15.116:0/3901829690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.959+0000 7f462a000640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4634108e20 0x7f463419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4633400640 1 -- 172.21.15.116:0/3901829690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4634108e20 msgr2=0x7f463419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4633400640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4634108e20 0x7f463419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4633400640 1 -- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4634107c30 msgr2=0x7f463419de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4633400640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4634107c30 0x7f463419de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4633400640 1 -- 172.21.15.116:0/3901829690 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46341a5a60 con 0x7f46341097e0 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4633400640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 0x7f46341a53f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f462c037040 tx=0x7f462c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f462be00640 1 -- 172.21.15.116:0/3901829690 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c039040 con 0x7f46341097e0 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f462be00640 1 -- 172.21.15.116:0/3901829690 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f462c044070 con 0x7f46341097e0 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46341a5cf0 con 0x7f46341097e0 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f462be00640 1 -- 172.21.15.116:0/3901829690 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c004450 con 0x7f46341097e0 2024-03-04T07:44:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.960+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46341a61b0 con 0x7f46341097e0 2024-03-04T07:44:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.963+0000 7f463c9d8740 1 Processor -- start 2024-03-04T07:44:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f462be00640 1 -- 172.21.15.116:0/3901829690 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f462c0045f0 con 0x7f46341097e0 2024-03-04T07:44:02.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f463c9d8740 1 -- start start 2024-03-04T07:44:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688ba6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688babd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f463c9d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560688b0c370 con 0x560688ba6850 2024-03-04T07:44:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f463c9d8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560688ab5550 con 0x560688bac3b0 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f463c9d8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560688b08410 con 0x560688ba7240 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f462a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688ba6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f462a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688ba6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38650/0 (socket says 172.21.15.116:38650) 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f462a000640 1 -- 172.21.15.116:0/1872787134 learned_addr learned my addr 172.21.15.116:0/1872787134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f4632a00640 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688babd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.964+0000 7f462a000640 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462a000640 1 -- 172.21.15.116:0/1872787134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 msgr2=0x560688babd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462a000640 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688babd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462a000640 1 -- 172.21.15.116:0/1872787134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 msgr2=0x560688bae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462a000640 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462a000640 1 -- 172.21.15.116:0/1872787134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560688ac93c0 con 0x560688ba6850 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462a000640 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688ba6c70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f461c0136c0 tx=0x7f461c009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20e0a57c75d1b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462be00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46180a1ea0 0x7f46180a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f462be00640 1 -- 172.21.15.116:0/3901829690 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f462c0f01b0 con 0x7f46341097e0 2024-03-04T07:44:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f4623e00640 1 -- 172.21.15.116:0/1872787134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c013080 con 0x560688ba6850 2024-03-04T07:44:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f4623e00640 1 -- 172.21.15.116:0/1872787134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f461c013220 con 0x560688ba6850 2024-03-04T07:44:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f4623e00640 1 -- 172.21.15.116:0/1872787134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00fcd0 con 0x560688ba6850 2024-03-04T07:44:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.965+0000 7f4632a00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46180a1ea0 0x7f46180a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 -- 172.21.15.116:0/1872787134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 msgr2=0x560688ba6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688ba6c70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f461c0136c0 tx=0x7f461c009630 comp rx=0 tx=0).stop 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 -- 172.21.15.116:0/1872787134 shutdown_connections 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688babd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1872787134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688ba6c70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f4632a00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46180a1ea0 0x7f46180a4360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x560688ba7640 tx=0x7f462400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 -- 172.21.15.116:0/1872787134 >> 172.21.15.116:0/1872787134 conn(0x560688ba1dc0 msgr2=0x560688ba21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 -- 172.21.15.116:0/1872787134 shutdown_connections 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.966+0000 7f463c9d8740 1 -- 172.21.15.116:0/1872787134 wait complete. 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 -- start start 2024-03-04T07:44:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 Processor -- start 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688bb1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bb80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560688baf300 con 0x560688ba6850 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560688ac89b0 con 0x560688bac3b0 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560688baf480 con 0x560688ba7240 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.967+0000 7f463c9d8740 10 client.0 _ll_register_callbacks cb 0x560688b04340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4632a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4633400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bb80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4632a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38658/0 (socket says 172.21.15.116:38658) 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4632a00640 1 -- 172.21.15.116:0/1072766041 learned_addr learned my addr 172.21.15.116:0/1072766041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f462a000640 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688bb1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4633400640 1 -- 172.21.15.116:0/1072766041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 msgr2=0x560688bb1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4633400640 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688bb1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4633400640 1 -- 172.21.15.116:0/1072766041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 msgr2=0x560688bb0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4633400640 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4633400640 1 -- 172.21.15.116:0/1072766041 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560688821210 con 0x560688bac3b0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4632a00640 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4633400640 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bb80d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f462c006010 tx=0x7f462c0bc800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f101a1c8804626ce server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4620c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4620c00640 1 -- 172.21.15.116:0/1072766041 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c0bcea0 con 0x560688bac3b0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.968+0000 7f4620c00640 1 -- 172.21.15.116:0/1072766041 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f462c032890 con 0x560688bac3b0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f461c013220 con 0x560688bac3b0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f4620c00640 1 -- 172.21.15.116:0/1072766041 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f462c0c0ff0 con 0x560688bac3b0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 10 client.26069 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5606888ed620 con 0x560688bac3b0 2024-03-04T07:44:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f4620c00640 1 -- 172.21.15.116:0/1072766041 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f461c013220 con 0x560688bac3b0 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f4620c00640 1 -- 172.21.15.116:0/1072766041 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f462c03c030 con 0x560688bac3b0 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 10 client.26069 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 10 client.26069 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 20 client.26069 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bbc950 0x560688bbed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 4 client.26069 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.969+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560688b198b0 con 0x560688bbc950 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.970+0000 7f4632a00640 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bbc950 0x560688bbed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.970+0000 7f4632a00640 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bbc950 0x560688bbed60 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9afb4ddf65a9d6b server_cookie=8fbd3f5ae422404a in_seq=0 out_seq=0 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.970+0000 7f4620c00640 10 client.26069 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f4620c00640 1 -- 172.21.15.116:0/1072766041 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560688b198b0 con 0x560688bbc950 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f4620c00640 10 client.26069 handle_command_reply: tid=0 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 client.26069 shutdown 2024-03-04T07:44:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 msgr2=0x560688bb80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bb80d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f462c006010 tx=0x7f462c0bc800 comp rx=0 tx=0).stop 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 shutdown_connections 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bbc950 0x560688bbed60 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688bac3b0 0x560688bb80d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba7240 0x560688bb1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f4620c00640 0 client.26069 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1072766041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 >> 172.21.15.116:0/1072766041 conn(0x560688ba1dc0 msgr2=0x560688ba0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 shutdown_connections 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 1 -- 172.21.15.116:0/1072766041 wait complete. 2024-03-04T07:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.971+0000 7f463c9d8740 20 client.26069 trim_cache size 0 max 16384 2024-03-04T07:44:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.986+0000 7f463c9d8740 1 Processor -- start 2024-03-04T07:44:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.986+0000 7f463c9d8740 1 -- start start 2024-03-04T07:44:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688baf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba7240 0x560688ba0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f463c9d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341acf00 con 0x560688ba6850 2024-03-04T07:44:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f463c9d8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341af4e0 con 0x560688ba7240 2024-03-04T07:44:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f463c9d8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341ad500 con 0x560688ba1dc0 2024-03-04T07:44:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38674/0 (socket says 172.21.15.116:38674) 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 -- 172.21.15.116:0/1060088132 learned_addr learned my addr 172.21.15.116:0/1060088132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f4633400640 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba7240 0x560688ba0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f4632a00640 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688baf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 -- 172.21.15.116:0/1060088132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 msgr2=0x560688baf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688baf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 -- 172.21.15.116:0/1060088132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba7240 msgr2=0x560688ba0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba7240 0x560688ba0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f462a000640 1 -- 172.21.15.116:0/1060088132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560688bbf3e0 con 0x560688ba6850 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.987+0000 7f4633400640 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba7240 0x560688ba0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.988+0000 7f462a000640 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb8090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f461c009380 tx=0x7f461c009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a334ef6b43f500e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:02.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.988+0000 7f4623400640 1 -- 172.21.15.116:0/1060088132 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c026070 con 0x560688ba6850 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.988+0000 7f4623400640 1 -- 172.21.15.116:0/1060088132 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f461c005520 con 0x560688ba6850 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.988+0000 7f4623400640 1 -- 172.21.15.116:0/1060088132 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00fa90 con 0x560688ba6850 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.988+0000 7f463c9d8740 1 -- 172.21.15.116:0/1060088132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 msgr2=0x560688bb8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb8090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f461c009380 tx=0x7f461c009cc0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 -- 172.21.15.116:0/1060088132 shutdown_connections 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba7240 0x560688ba0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba6850 0x560688bb8090 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 --2- 172.21.15.116:0/1060088132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688baf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 -- 172.21.15.116:0/1060088132 >> 172.21.15.116:0/1060088132 conn(0x560688babef0 msgr2=0x560688ba21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 -- 172.21.15.116:0/1060088132 shutdown_connections 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 -- 172.21.15.116:0/1060088132 wait complete. 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 -- start start 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.989+0000 7f463c9d8740 1 Processor -- start 2024-03-04T07:44:02.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688bb5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba6850 0x560688bb6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f463c9d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba7240 0x560688bb7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f463c9d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341adb60 con 0x560688ba7240 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f463c9d8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341aee80 con 0x560688ba6850 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f463c9d8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341ae1c0 con 0x560688ba1dc0 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f463c9d8740 10 client.0 _ll_register_callbacks cb 0x7f462c035f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba6850 0x560688bb6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f4633400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba7240 0x560688bb7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba6850 0x560688bb6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58286/0 (socket says 172.21.15.116:58286) 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 -- 172.21.15.116:0/2723384967 learned_addr learned my addr 172.21.15.116:0/2723384967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f4632a00640 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688bb5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 -- 172.21.15.116:0/2723384967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 msgr2=0x560688bb5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688bb5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 -- 172.21.15.116:0/2723384967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba7240 msgr2=0x560688bb7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba7240 0x560688bb7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f462a000640 1 -- 172.21.15.116:0/2723384967 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560688bb7a30 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.990+0000 7f4633400640 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba7240 0x560688bb7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f462a000640 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba6850 0x560688bb6430 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f461c009350 tx=0x7f461c013260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f975372e972290f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f4620c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f4620c00640 1 -- 172.21.15.116:0/2723384967 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c026070 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f461c005520 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f4620c00640 1 -- 172.21.15.116:0/2723384967 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f461c00fe20 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f4620c00640 1 -- 172.21.15.116:0/2723384967 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f461c0035f0 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f463c9d8740 10 client.26075 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560688c189a0 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.991+0000 7f4620c00640 1 -- 172.21.15.116:0/2723384967 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f461c005520 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f4620c00640 1 -- 172.21.15.116:0/2723384967 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f461c0208b0 con 0x560688ba6850 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f463c9d8740 10 client.26075 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f463c9d8740 10 client.26075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f463c9d8740 20 client.26075 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f463c9d8740 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bac3b0 0x560688c1b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f463c9d8740 4 client.26075 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:44:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560688b13c00 con 0x560688bac3b0 2024-03-04T07:44:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f4632a00640 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bac3b0 0x560688c1b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f4632a00640 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bac3b0 0x560688c1b190 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6c21a8d47b5301f server_cookie=a727e747bce529d1 in_seq=0 out_seq=0 2024-03-04T07:44:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:02.992+0000 7f4620c00640 10 client.26075 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.006+0000 7f4620c00640 1 -- 172.21.15.116:0/2723384967 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560688b13c00 con 0x560688bac3b0 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.006+0000 7f4620c00640 10 client.26075 handle_command_reply: tid=0 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.006+0000 7f463c9d8740 1 client.26075 shutdown 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba6850 msgr2=0x560688bb6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba6850 0x560688bb6430 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f461c009350 tx=0x7f461c013260 comp rx=0 tx=0).stop 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 shutdown_connections 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560688bac3b0 0x560688c1b190 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560688ba7240 0x560688bb7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f4620c00640 0 client.26075 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560688ba6850 0x560688bb6430 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 --2- 172.21.15.116:0/2723384967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560688ba1dc0 0x560688bb5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 >> 172.21.15.116:0/2723384967 conn(0x560688babef0 msgr2=0x560688ba5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 shutdown_connections 2024-03-04T07:44:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 1 -- 172.21.15.116:0/2723384967 wait complete. 2024-03-04T07:44:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.007+0000 7f463c9d8740 20 client.26075 trim_cache size 0 max 16384 2024-03-04T07:44:03.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:44:03.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:44:03.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:44:03.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:44:03.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46180a1ea0 msgr2=0x7f46180a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:03.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46180a1ea0 0x7f46180a4360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x560688ba7640 tx=0x7f462400b4a0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 msgr2=0x7f46341a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:03.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 0x7f46341a53f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f462c037040 tx=0x7f462c035d10 comp rx=0 tx=0).stop 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 shutdown_connections 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46341097e0 0x7f46341a53f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46180a1ea0 0x7f46180a4360 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4634108e20 0x7f463419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 --2- 172.21.15.116:0/3901829690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4634107c30 0x7f463419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 >> 172.21.15.116:0/3901829690 conn(0x7f4634076be0 msgr2=0x7f463410c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 shutdown_connections 2024-03-04T07:44:03.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:03.008+0000 7f4639c00640 1 -- 172.21.15.116:0/3901829690 wait complete. 2024-03-04T07:44:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:02 smithi152 ceph-mon[37154]: pgmap v499: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:44:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:02 smithi133 ceph-mon[36505]: pgmap v499: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:44:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:02 smithi116 ceph-mon[32150]: pgmap v499: 97 pgs: 97 active+clean; 2.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:44:05.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:04 smithi133 ceph-mon[36505]: pgmap v500: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:04 smithi116 ceph-mon[32150]: pgmap v500: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:04 smithi152 ceph-mon[37154]: pgmap v500: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:07.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:06 smithi133 ceph-mon[36505]: pgmap v501: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:44:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:06 smithi116 ceph-mon[32150]: pgmap v501: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:44:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:06 smithi152 ceph-mon[37154]: pgmap v501: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:44:09.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:08 smithi133 ceph-mon[36505]: pgmap v502: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:08 smithi116 ceph-mon[32150]: pgmap v502: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:08 smithi152 ceph-mon[37154]: pgmap v502: 97 pgs: 97 active+clean; 2.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:11.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:10 smithi133 ceph-mon[36505]: pgmap v503: 97 pgs: 97 active+clean; 2.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:10 smithi116 ceph-mon[32150]: pgmap v503: 97 pgs: 97 active+clean; 2.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:10 smithi152 ceph-mon[37154]: pgmap v503: 97 pgs: 97 active+clean; 2.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:13.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:12 smithi133 ceph-mon[36505]: pgmap v504: 97 pgs: 97 active+clean; 2.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:12 smithi116 ceph-mon[32150]: pgmap v504: 97 pgs: 97 active+clean; 2.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:12 smithi152 ceph-mon[37154]: pgmap v504: 97 pgs: 97 active+clean; 2.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:15.074 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:14 smithi152 ceph-mon[37154]: pgmap v505: 97 pgs: 97 active+clean; 2.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:15.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:14 smithi133 ceph-mon[36505]: pgmap v505: 97 pgs: 97 active+clean; 2.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:14 smithi116 ceph-mon[32150]: pgmap v505: 97 pgs: 97 active+clean; 2.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:16.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 -- 172.21.15.116:0/1256195027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 msgr2=0x7fdfdc104cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/1256195027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 0x7fdfdc104cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009900 tx=0x7fdfc402bc40 comp rx=0 tx=0).stop 2024-03-04T07:44:16.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 -- 172.21.15.116:0/1256195027 shutdown_connections 2024-03-04T07:44:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/1256195027 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc105200 0x7fdfdc10ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/1256195027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 0x7fdfdc104cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/1256195027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfdc103650 0x7fdfdc103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 -- 172.21.15.116:0/1256195027 >> 172.21.15.116:0/1256195027 conn(0x7fdfdc0febe0 msgr2=0x7fdfdc101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 -- 172.21.15.116:0/1256195027 shutdown_connections 2024-03-04T07:44:16.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.149+0000 7fdfe0e00640 1 -- 172.21.15.116:0/1256195027 wait complete. 2024-03-04T07:44:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.150+0000 7fdfe0e00640 1 Processor -- start 2024-03-04T07:44:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.150+0000 7fdfe0e00640 1 -- start start 2024-03-04T07:44:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfe0e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc103650 0x7fdfdc199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfe0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 0x7fdfdc199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfe0e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfdc105200 0x7fdfdc1a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfe0e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc10f300 con 0x7fdfdc104840 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfe0e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc10f180 con 0x7fdfdc103650 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfe0e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc10f480 con 0x7fdfdc105200 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc103650 0x7fdfdc199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdaa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 0x7fdfdc199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc103650 0x7fdfdc199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49676/0 (socket says 172.21.15.116:49676) 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 -- 172.21.15.116:0/3446545525 learned_addr learned my addr 172.21.15.116:0/3446545525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfdc105200 0x7fdfdc1a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 -- 172.21.15.116:0/3446545525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfdc105200 msgr2=0x7fdfdc1a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfdc105200 0x7fdfdc1a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 -- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 msgr2=0x7fdfdc199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 0x7fdfdc199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdb400640 1 -- 172.21.15.116:0/3446545525 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc1a1590 con 0x7fdfdc103650 2024-03-04T07:44:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.151+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfdc105200 0x7fdfdc1a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.152+0000 7fdfdb400640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc103650 0x7fdfdc199970 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00d950 tx=0x7fdfcc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.152+0000 7fdfd3e00640 1 -- 172.21.15.116:0/3446545525 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc017020 con 0x7fdfdc103650 2024-03-04T07:44:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.152+0000 7fdfd3e00640 1 -- 172.21.15.116:0/3446545525 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfcc009c30 con 0x7fdfdc103650 2024-03-04T07:44:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.152+0000 7fdfd3e00640 1 -- 172.21.15.116:0/3446545525 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00f720 con 0x7fdfdc103650 2024-03-04T07:44:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.152+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfdc1a1880 con 0x7fdfdc103650 2024-03-04T07:44:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.152+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfdc1a1d50 con 0x7fdfdc103650 2024-03-04T07:44:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.155+0000 7fdfd3e00640 1 -- 172.21.15.116:0/3446545525 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fdfcc01a020 con 0x7fdfdc103650 2024-03-04T07:44:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.155+0000 7fdfe521f740 1 Processor -- start 2024-03-04T07:44:16.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.155+0000 7fdfe521f740 1 -- start start 2024-03-04T07:44:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e3fde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e402f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e405a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfe521f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f3e363370 con 0x560f3e3fda10 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfe521f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f3e30c550 con 0x560f3e403570 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfe521f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f3e35f410 con 0x560f3e3fe400 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e3fde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e3fde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44584/0 (socket says 172.21.15.116:44584) 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfdbe00640 1 -- 172.21.15.116:0/832301255 learned_addr learned my addr 172.21.15.116:0/832301255 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e405a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.156+0000 7fdfdb400640 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e402f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdbe00640 1 -- 172.21.15.116:0/832301255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 msgr2=0x560f3e402f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e402f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdbe00640 1 -- 172.21.15.116:0/832301255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 msgr2=0x560f3e405a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e405a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdbe00640 1 -- 172.21.15.116:0/832301255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f3e3203c0 con 0x560f3e3fda10 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e405a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfd3e00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfc00a1df0 0x7fdfc00a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e3fde30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4005f40 tx=0x7fdfd4010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f1f2cd5b32d64f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfd3e00640 1 -- 172.21.15.116:0/3446545525 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdfcc0cdc90 con 0x7fdfdc103650 2024-03-04T07:44:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfdb400640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfc00a1df0 0x7fdfc00a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfd0c00640 1 -- 172.21.15.116:0/832301255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4011700 con 0x560f3e3fda10 2024-03-04T07:44:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfd0c00640 1 -- 172.21.15.116:0/832301255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfd4011d80 con 0x560f3e3fda10 2024-03-04T07:44:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.157+0000 7fdfd0c00640 1 -- 172.21.15.116:0/832301255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd400e3c0 con 0x560f3e3fda10 2024-03-04T07:44:16.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 -- 172.21.15.116:0/832301255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 msgr2=0x560f3e3fde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfdb400640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfc00a1df0 0x7fdfc00a42b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc09a950 tx=0x7fdfcc0d1710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e3fde30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4005f40 tx=0x7fdfd4010860 comp rx=0 tx=0).stop 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 -- 172.21.15.116:0/832301255 shutdown_connections 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e405a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e402f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 --2- 172.21.15.116:0/832301255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e3fde30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 -- 172.21.15.116:0/832301255 >> 172.21.15.116:0/832301255 conn(0x560f3e3f8f80 msgr2=0x560f3e3f9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 -- 172.21.15.116:0/832301255 shutdown_connections 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.158+0000 7fdfe521f740 1 -- 172.21.15.116:0/832301255 wait complete. 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 -- start start 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 Processor -- start 2024-03-04T07:44:16.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e407d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e3fe400 0x560f3e408280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e403570 0x560f3e40f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f3e406520 con 0x560f3e3fda10 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f3e31f9b0 con 0x560f3e3fe400 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f3e4066a0 con 0x560f3e403570 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.159+0000 7fdfe521f740 10 client.0 _ll_register_callbacks cb 0x560f3e35b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e3fe400 0x560f3e408280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e407d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e3fe400 0x560f3e408280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49700/0 (socket says 172.21.15.116:49700) 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 -- 172.21.15.116:0/2988335784 learned_addr learned my addr 172.21.15.116:0/2988335784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 -- 172.21.15.116:0/2988335784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e403570 msgr2=0x560f3e40f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e403570 0x560f3e40f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 -- 172.21.15.116:0/2988335784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 msgr2=0x560f3e407d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e407d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 -- 172.21.15.116:0/2988335784 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f3e078210 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e3fe400 0x560f3e408280 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4006010 tx=0x7fdfc4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a73e4a6d392c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.160+0000 7fdfc9600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfc9600640 1 -- 172.21.15.116:0/2988335784 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4005d50 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfc9600640 1 -- 172.21.15.116:0/2988335784 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdfc40046c0 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfd4011d80 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfc9600640 1 -- 172.21.15.116:0/2988335784 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfc40048c0 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfe521f740 10 client.26087 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560f3e144620 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfc9600640 1 -- 172.21.15.116:0/2988335784 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfd4011d80 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfc9600640 1 -- 172.21.15.116:0/2988335784 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdfc4003710 con 0x560f3e3fe400 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfe521f740 10 client.26087 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfe521f740 10 client.26087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.161+0000 7fdfe521f740 20 client.26087 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.162+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e413af0 0x560f3e415ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.162+0000 7fdfe521f740 4 client.26087 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.162+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f3e3708b0 con 0x560f3e413af0 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.162+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e413af0 0x560f3e415ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.162+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e413af0 0x560f3e415ee0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7854e5723a1e32c server_cookie=6580133be0deb20e in_seq=0 out_seq=0 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.162+0000 7fdfc9600640 10 client.26087 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfc9600640 1 -- 172.21.15.116:0/2988335784 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560f3e3708b0 con 0x560f3e413af0 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfc9600640 10 client.26087 handle_command_reply: tid=0 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 client.26087 shutdown 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e3fe400 msgr2=0x560f3e408280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e3fe400 0x560f3e408280 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4006010 tx=0x7fdfc4004270 comp rx=0 tx=0).stop 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 shutdown_connections 2024-03-04T07:44:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e413af0 0x560f3e415ee0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e403570 0x560f3e40f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e3fe400 0x560f3e408280 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfc9600640 0 client.26087 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2988335784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e407d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 >> 172.21.15.116:0/2988335784 conn(0x560f3e3f8f80 msgr2=0x560f3e3f8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 shutdown_connections 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.163+0000 7fdfe521f740 1 -- 172.21.15.116:0/2988335784 wait complete. 2024-03-04T07:44:16.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.164+0000 7fdfe521f740 20 client.26087 trim_cache size 0 max 16384 2024-03-04T07:44:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.178+0000 7fdfe521f740 1 Processor -- start 2024-03-04T07:44:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.178+0000 7fdfe521f740 1 -- start start 2024-03-04T07:44:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fda10 0x560f3e3ff720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fe400 0x560f3e3ffc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfe521f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1a8ba0 con 0x560f3e3fe400 2024-03-04T07:44:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfe521f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1ab180 con 0x560f3e403570 2024-03-04T07:44:16.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfe521f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1a91a0 con 0x560f3e3fda10 2024-03-04T07:44:16.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdaa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fe400 0x560f3e3ffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49702/0 (socket says 172.21.15.116:49702) 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 -- 172.21.15.116:0/2224952768 learned_addr learned my addr 172.21.15.116:0/2224952768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fda10 0x560f3e3ff720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 -- 172.21.15.116:0/2224952768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fda10 msgr2=0x560f3e3ff720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fda10 0x560f3e3ff720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 -- 172.21.15.116:0/2224952768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fe400 msgr2=0x560f3e3ffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fe400 0x560f3e3ffc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 -- 172.21.15.116:0/2224952768 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f3e3ff210 con 0x560f3e403570 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.179+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40f280 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4000f80 tx=0x7fdfd401d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35730c62d298f7be server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfcbe00640 1 -- 172.21.15.116:0/2224952768 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd401bac0 con 0x560f3e403570 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfcbe00640 1 -- 172.21.15.116:0/2224952768 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfd401bc60 con 0x560f3e403570 2024-03-04T07:44:16.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfcbe00640 1 -- 172.21.15.116:0/2224952768 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4029980 con 0x560f3e403570 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 -- 172.21.15.116:0/2224952768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 msgr2=0x560f3e40f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40f280 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4000f80 tx=0x7fdfd401d040 comp rx=0 tx=0).stop 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 -- 172.21.15.116:0/2224952768 shutdown_connections 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40f280 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fe400 0x560f3e3ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 --2- 172.21.15.116:0/2224952768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fda10 0x560f3e3ff720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 -- 172.21.15.116:0/2224952768 >> 172.21.15.116:0/2224952768 conn(0x560f3e3f8f80 msgr2=0x560f3e3f9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 -- 172.21.15.116:0/2224952768 shutdown_connections 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.180+0000 7fdfe521f740 1 -- 172.21.15.116:0/2224952768 wait complete. 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 -- start start 2024-03-04T07:44:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 Processor -- start 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e40cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e40d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1a9800 con 0x560f3e3fda10 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1aab20 con 0x560f3e403570 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfdc1a9e60 con 0x560f3e3fe400 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.181+0000 7fdfe521f740 10 client.0 _ll_register_callbacks cb 0x7fdfc402c5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:16.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e40cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e40cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44606/0 (socket says 172.21.15.116:44606) 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 -- 172.21.15.116:0/1191631177 learned_addr learned my addr 172.21.15.116:0/1191631177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e40d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 -- 172.21.15.116:0/1191631177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 msgr2=0x560f3e40d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e40d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 -- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 msgr2=0x560f3e40e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 -- 172.21.15.116:0/1191631177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f3e40e850 con 0x560f3e3fda10 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e40d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfdbe00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e40cd10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009bd0 tx=0x7fdfd4009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9d72a08d2daf0f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.182+0000 7fdfc9600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfc9600640 1 -- 172.21.15.116:0/1191631177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4023040 con 0x560f3e3fda10 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfc9600640 1 -- 172.21.15.116:0/1191631177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdfd401b430 con 0x560f3e3fda10 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfd401bc60 con 0x560f3e3fda10 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfc9600640 1 -- 172.21.15.116:0/1191631177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfd401b630 con 0x560f3e3fda10 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 10 client.15972 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560f3e46f6e0 con 0x560f3e3fda10 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfc9600640 1 -- 172.21.15.116:0/1191631177 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfd401bc60 con 0x560f3e3fda10 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfc9600640 1 -- 172.21.15.116:0/1191631177 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdfd4029b30 con 0x560f3e3fda10 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 10 client.15972 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 20 client.15972 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e46fd60 0x560f3e472180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 4 client.15972 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:44:16.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.183+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560f3e36ac00 con 0x560f3e46fd60 2024-03-04T07:44:16.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.184+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e46fd60 0x560f3e472180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:16.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.184+0000 7fdfdaa00640 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e46fd60 0x560f3e472180 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4fc0e926c548865 server_cookie=6481ac25e894a2ad in_seq=0 out_seq=0 2024-03-04T07:44:16.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.184+0000 7fdfc9600640 10 client.15972 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:16.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.198+0000 7fdfc9600640 1 -- 172.21.15.116:0/1191631177 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560f3e36ac00 con 0x560f3e46fd60 2024-03-04T07:44:16.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.198+0000 7fdfc9600640 10 client.15972 handle_command_reply: tid=0 2024-03-04T07:44:16.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.198+0000 7fdfe521f740 1 client.15972 shutdown 2024-03-04T07:44:16.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 msgr2=0x560f3e40cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e40cd10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009bd0 tx=0x7fdfd4009920 comp rx=0 tx=0).stop 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 shutdown_connections 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560f3e46fd60 0x560f3e472180 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560f3e403570 0x560f3e40e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f3e3fe400 0x560f3e40d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 --2- 172.21.15.116:0/1191631177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560f3e3fda10 0x560f3e40cd10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfc9600640 0 client.15972 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:16.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 >> 172.21.15.116:0/1191631177 conn(0x560f3e3f8f80 msgr2=0x560f3e402970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 shutdown_connections 2024-03-04T07:44:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 1 -- 172.21.15.116:0/1191631177 wait complete. 2024-03-04T07:44:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.199+0000 7fdfe521f740 20 client.15972 trim_cache size 0 max 16384 2024-03-04T07:44:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:44:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:44:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:44:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:44:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfc00a1df0 msgr2=0x7fdfc00a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfc00a1df0 0x7fdfc00a42b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc09a950 tx=0x7fdfcc0d1710 comp rx=0 tx=0).stop 2024-03-04T07:44:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc103650 msgr2=0x7fdfdc199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc103650 0x7fdfdc199970 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00d950 tx=0x7fdfcc00de10 comp rx=0 tx=0).stop 2024-03-04T07:44:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 shutdown_connections 2024-03-04T07:44:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfc00a1df0 0x7fdfc00a42b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfdc105200 0x7fdfdc1a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc104840 0x7fdfdc199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 --2- 172.21.15.116:0/3446545525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfdc103650 0x7fdfdc199970 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 >> 172.21.15.116:0/3446545525 conn(0x7fdfdc0febe0 msgr2=0x7fdfdc107b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 shutdown_connections 2024-03-04T07:44:16.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:16.200+0000 7fdfe0e00640 1 -- 172.21.15.116:0/3446545525 wait complete. 2024-03-04T07:44:17.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:16 smithi133 ceph-mon[36505]: pgmap v506: 97 pgs: 97 active+clean; 2.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:16 smithi116 ceph-mon[32150]: pgmap v506: 97 pgs: 97 active+clean; 2.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:16 smithi152 ceph-mon[37154]: pgmap v506: 97 pgs: 97 active+clean; 2.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:19.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:18 smithi133 ceph-mon[36505]: pgmap v507: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:44:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:18 smithi116 ceph-mon[32150]: pgmap v507: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:44:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:18 smithi152 ceph-mon[37154]: pgmap v507: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:44:20.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:44:20.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:44:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 -- 172.21.15.116:0/1231492826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540107bf0 msgr2=0x7fb540108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 --2- 172.21.15.116:0/1231492826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540107bf0 0x7fb540108010 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb530009860 tx=0x7fb53002bb70 comp rx=0 tx=0).stop 2024-03-04T07:44:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 -- 172.21.15.116:0/1231492826 shutdown_connections 2024-03-04T07:44:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 --2- 172.21.15.116:0/1231492826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5401097a0 0x7fb540110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 --2- 172.21.15.116:0/1231492826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb540108de0 0x7fb540109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 --2- 172.21.15.116:0/1231492826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540107bf0 0x7fb540108010 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 -- 172.21.15.116:0/1231492826 >> 172.21.15.116:0/1231492826 conn(0x7fb540076b90 msgr2=0x7fb540078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 -- 172.21.15.116:0/1231492826 shutdown_connections 2024-03-04T07:44:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.544+0000 7fb547000640 1 -- 172.21.15.116:0/1231492826 wait complete. 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.545+0000 7fb547000640 1 Processor -- start 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.545+0000 7fb547000640 1 -- start start 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb547000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb540107bf0 0x7fb54019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb547000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540108de0 0x7fb54019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb547000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5401097a0 0x7fb5401a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb547000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb540113870 con 0x7fb5401097a0 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb547000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401136f0 con 0x7fb540108de0 2024-03-04T07:44:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb547000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401139f0 con 0x7fb540107bf0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540108de0 0x7fb54019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb545000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5401097a0 0x7fb5401a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540108de0 0x7fb54019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49730/0 (socket says 172.21.15.116:49730) 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 -- 172.21.15.116:0/2655411122 learned_addr learned my addr 172.21.15.116:0/2655411122 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb53fe00640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb540107bf0 0x7fb54019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 -- 172.21.15.116:0/2655411122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb540107bf0 msgr2=0x7fb54019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb540107bf0 0x7fb54019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 -- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5401097a0 msgr2=0x7fb5401a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5401097a0 0x7fb5401a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 -- 172.21.15.116:0/2655411122 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5401a5b10 con 0x7fb540108de0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.546+0000 7fb537400640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540108de0 0x7fb54019e430 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e9c0 tx=0x7fb52800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.547+0000 7fb53d600640 1 -- 172.21.15.116:0/2655411122 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52800ccc0 con 0x7fb540108de0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.547+0000 7fb53d600640 1 -- 172.21.15.116:0/2655411122 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb528005500 con 0x7fb540108de0 2024-03-04T07:44:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.547+0000 7fb53d600640 1 -- 172.21.15.116:0/2655411122 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528010430 con 0x7fb540108de0 2024-03-04T07:44:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.547+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5401a5e00 con 0x7fb540108de0 2024-03-04T07:44:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.547+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5401a6270 con 0x7fb540108de0 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 Processor -- start 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb53d600640 1 -- 172.21.15.116:0/2655411122 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb5280046c0 con 0x7fb540108de0 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 -- start start 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd51f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd57040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd59b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24cc7f090 con 0x55e24cd576b0 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24ccb6d30 con 0x55e24cd51b50 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.550+0000 7fb549c77740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24cc5fa30 con 0x55e24cd52540 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd51f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd51f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49736/0 (socket says 172.21.15.116:49736) 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 -- 172.21.15.116:0/3835890381 learned_addr learned my addr 172.21.15.116:0/3835890381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb537400640 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd59b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb53fe00640 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd57040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 -- 172.21.15.116:0/3835890381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 msgr2=0x55e24cd57040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd57040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 -- 172.21.15.116:0/3835890381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 msgr2=0x55e24cd59b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd59b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 -- 172.21.15.116:0/3835890381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24ca957d0 con 0x55e24cd51b50 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.551+0000 7fb545000640 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd51f70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb538009fd0 tx=0x7fb5380118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65a747367bcd29c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb535600640 1 -- 172.21.15.116:0/3835890381 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538004d20 con 0x55e24cd51b50 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb535600640 1 -- 172.21.15.116:0/3835890381 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb538004050 con 0x55e24cd51b50 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb535600640 1 -- 172.21.15.116:0/3835890381 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538005260 con 0x55e24cd51b50 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb53d600640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb5240a1d80 0x7fb5240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb53d600640 1 -- 172.21.15.116:0/2655411122 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb5280ce060 con 0x7fb540108de0 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb53fe00640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb5240a1d80 0x7fb5240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb549c77740 1 -- 172.21.15.116:0/3835890381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 msgr2=0x55e24cd51f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.552+0000 7fb549c77740 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd51f70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb538009fd0 tx=0x7fb5380118d0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 -- 172.21.15.116:0/3835890381 shutdown_connections 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd59b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd57040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 --2- 172.21.15.116:0/3835890381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd51f70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 -- 172.21.15.116:0/3835890381 >> 172.21.15.116:0/3835890381 conn(0x55e24cd4d0c0 msgr2=0x55e24cd4d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 -- 172.21.15.116:0/3835890381 shutdown_connections 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 -- 172.21.15.116:0/3835890381 wait complete. 2024-03-04T07:44:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb53fe00640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb5240a1d80 0x7fb5240a4240 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb530009830 tx=0x7fb530034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 -- start start 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.553+0000 7fb549c77740 1 Processor -- start 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb549c77740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd51b50 0x55e24cd5bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb549c77740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd52540 0x55e24cd5c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb549c77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd63490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb549c77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24cc550c0 con 0x55e24cd576b0 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb549c77740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24ccb2dd0 con 0x55e24cd52540 2024-03-04T07:44:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb549c77740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24cd5a660 con 0x55e24cd51b50 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb549c77740 10 client.0 _ll_register_callbacks cb 0x55e24cc72860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd51b50 0x55e24cd5bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb545000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd63490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb537400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd52540 0x55e24cd5c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd51b50 0x55e24cd5bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53588/0 (socket says 172.21.15.116:53588) 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.554+0000 7fb53fe00640 1 -- 172.21.15.116:0/467963474 learned_addr learned my addr 172.21.15.116:0/467963474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb537400640 1 -- 172.21.15.116:0/467963474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd51b50 msgr2=0x55e24cd5bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb537400640 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd51b50 0x55e24cd5bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb537400640 1 -- 172.21.15.116:0/467963474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 msgr2=0x55e24cd63490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb537400640 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd63490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb537400640 1 -- 172.21.15.116:0/467963474 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e24c9cb3f0 con 0x55e24cd52540 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb537400640 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd52540 0x55e24cd5c450 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb52800c6a0 tx=0x7fb52809ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d7c76b916d5742 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb52e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb52e000640 1 -- 172.21.15.116:0/467963474 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280d0310 con 0x55e24cd52540 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb52e000640 1 -- 172.21.15.116:0/467963474 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb5280a30a0 con 0x55e24cd52540 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb538004050 con 0x55e24cd52540 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb52e000640 1 -- 172.21.15.116:0/467963474 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb52809b490 con 0x55e24cd52540 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb549c77740 10 client.26105 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.555+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e24c98b0f0 con 0x55e24cd52540 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb52e000640 1 -- 172.21.15.116:0/467963474 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb538004050 con 0x55e24cd52540 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb52e000640 1 -- 172.21.15.116:0/467963474 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb52809a1e0 con 0x55e24cd52540 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb549c77740 10 client.26105 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb549c77740 10 client.26105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb549c77740 20 client.26105 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb549c77740 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cd679b0 0x55e24cd69dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb549c77740 4 client.26105 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e24ccc4270 con 0x55e24cd679b0 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.556+0000 7fb53fe00640 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cd679b0 0x55e24cd69dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.557+0000 7fb53fe00640 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cd679b0 0x55e24cd69dc0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d798e2ed8e548eca server_cookie=51df180cd3cd78b in_seq=0 out_seq=0 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.557+0000 7fb52e000640 10 client.26105 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb52e000640 1 -- 172.21.15.116:0/467963474 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e24ccc4270 con 0x55e24cd679b0 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb52e000640 10 client.26105 handle_command_reply: tid=0 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 client.26105 shutdown 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd52540 msgr2=0x55e24cd5c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd52540 0x55e24cd5c450 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb52800c6a0 tx=0x7fb52809ab60 comp rx=0 tx=0).stop 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 shutdown_connections 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cd679b0 0x55e24cd69dc0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd63490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd52540 0x55e24cd5c450 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 --2- 172.21.15.116:0/467963474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd51b50 0x55e24cd5bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb52e000640 0 client.26105 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:20.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 >> 172.21.15.116:0/467963474 conn(0x55e24cd4d0c0 msgr2=0x55e24cd4c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:20.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 shutdown_connections 2024-03-04T07:44:20.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 1 -- 172.21.15.116:0/467963474 wait complete. 2024-03-04T07:44:20.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.558+0000 7fb549c77740 20 client.26105 trim_cache size 0 max 16384 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.575+0000 7fb549c77740 1 Processor -- start 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.575+0000 7fb549c77740 1 -- start start 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb549c77740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd4bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb549c77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd52540 0x55e24cd63450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb549c77740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd576b0 0x55e24cd4c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb549c77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401ad100 con 0x55e24cd52540 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb549c77740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401af6e0 con 0x55e24cd51b50 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb549c77740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401ad700 con 0x55e24cd576b0 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd52540 0x55e24cd63450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd52540 0x55e24cd63450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44646/0 (socket says 172.21.15.116:44646) 2024-03-04T07:44:20.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 -- 172.21.15.116:0/1700102106 learned_addr learned my addr 172.21.15.116:0/1700102106 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb545000640 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd576b0 0x55e24cd4c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb53fe00640 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd4bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 -- 172.21.15.116:0/1700102106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd576b0 msgr2=0x55e24cd4c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd576b0 0x55e24cd4c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 -- 172.21.15.116:0/1700102106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 msgr2=0x55e24cd4bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd4bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb537400640 1 -- 172.21.15.116:0/1700102106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24cd56d30 con 0x55e24cd52540 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb53fe00640 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd4bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.576+0000 7fb545000640 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd576b0 0x55e24cd4c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.577+0000 7fb537400640 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd52540 0x55e24cd63450 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb528005760 tx=0x7fb52809ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6290c08294522 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.577+0000 7fb534c00640 1 -- 172.21.15.116:0/1700102106 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52809a8d0 con 0x55e24cd52540 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.577+0000 7fb534c00640 1 -- 172.21.15.116:0/1700102106 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5280b91a0 con 0x55e24cd52540 2024-03-04T07:44:20.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.577+0000 7fb534c00640 1 -- 172.21.15.116:0/1700102106 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280a30a0 con 0x55e24cd52540 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.577+0000 7fb549c77740 1 -- 172.21.15.116:0/1700102106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd52540 msgr2=0x55e24cd63450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.577+0000 7fb549c77740 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd52540 0x55e24cd63450 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb528005760 tx=0x7fb52809ac90 comp rx=0 tx=0).stop 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 -- 172.21.15.116:0/1700102106 shutdown_connections 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd576b0 0x55e24cd4c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd52540 0x55e24cd63450 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 --2- 172.21.15.116:0/1700102106 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd4bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 -- 172.21.15.116:0/1700102106 >> 172.21.15.116:0/1700102106 conn(0x55e24cd4d0c0 msgr2=0x55e24cd53a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 -- 172.21.15.116:0/1700102106 shutdown_connections 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 -- 172.21.15.116:0/1700102106 wait complete. 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 -- start start 2024-03-04T07:44:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.578+0000 7fb549c77740 1 Processor -- start 2024-03-04T07:44:20.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb549c77740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd61410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb549c77740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd61950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb549c77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd62960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb549c77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401add60 con 0x55e24cd576b0 2024-03-04T07:44:20.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb549c77740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401af080 con 0x55e24cd51b50 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb549c77740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401ae3c0 con 0x55e24cd52540 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb549c77740 10 client.0 _ll_register_callbacks cb 0x7fb52800ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd61410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd61410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49770/0 (socket says 172.21.15.116:49770) 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb545000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd62960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 -- 172.21.15.116:0/599893720 learned_addr learned my addr 172.21.15.116:0/599893720 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb537400640 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd61950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 -- 172.21.15.116:0/599893720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 msgr2=0x55e24cd61950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd61950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 -- 172.21.15.116:0/599893720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 msgr2=0x55e24cd62960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd62960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 -- 172.21.15.116:0/599893720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e24cd62f20 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.579+0000 7fb53fe00640 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd61410 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb530034940 tx=0x7fb53003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6797cbc21f26de95 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb52e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb52e000640 1 -- 172.21.15.116:0/599893720 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530043040 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb52e000640 1 -- 172.21.15.116:0/599893720 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb530004240 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e24cd57420 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb52e000640 1 -- 172.21.15.116:0/599893720 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb530039040 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb549c77740 10 client.26111 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e24cdc3b90 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb52e000640 1 -- 172.21.15.116:0/599893720 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e24cd57420 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb52e000640 1 -- 172.21.15.116:0/599893720 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb53005b050 con 0x55e24cd51b50 2024-03-04T07:44:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb549c77740 10 client.26111 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb549c77740 10 client.26111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb549c77740 20 client.26111 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.580+0000 7fb549c77740 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cdc44c0 0x55e24cdc68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb549c77740 4 client.26111 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e24ccbe5c0 con 0x55e24cdc44c0 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb537400640 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cdc44c0 0x55e24cdc68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb537400640 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cdc44c0 0x55e24cdc68e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a16c3f210dacc8c server_cookie=3bb92f3086ae3614 in_seq=0 out_seq=0 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb52e000640 10 client.26111 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb52e000640 1 -- 172.21.15.116:0/599893720 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e24ccbe5c0 con 0x55e24cdc44c0 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb52e000640 10 client.26111 handle_command_reply: tid=0 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.581+0000 7fb549c77740 1 client.26111 shutdown 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 msgr2=0x55e24cd61410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd61410 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb530034940 tx=0x7fb53003f000 comp rx=0 tx=0).stop 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 shutdown_connections 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24cdc44c0 0x55e24cdc68e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e24cd576b0 0x55e24cd62960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e24cd52540 0x55e24cd61950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 --2- 172.21.15.116:0/599893720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e24cd51b50 0x55e24cd61410 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 >> 172.21.15.116:0/599893720 conn(0x55e24cd4d0c0 msgr2=0x55e24cd4bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb52e000640 0 client.26111 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 shutdown_connections 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 1 -- 172.21.15.116:0/599893720 wait complete. 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb549c77740 20 client.26111 trim_cache size 0 max 16384 2024-03-04T07:44:20.589 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:20.590 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:44:20.590 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:44:20.590 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb5240a1d80 msgr2=0x7fb5240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb547000640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb5240a1d80 0x7fb5240a4240 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb530009830 tx=0x7fb530034ad0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540108de0 msgr2=0x7fb54019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.582+0000 7fb547000640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540108de0 0x7fb54019e430 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e9c0 tx=0x7fb52800ee80 comp rx=0 tx=0).stop 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 shutdown_connections 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5401097a0 0x7fb5401a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb540108de0 0x7fb54019e430 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb5240a1d80 0x7fb5240a4240 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 --2- 172.21.15.116:0/2655411122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb540107bf0 0x7fb54019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 >> 172.21.15.116:0/2655411122 conn(0x7fb540076b90 msgr2=0x7fb54010c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 shutdown_connections 2024-03-04T07:44:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.583+0000 7fb547000640 1 -- 172.21.15.116:0/2655411122 wait complete. 2024-03-04T07:44:20.600 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:44:20.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 -- 172.21.15.116:0/1174706258 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0081097c0 msgr2=0x7fa008110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1174706258 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0081097c0 0x7fa008110050 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa000009ca0 tx=0x7fa00002c0f0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 -- 172.21.15.116:0/1174706258 shutdown_connections 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1174706258 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa0081097c0 0x7fa008110050 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1174706258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa008108e00 0x7fa008109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1174706258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa008107c10 0x7fa008108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 -- 172.21.15.116:0/1174706258 >> 172.21.15.116:0/1174706258 conn(0x7fa008076be0 msgr2=0x7fa008079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 -- 172.21.15.116:0/1174706258 shutdown_connections 2024-03-04T07:44:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.807+0000 7fa00ce00640 1 -- 172.21.15.116:0/1174706258 wait complete. 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.808+0000 7fa00ce00640 1 Processor -- start 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.808+0000 7fa00ce00640 1 -- start start 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.808+0000 7fa00ce00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa008107c10 0x7fa00819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.808+0000 7fa00ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa008108e00 0x7fa00819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa00ce00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0081097c0 0x7fa0081a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa00ce00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa008113870 con 0x7fa008108e00 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa00ce00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0081136f0 con 0x7fa008107c10 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa00ce00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0081139f0 con 0x7fa0081097c0 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa008107c10 0x7fa00819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa008107c10 0x7fa00819de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49784/0 (socket says 172.21.15.116:49784) 2024-03-04T07:44:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 -- 172.21.15.116:0/1111912349 learned_addr learned my addr 172.21.15.116:0/1111912349 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005400640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa008108e00 0x7fa00819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa006800640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0081097c0 0x7fa0081a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 -- 172.21.15.116:0/1111912349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0081097c0 msgr2=0x7fa0081a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0081097c0 0x7fa0081a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 -- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa008108e00 msgr2=0x7fa00819e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa008108e00 0x7fa00819e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.809+0000 7fa005e00640 1 -- 172.21.15.116:0/1111912349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0081a5aa0 con 0x7fa008107c10 2024-03-04T07:44:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.810+0000 7fa005e00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa008107c10 0x7fa00819de80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800e9c0 tx=0x7f9ff800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.811+0000 7f9ffea00640 1 -- 172.21.15.116:0/1111912349 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff800cd80 con 0x7fa008107c10 2024-03-04T07:44:20.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.811+0000 7f9ffea00640 1 -- 172.21.15.116:0/1111912349 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ff80035f0 con 0x7fa008107c10 2024-03-04T07:44:20.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.811+0000 7f9ffea00640 1 -- 172.21.15.116:0/1111912349 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8010690 con 0x7fa008107c10 2024-03-04T07:44:20.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.811+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0081a5d90 con 0x7fa008107c10 2024-03-04T07:44:20.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.811+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0081a61d0 con 0x7fa008107c10 2024-03-04T07:44:20.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.813+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00810ef30 con 0x7fa008107c10 2024-03-04T07:44:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.814+0000 7f9ffea00640 1 -- 172.21.15.116:0/1111912349 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9ff8004020 con 0x7fa008107c10 2024-03-04T07:44:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.816+0000 7f9ffea00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9fec0a1e00 0x7f9fec0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.816+0000 7f9ffea00640 1 -- 172.21.15.116:0/1111912349 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9ff80ce100 con 0x7fa008107c10 2024-03-04T07:44:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.816+0000 7fa005400640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9fec0a1e00 0x7f9fec0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.817+0000 7fa005400640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9fec0a1e00 0x7f9fec0a42c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa00819f390 tx=0x7f9ff4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:20.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:20.824+0000 7f9ffea00640 1 -- 172.21.15.116:0/1111912349 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9ff809b530 con 0x7fa008107c10 2024-03-04T07:44:21.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:20 smithi116 ceph-mon[32150]: pgmap v508: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:44:21.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.063+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa00802cff0 con 0x7fa008107c10 2024-03-04T07:44:21.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.064+0000 7f9ffea00640 1 -- 172.21.15.116:0/1111912349 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fa00802cff0 con 0x7fa008107c10 2024-03-04T07:44:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:44:21.065 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:44:21.065 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.066+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9fec0a1e00 msgr2=0x7f9fec0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9fec0a1e00 0x7f9fec0a42c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa00819f390 tx=0x7f9ff4005f90 comp rx=0 tx=0).stop 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa008107c10 msgr2=0x7fa00819de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa008107c10 0x7fa00819de80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800e9c0 tx=0x7f9ff800ee80 comp rx=0 tx=0).stop 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 shutdown_connections 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9fec0a1e00 0x7f9fec0a42c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa0081097c0 0x7fa0081a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa008108e00 0x7fa00819e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 --2- 172.21.15.116:0/1111912349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa008107c10 0x7fa00819de80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 >> 172.21.15.116:0/1111912349 conn(0x7fa008076be0 msgr2=0x7fa00810c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:21.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 shutdown_connections 2024-03-04T07:44:21.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.067+0000 7fa00ce00640 1 -- 172.21.15.116:0/1111912349 wait complete. 2024-03-04T07:44:21.088 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:44:21.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:44:21.089 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:20 smithi133 ceph-mon[36505]: pgmap v508: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:44:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:20 smithi152 ceph-mon[37154]: pgmap v508: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 -- 172.21.15.116:0/635975965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 msgr2=0x7fdfc4108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/635975965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc4108010 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009900 tx=0x7fdfb402bc30 comp rx=0 tx=0).stop 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 -- 172.21.15.116:0/635975965 shutdown_connections 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/635975965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfc41097a0 0x7fdfc4110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/635975965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfc4108de0 0x7fdfc4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/635975965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc4108010 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 -- 172.21.15.116:0/635975965 >> 172.21.15.116:0/635975965 conn(0x7fdfc4076b90 msgr2=0x7fdfc4078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 -- 172.21.15.116:0/635975965 shutdown_connections 2024-03-04T07:44:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.301+0000 7fdfc8e00640 1 -- 172.21.15.116:0/635975965 wait complete. 2024-03-04T07:44:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.302+0000 7fdfc8e00640 1 Processor -- start 2024-03-04T07:44:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.302+0000 7fdfc8e00640 1 -- start start 2024-03-04T07:44:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc8e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc8e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfc4108de0 0x7fdfc419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc8e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfc41097a0 0x7fdfc41a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc8e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41138c0 con 0x7fdfc41097a0 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc8e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc4113740 con 0x7fdfc4107bf0 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc8e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc4113a40 con 0x7fdfc4108de0 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc419df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49796/0 (socket says 172.21.15.116:49796) 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc1e00640 1 -- 172.21.15.116:0/2928411070 learned_addr learned my addr 172.21.15.116:0/2928411070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfc2800640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfc41097a0 0x7fdfc41a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.303+0000 7fdfb9400640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfc4108de0 0x7fdfc419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc1e00640 1 -- 172.21.15.116:0/2928411070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfc4108de0 msgr2=0x7fdfc419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfc4108de0 0x7fdfc419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc1e00640 1 -- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfc41097a0 msgr2=0x7fdfc41a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfc41097a0 0x7fdfc41a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc1e00640 1 -- 172.21.15.116:0/2928411070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc41a5b20 con 0x7fdfc4107bf0 2024-03-04T07:44:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc419df00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4000c00 tx=0x7fdfb402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfbb400640 1 -- 172.21.15.116:0/2928411070 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4043020 con 0x7fdfc4107bf0 2024-03-04T07:44:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfbb400640 1 -- 172.21.15.116:0/2928411070 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfb40054e0 con 0x7fdfc4107bf0 2024-03-04T07:44:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfbb400640 1 -- 172.21.15.116:0/2928411070 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4003c40 con 0x7fdfc4107bf0 2024-03-04T07:44:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc41a5db0 con 0x7fdfc4107bf0 2024-03-04T07:44:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.304+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc41a6270 con 0x7fdfc4107bf0 2024-03-04T07:44:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 Processor -- start 2024-03-04T07:44:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfbb400640 1 -- 172.21.15.116:0/2928411070 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fdfb4046020 con 0x7fdfc4107bf0 2024-03-04T07:44:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 -- start start 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98bf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa9823d60 con 0x564fa98beca0 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa97cccf0 con 0x564fa98bf690 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa981fe00 con 0x564fa98c4800 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfc2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98bf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49806/0 (socket says 172.21.15.116:49806) 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.309+0000 7fdfc1e00640 1 -- 172.21.15.116:0/1866725359 learned_addr learned my addr 172.21.15.116:0/1866725359 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfb9400640 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfc1e00640 1 -- 172.21.15.116:0/1866725359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 msgr2=0x564fa98c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfc1e00640 1 -- 172.21.15.116:0/1866725359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 msgr2=0x564fa98bf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98bf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfc1e00640 1 -- 172.21.15.116:0/1866725359 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fa95383f0 con 0x564fa98bf690 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c4190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40f3ce0 tx=0x7fdfb40f3dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5682066e0ae18521 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfb3400640 1 -- 172.21.15.116:0/1866725359 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4102470 con 0x564fa98bf690 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.310+0000 7fdfbb400640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfa80a1e00 0x7fdfa80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfb3400640 1 -- 172.21.15.116:0/1866725359 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfb40fd070 con 0x564fa98bf690 2024-03-04T07:44:21.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfb3400640 1 -- 172.21.15.116:0/1866725359 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40bc390 con 0x564fa98bf690 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfbb400640 1 -- 172.21.15.116:0/2928411070 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdfb40efd00 con 0x7fdfc4107bf0 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfc2800640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfa80a1e00 0x7fdfa80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1866725359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 msgr2=0x564fa98c4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c4190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40f3ce0 tx=0x7fdfb40f3dd0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1866725359 shutdown_connections 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c4190 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1866725359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98bf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1866725359 >> 172.21.15.116:0/1866725359 conn(0x564fa98ba210 msgr2=0x564fa98ba610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1866725359 shutdown_connections 2024-03-04T07:44:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfc2800640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfa80a1e00 0x7fdfa80a42c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x564fa98bfef0 tx=0x7fdfbc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.311+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1866725359 wait complete. 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.312+0000 7fdfcbbf1740 1 -- start start 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.312+0000 7fdfcbbf1740 1 Processor -- start 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.312+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.312+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.312+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98c4800 0x564fa98d0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.312+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa98c7570 con 0x564fa98c4800 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa97c2220 con 0x564fa98bf690 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa98c76f0 con 0x564fa98beca0 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfcbbf1740 10 client.0 _ll_register_callbacks cb 0x564fa9537be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfc2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98c4800 0x564fa98d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49810/0 (socket says 172.21.15.116:49810) 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 -- 172.21.15.116:0/1151880537 learned_addr learned my addr 172.21.15.116:0/1151880537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 -- 172.21.15.116:0/1151880537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 msgr2=0x564fa98c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98c8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 -- 172.21.15.116:0/1151880537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98c4800 msgr2=0x564fa98d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98c4800 0x564fa98d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 -- 172.21.15.116:0/1151880537 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fa96027d0 con 0x564fa98bf690 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfb9400640 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c9410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdfac012720 tx=0x7fdfac012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d32c9a44ddfd36e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.313+0000 7fdfa7e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.314+0000 7fdfa7e00640 1 -- 172.21.15.116:0/1151880537 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac00f920 con 0x564fa98bf690 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.314+0000 7fdfa7e00640 1 -- 172.21.15.116:0/1151880537 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdfac00fac0 con 0x564fa98bf690 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.314+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfb40fd070 con 0x564fa98bf690 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.314+0000 7fdfa7e00640 1 -- 172.21.15.116:0/1151880537 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfac00fcc0 con 0x564fa98bf690 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.314+0000 7fdfcbbf1740 10 client.26129 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.314+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564fa98d3830 con 0x564fa98bf690 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.314+0000 7fdfa7e00640 1 -- 172.21.15.116:0/1151880537 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfb40fd070 con 0x564fa98bf690 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfa7e00640 1 -- 172.21.15.116:0/1151880537 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdfac01c420 con 0x564fa98bf690 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfcbbf1740 10 client.26129 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfcbbf1740 10 client.26129 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfcbbf1740 20 client.26129 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa98d4e60 0x564fa98d72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfcbbf1740 4 client.26129 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564fa98312a0 con 0x564fa98d4e60 2024-03-04T07:44:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa98d4e60 0x564fa98d72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa98d4e60 0x564fa98d72c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a17add208ce563 server_cookie=53159c241327f0e in_seq=0 out_seq=0 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.315+0000 7fdfa7e00640 10 client.26129 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.316+0000 7fdfa7e00640 1 -- 172.21.15.116:0/1151880537 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564fa98312a0 con 0x564fa98d4e60 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.316+0000 7fdfa7e00640 10 client.26129 handle_command_reply: tid=0 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.316+0000 7fdfcbbf1740 1 client.26129 shutdown 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 msgr2=0x564fa98c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c9410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdfac012720 tx=0x7fdfac012be0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 shutdown_connections 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa98d4e60 0x564fa98d72c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98c4800 0x564fa98d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98c9410 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1151880537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98c8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 >> 172.21.15.116:0/1151880537 conn(0x564fa98ba210 msgr2=0x564fa98b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfa7e00640 0 client.26129 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 shutdown_connections 2024-03-04T07:44:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1151880537 wait complete. 2024-03-04T07:44:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.317+0000 7fdfcbbf1740 20 client.26129 trim_cache size 0 max 16384 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.330+0000 7fdfcbbf1740 1 Processor -- start 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfcbbf1740 1 -- start start 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98bf690 0x564fa98d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98c4800 0x564fa98b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41ad0c0 con 0x564fa98bf690 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41af6a0 con 0x564fa98c4800 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41ad6c0 con 0x564fa98beca0 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfb9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98bf690 0x564fa98d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfc2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98c4800 0x564fa98b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfb9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98bf690 0x564fa98d0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44712/0 (socket says 172.21.15.116:44712) 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfb9400640 1 -- 172.21.15.116:0/1576295249 learned_addr learned my addr 172.21.15.116:0/1576295249 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfc2800640 1 -- 172.21.15.116:0/1576295249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 msgr2=0x564fa98b8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfc2800640 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98b8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfc2800640 1 -- 172.21.15.116:0/1576295249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98bf690 msgr2=0x564fa98d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.331+0000 7fdfc2800640 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98bf690 0x564fa98d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.332+0000 7fdfc2800640 1 -- 172.21.15.116:0/1576295249 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fa98c9760 con 0x564fa98c4800 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.332+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.332+0000 7fdfb9400640 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98bf690 0x564fa98d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.332+0000 7fdfc2800640 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98c4800 0x564fa98b9160 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc012010 tx=0x7fdfbc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc71d068d68f5b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.332+0000 7fdfb2a00640 1 -- 172.21.15.116:0/1576295249 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0045d0 con 0x564fa98c4800 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.332+0000 7fdfb2a00640 1 -- 172.21.15.116:0/1576295249 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfbc004770 con 0x564fa98c4800 2024-03-04T07:44:21.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.332+0000 7fdfb2a00640 1 -- 172.21.15.116:0/1576295249 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc00fd30 con 0x564fa98c4800 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1576295249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98c4800 msgr2=0x564fa98b9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98c4800 0x564fa98b9160 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc012010 tx=0x7fdfbc016000 comp rx=0 tx=0).stop 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1576295249 shutdown_connections 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98c4800 0x564fa98b9160 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98bf690 0x564fa98d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/1576295249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98beca0 0x564fa98b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1576295249 >> 172.21.15.116:0/1576295249 conn(0x564fa98ba210 msgr2=0x564fa98c0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1576295249 shutdown_connections 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/1576295249 wait complete. 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.333+0000 7fdfcbbf1740 1 -- start start 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 1 Processor -- start 2024-03-04T07:44:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41add20 con 0x564fa98beca0 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41af040 con 0x564fa98bf690 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41ae380 con 0x564fa98c4800 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfcbbf1740 10 client.0 _ll_register_callbacks cb 0x7fdfac009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfc1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfb9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfc1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98d00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44722/0 (socket says 172.21.15.116:44722) 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.334+0000 7fdfc1e00640 1 -- 172.21.15.116:0/4164168168 learned_addr learned my addr 172.21.15.116:0/4164168168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfc2800640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfb9400640 1 -- 172.21.15.116:0/4164168168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 msgr2=0x564fa98cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfb9400640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfb9400640 1 -- 172.21.15.116:0/4164168168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 msgr2=0x564fa98d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfb9400640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfb9400640 1 -- 172.21.15.116:0/4164168168 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fa98cf140 con 0x564fa98bf690 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfc2800640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfb9400640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98ce6c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x564fa98c4140 tx=0x7fdfac00fbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b301009742d5a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfa7e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfa7e00640 1 -- 172.21.15.116:0/4164168168 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac01ca80 con 0x564fa98bf690 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfa7e00640 1 -- 172.21.15.116:0/4164168168 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdfac01cc20 con 0x564fa98bf690 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdfbc004770 con 0x564fa98bf690 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.335+0000 7fdfa7e00640 1 -- 172.21.15.116:0/4164168168 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfac01ce20 con 0x564fa98bf690 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 10 client.26135 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564fa9931530 con 0x564fa98bf690 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfa7e00640 1 -- 172.21.15.116:0/4164168168 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfbc004770 con 0x564fa98bf690 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfa7e00640 1 -- 172.21.15.116:0/4164168168 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdfac003990 con 0x564fa98bf690 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 10 client.26135 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 10 client.26135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 20 client.26135 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa9931e60 0x564fa9934280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 4 client.26135 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564fa982b5f0 con 0x564fa9931e60 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.336+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa9931e60 0x564fa9934280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfc1e00640 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa9931e60 0x564fa9934280 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dddd6537ecce5e41 server_cookie=2d19bac0b4012eb4 in_seq=0 out_seq=0 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfa7e00640 10 client.26135 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfa7e00640 1 -- 172.21.15.116:0/4164168168 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564fa982b5f0 con 0x564fa9931e60 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfa7e00640 10 client.26135 handle_command_reply: tid=0 2024-03-04T07:44:21.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 client.26135 shutdown 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 msgr2=0x564fa98ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98ce6c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x564fa98c4140 tx=0x7fdfac00fbd0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 shutdown_connections 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564fa9931e60 0x564fa9934280 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564fa98c4800 0x564fa98cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564fa98bf690 0x564fa98ce6c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 --2- 172.21.15.116:0/4164168168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564fa98beca0 0x564fa98d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfa7e00640 0 client.26135 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 >> 172.21.15.116:0/4164168168 conn(0x564fa98ba210 msgr2=0x564fa98b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 shutdown_connections 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 1 -- 172.21.15.116:0/4164168168 wait complete. 2024-03-04T07:44:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.337+0000 7fdfcbbf1740 20 client.26135 trim_cache size 0 max 16384 2024-03-04T07:44:21.344 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:44:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfa80a1e00 msgr2=0x7fdfa80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfa80a1e00 0x7fdfa80a42c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x564fa98bfef0 tx=0x7fdfbc009500 comp rx=0 tx=0).stop 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 msgr2=0x7fdfc419df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc419df00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4000c00 tx=0x7fdfb402c650 comp rx=0 tx=0).stop 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 shutdown_connections 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfc41097a0 0x7fdfc41a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdfa80a1e00 0x7fdfa80a42c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfc4108de0 0x7fdfc419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 --2- 172.21.15.116:0/2928411070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdfc4107bf0 0x7fdfc419df00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 >> 172.21.15.116:0/2928411070 conn(0x7fdfc4076b90 msgr2=0x7fdfc410c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 shutdown_connections 2024-03-04T07:44:21.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:21.338+0000 7fdfc8e00640 1 -- 172.21.15.116:0/2928411070 wait complete. 2024-03-04T07:44:22.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:21 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1111912349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:21 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1111912349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:22.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:21 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1111912349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:22.358 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:44:22.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:44:22.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.578+0000 7fdd43200640 1 -- 172.21.15.116:0/3575791900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 msgr2=0x7fdd3c10bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.578+0000 7fdd43200640 1 --2- 172.21.15.116:0/3575791900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 0x7fdd3c10bb30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009ca0 tx=0x7fdd3802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.579+0000 7fdd43200640 1 -- 172.21.15.116:0/3575791900 shutdown_connections 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.579+0000 7fdd43200640 1 --2- 172.21.15.116:0/3575791900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 0x7fdd3c10bb30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.579+0000 7fdd43200640 1 --2- 172.21.15.116:0/3575791900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd3c1048e0 0x7fdd3c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.579+0000 7fdd43200640 1 --2- 172.21.15.116:0/3575791900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1036f0 0x7fdd3c103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.579+0000 7fdd43200640 1 -- 172.21.15.116:0/3575791900 >> 172.21.15.116:0/3575791900 conn(0x7fdd3c0fec80 msgr2=0x7fdd3c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.579+0000 7fdd43200640 1 -- 172.21.15.116:0/3575791900 shutdown_connections 2024-03-04T07:44:22.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.579+0000 7fdd43200640 1 -- 172.21.15.116:0/3575791900 wait complete. 2024-03-04T07:44:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 Processor -- start 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 -- start start 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd3c1036f0 0x7fdd3c199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1048e0 0x7fdd3c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 0x7fdd3c1a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c10f370 con 0x7fdd3c1052a0 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c10f1f0 con 0x7fdd3c1048e0 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd43200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c10f4f0 con 0x7fdd3c1036f0 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd41400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1048e0 0x7fdd3c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd41400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1048e0 0x7fdd3c199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49846/0 (socket says 172.21.15.116:49846) 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd42800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 0x7fdd3c1a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd41400640 1 -- 172.21.15.116:0/886165108 learned_addr learned my addr 172.21.15.116:0/886165108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.580+0000 7fdd41e00640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd3c1036f0 0x7fdd3c199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd41400640 1 -- 172.21.15.116:0/886165108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd3c1036f0 msgr2=0x7fdd3c199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd41400640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd3c1036f0 0x7fdd3c199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd41400640 1 -- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 msgr2=0x7fdd3c1a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd41400640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 0x7fdd3c1a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd41400640 1 -- 172.21.15.116:0/886165108 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c1a15a0 con 0x7fdd3c1048e0 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd42800640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 0x7fdd3c1a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd41400640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1048e0 0x7fdd3c199ec0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800e9f0 tx=0x7fdd2800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd36a00640 1 -- 172.21.15.116:0/886165108 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800cd80 con 0x7fdd3c1048e0 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd36a00640 1 -- 172.21.15.116:0/886165108 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd28005500 con 0x7fdd3c1048e0 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd36a00640 1 -- 172.21.15.116:0/886165108 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28010690 con 0x7fdd3c1048e0 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c1a1890 con 0x7fdd3c1048e0 2024-03-04T07:44:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.581+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c1a1d90 con 0x7fdd3c1048e0 2024-03-04T07:44:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.584+0000 7fdd36a00640 1 -- 172.21.15.116:0/886165108 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fdd280046c0 con 0x7fdd3c1048e0 2024-03-04T07:44:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.584+0000 7fdd47616740 1 Processor -- start 2024-03-04T07:44:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd47616740 1 -- start start 2024-03-04T07:44:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd47616740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd47616740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7a87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd47616740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd47616740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fa6b6e20 con 0x5628fa7a32b0 2024-03-04T07:44:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd47616740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fa708a20 con 0x5628fa7a3ca0 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd47616740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fa6b1b00 con 0x5628fa7a8e10 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd42800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd41e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7a87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd42800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7a36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44760/0 (socket says 172.21.15.116:44760) 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd42800640 1 -- 172.21.15.116:0/334486785 learned_addr learned my addr 172.21.15.116:0/334486785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.585+0000 7fdd41400640 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd41e00640 1 -- 172.21.15.116:0/334486785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 msgr2=0x5628fa7ab2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd41e00640 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7ab2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd41e00640 1 -- 172.21.15.116:0/334486785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 msgr2=0x5628fa7a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd41e00640 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd41e00640 1 -- 172.21.15.116:0/334486785 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628fa6c6e10 con 0x5628fa7a3ca0 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd41e00640 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7a87a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009bf0 tx=0x7fdd30011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a822e5b875bfedb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd36a00640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd240a1d80 0x7fdd240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd2f000640 1 -- 172.21.15.116:0/334486785 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30004050 con 0x5628fa7a3ca0 2024-03-04T07:44:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd2f000640 1 -- 172.21.15.116:0/334486785 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd30005500 con 0x5628fa7a3ca0 2024-03-04T07:44:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd2f000640 1 -- 172.21.15.116:0/334486785 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30004de0 con 0x5628fa7a3ca0 2024-03-04T07:44:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd36a00640 1 -- 172.21.15.116:0/886165108 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdd280ce1a0 con 0x7fdd3c1048e0 2024-03-04T07:44:22.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.586+0000 7fdd42800640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd240a1d80 0x7fdd240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 -- 172.21.15.116:0/334486785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 msgr2=0x5628fa7a87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7a87a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009bf0 tx=0x7fdd30011810 comp rx=0 tx=0).stop 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 -- 172.21.15.116:0/334486785 shutdown_connections 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7a87a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 --2- 172.21.15.116:0/334486785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 -- 172.21.15.116:0/334486785 >> 172.21.15.116:0/334486785 conn(0x5628fa79e820 msgr2=0x5628fa79ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd42800640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd240a1d80 0x7fdd240a4240 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdd38032420 tx=0x7fdd38032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 -- 172.21.15.116:0/334486785 shutdown_connections 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.587+0000 7fdd47616740 1 -- 172.21.15.116:0/334486785 wait complete. 2024-03-04T07:44:22.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 -- start start 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 Processor -- start 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a32b0 0x5628fa7ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a3ca0 0x5628fa7adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fa6c4f40 con 0x5628fa7a3ca0 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fa704ac0 con 0x5628fa7a32b0 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fa7abdc0 con 0x5628fa7a8e10 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd47616740 10 client.0 _ll_register_callbacks cb 0x5628fa703aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd41e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a32b0 0x5628fa7ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd41400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a3ca0 0x5628fa7adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd41e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a32b0 0x5628fa7ad6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49858/0 (socket says 172.21.15.116:49858) 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.588+0000 7fdd41e00640 1 -- 172.21.15.116:0/845791996 learned_addr learned my addr 172.21.15.116:0/845791996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd42800640 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd41400640 1 -- 172.21.15.116:0/845791996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 msgr2=0x5628fa7b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd41400640 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd41400640 1 -- 172.21.15.116:0/845791996 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a32b0 msgr2=0x5628fa7ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd41400640 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a32b0 0x5628fa7ad6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd41400640 1 -- 172.21.15.116:0/845791996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628fa4e57e0 con 0x5628fa7a3ca0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd41400640 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a3ca0 0x5628fa7adbf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdd2809ae60 tx=0x7fdd2809ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640cf6468dc6f2c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd23400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd23400640 1 -- 172.21.15.116:0/845791996 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd280a33e0 con 0x5628fa7a3ca0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd23400640 1 -- 172.21.15.116:0/845791996 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdd2809a320 con 0x5628fa7a3ca0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.589+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd30005500 con 0x5628fa7a3ca0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd23400640 1 -- 172.21.15.116:0/845791996 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd2809aed0 con 0x5628fa7a3ca0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 10 client.16026 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5628fa545f70 con 0x5628fa7a3ca0 2024-03-04T07:44:22.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd23400640 1 -- 172.21.15.116:0/845791996 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd30005500 con 0x5628fa7a3ca0 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd23400640 1 -- 172.21.15.116:0/845791996 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdd280b91e0 con 0x5628fa7a3ca0 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 10 client.16026 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 10 client.16026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 20 client.16026 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa7b94d0 0x5628fa7bb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 4 client.16026 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.590+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628fa715f60 con 0x5628fa7b94d0 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.591+0000 7fdd41e00640 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa7b94d0 0x5628fa7bb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.591+0000 7fdd41e00640 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa7b94d0 0x5628fa7bb920 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a374f0fc174224c server_cookie=36afd3c5483202fc in_seq=0 out_seq=0 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.591+0000 7fdd23400640 10 client.16026 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd23400640 1 -- 172.21.15.116:0/845791996 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5628fa715f60 con 0x5628fa7b94d0 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd23400640 10 client.16026 handle_command_reply: tid=0 2024-03-04T07:44:22.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 client.16026 shutdown 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a3ca0 msgr2=0x5628fa7adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a3ca0 0x5628fa7adbf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdd2809ae60 tx=0x7fdd2809ad60 comp rx=0 tx=0).stop 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 shutdown_connections 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa7b94d0 0x5628fa7bb920 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a8e10 0x5628fa7b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a3ca0 0x5628fa7adbf0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd23400640 0 client.16026 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 --2- 172.21.15.116:0/845791996 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a32b0 0x5628fa7ad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 >> 172.21.15.116:0/845791996 conn(0x5628fa79e820 msgr2=0x5628fa79da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 shutdown_connections 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 1 -- 172.21.15.116:0/845791996 wait complete. 2024-03-04T07:44:22.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.592+0000 7fdd47616740 20 client.16026 trim_cache size 0 max 16384 2024-03-04T07:44:22.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 Processor -- start 2024-03-04T07:44:22.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 -- start start 2024-03-04T07:44:22.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a32b0 0x5628fa7a2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a8e10 0x5628fa7c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1a8a00 con 0x5628fa7a8e10 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1aafe0 con 0x5628fa7a3ca0 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.610+0000 7fdd47616740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1a9000 con 0x5628fa7a32b0 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd42800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a8e10 0x5628fa7c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49864/0 (socket says 172.21.15.116:49864) 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 -- 172.21.15.116:0/196509870 learned_addr learned my addr 172.21.15.116:0/196509870 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41e00640 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a32b0 0x5628fa7a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 -- 172.21.15.116:0/196509870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a32b0 msgr2=0x5628fa7a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a32b0 0x5628fa7a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 -- 172.21.15.116:0/196509870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a8e10 msgr2=0x5628fa7c01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a8e10 0x5628fa7c01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 -- 172.21.15.116:0/196509870 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628fa3dd170 con 0x5628fa7a3ca0 2024-03-04T07:44:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd42800640 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a8e10 0x5628fa7c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd41400640 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b4bf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdd2809abd0 tx=0x7fdd2809add0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b78c4839d796c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd2e600640 1 -- 172.21.15.116:0/196509870 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd280a7070 con 0x5628fa7a3ca0 2024-03-04T07:44:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd2e600640 1 -- 172.21.15.116:0/196509870 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2809f020 con 0x5628fa7a3ca0 2024-03-04T07:44:22.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.611+0000 7fdd2e600640 1 -- 172.21.15.116:0/196509870 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2809a580 con 0x5628fa7a3ca0 2024-03-04T07:44:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 -- 172.21.15.116:0/196509870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 msgr2=0x5628fa7b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b4bf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdd2809abd0 tx=0x7fdd2809add0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 -- 172.21.15.116:0/196509870 shutdown_connections 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a8e10 0x5628fa7c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b4bf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 --2- 172.21.15.116:0/196509870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a32b0 0x5628fa7a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 -- 172.21.15.116:0/196509870 >> 172.21.15.116:0/196509870 conn(0x5628fa79e820 msgr2=0x5628fa7a1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 -- 172.21.15.116:0/196509870 shutdown_connections 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.612+0000 7fdd47616740 1 -- 172.21.15.116:0/196509870 wait complete. 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 -- start start 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 Processor -- start 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a8e10 0x5628fa7b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1a9660 con 0x5628fa7a32b0 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1aa980 con 0x5628fa7a8e10 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c1a9cc0 con 0x5628fa7a3ca0 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd47616740 10 client.0 _ll_register_callbacks cb 0x7fdd28009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd41e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd42800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a8e10 0x5628fa7b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd41e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7b2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44784/0 (socket says 172.21.15.116:44784) 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd42800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a8e10 0x5628fa7b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49870/0 (socket says 172.21.15.116:49870) 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.613+0000 7fdd41e00640 1 -- 172.21.15.116:0/123221730 learned_addr learned my addr 172.21.15.116:0/123221730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd41400640 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd41e00640 1 -- 172.21.15.116:0/123221730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a3ca0 msgr2=0x5628fa7b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd41e00640 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd41e00640 1 -- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a8e10 msgr2=0x5628fa7b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd41e00640 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a8e10 0x5628fa7b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd41e00640 1 -- 172.21.15.116:0/123221730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628fa7b46d0 con 0x5628fa7a32b0 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd42800640 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a8e10 0x5628fa7b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd41e00640 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7b2b90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009250 tx=0x7fdd30009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62eaf02050cb76f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd23400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd23400640 1 -- 172.21.15.116:0/123221730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30004450 con 0x5628fa7a32b0 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd23400640 1 -- 172.21.15.116:0/123221730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdd300045f0 con 0x5628fa7a32b0 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.614+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd2809f020 con 0x5628fa7a32b0 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd23400640 1 -- 172.21.15.116:0/123221730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd300035f0 con 0x5628fa7a32b0 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 10 client.16032 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5628fa8156d0 con 0x5628fa7a32b0 2024-03-04T07:44:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd23400640 1 -- 172.21.15.116:0/123221730 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2809f020 con 0x5628fa7a32b0 2024-03-04T07:44:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd23400640 1 -- 172.21.15.116:0/123221730 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdd3002e930 con 0x5628fa7a32b0 2024-03-04T07:44:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 10 client.16032 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 10 client.16032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 20 client.16032 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa815ee0 0x5628fa818300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 4 client.16032 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.615+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5628fa7102b0 con 0x5628fa815ee0 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.616+0000 7fdd41400640 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa815ee0 0x5628fa818300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.616+0000 7fdd41400640 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa815ee0 0x5628fa818300 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=935aaced7eb5e7d8 server_cookie=398bb94fa2383f74 in_seq=0 out_seq=0 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.616+0000 7fdd23400640 10 client.16032 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd23400640 1 -- 172.21.15.116:0/123221730 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5628fa7102b0 con 0x5628fa815ee0 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd23400640 10 client.16032 handle_command_reply: tid=0 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 client.16032 shutdown 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 msgr2=0x5628fa7b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7b2b90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009250 tx=0x7fdd30009fb0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 shutdown_connections 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5628fa815ee0 0x5628fa818300 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5628fa7a8e10 0x5628fa7b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5628fa7a3ca0 0x5628fa7b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 --2- 172.21.15.116:0/123221730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628fa7a32b0 0x5628fa7b2b90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd23400640 0 client.16032 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 >> 172.21.15.116:0/123221730 conn(0x5628fa79e820 msgr2=0x5628fa7a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 shutdown_connections 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 1 -- 172.21.15.116:0/123221730 wait complete. 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.617+0000 7fdd47616740 20 client.16032 trim_cache size 0 max 16384 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:44:22.625 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "c5806475-9768-478e-8ec3-99541eecba99", 2024-03-04T07:44:22.626 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:44:22.626 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd240a1d80 msgr2=0x7fdd240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd240a1d80 0x7fdd240a4240 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdd38032420 tx=0x7fdd38032590 comp rx=0 tx=0).stop 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1048e0 msgr2=0x7fdd3c199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1048e0 0x7fdd3c199ec0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800e9f0 tx=0x7fdd2800edf0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 shutdown_connections 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd3c1052a0 0x7fdd3c1a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdd3c1048e0 0x7fdd3c199ec0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdd240a1d80 0x7fdd240a4240 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 --2- 172.21.15.116:0/886165108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd3c1036f0 0x7fdd3c199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 >> 172.21.15.116:0/886165108 conn(0x7fdd3c0fec80 msgr2=0x7fdd3c107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 shutdown_connections 2024-03-04T07:44:22.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.618+0000 7fdd43200640 1 -- 172.21.15.116:0/886165108 wait complete. 2024-03-04T07:44:22.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:44:22.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:44:22.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 -- 172.21.15.116:0/1781695446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 msgr2=0x7f68ac101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 --2- 172.21.15.116:0/1781695446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac101a00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f689c009860 tx=0x7f689c02bb70 comp rx=0 tx=0).stop 2024-03-04T07:44:22.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 -- 172.21.15.116:0/1781695446 shutdown_connections 2024-03-04T07:44:22.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 --2- 172.21.15.116:0/1781695446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68ac103190 0x7f68ac109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 --2- 172.21.15.116:0/1781695446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68ac1027d0 0x7f68ac102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 --2- 172.21.15.116:0/1781695446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac101a00 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 -- 172.21.15.116:0/1781695446 >> 172.21.15.116:0/1781695446 conn(0x7f68ac0fcb70 msgr2=0x7f68ac0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 -- 172.21.15.116:0/1781695446 shutdown_connections 2024-03-04T07:44:22.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.839+0000 7f68b1200640 1 -- 172.21.15.116:0/1781695446 wait complete. 2024-03-04T07:44:22.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 Processor -- start 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 -- start start 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68ac1027d0 0x7f68ac19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68ac103190 0x7f68ac1a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac10d230 con 0x7f68ac103190 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac10d0b0 con 0x7f68ac1015e0 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68b1200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac10d3b0 con 0x7f68ac1027d0 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.840+0000 7f68a9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68aa800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68ac103190 0x7f68ac1a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49876/0 (socket says 172.21.15.116:49876) 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 -- 172.21.15.116:0/465133970 learned_addr learned my addr 172.21.15.116:0/465133970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9400640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68ac1027d0 0x7f68ac19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 -- 172.21.15.116:0/465133970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68ac1027d0 msgr2=0x7f68ac19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68ac1027d0 0x7f68ac19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 -- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68ac103190 msgr2=0x7f68ac1a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68ac103190 0x7f68ac1a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 -- 172.21.15.116:0/465133970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac1a1760 con 0x7f68ac1015e0 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a9e00640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac199b40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f689c0349e0 tx=0x7f689c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a2a00640 1 -- 172.21.15.116:0/465133970 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c041020 con 0x7f68ac1015e0 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a2a00640 1 -- 172.21.15.116:0/465133970 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f689c0323b0 con 0x7f68ac1015e0 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac1a19f0 con 0x7f68ac1015e0 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.841+0000 7f68a2a00640 1 -- 172.21.15.116:0/465133970 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c039040 con 0x7f68ac1015e0 2024-03-04T07:44:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.842+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac1a1ee0 con 0x7f68ac1015e0 2024-03-04T07:44:22.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.844+0000 7f68a2a00640 1 -- 172.21.15.116:0/465133970 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f689c044020 con 0x7f68ac1015e0 2024-03-04T07:44:22.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.845+0000 7f68b3e2b740 1 Processor -- start 2024-03-04T07:44:22.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.845+0000 7f68b3e2b740 1 -- start start 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155af0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155af6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155af8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a155a1e090 con 0x55a155af1540 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a155a55d30 con 0x55a155af66b0 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1559fea30 con 0x55a155af0b50 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a2a00640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f68900a1d80 0x7f68900a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155af6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155af8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155af8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49884/0 (socket says 172.21.15.116:49884) 2024-03-04T07:44:22.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9400640 1 -- 172.21.15.116:0/270159504 learned_addr learned my addr 172.21.15.116:0/270159504 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68aa800640 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155af0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a2a00640 1 -- 172.21.15.116:0/465133970 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f689c0ef770 con 0x7f68ac1015e0 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68aa800640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f68900a1d80 0x7f68900a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9400640 1 -- 172.21.15.116:0/270159504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 msgr2=0x55a155af0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9400640 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155af0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9400640 1 -- 172.21.15.116:0/270159504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 msgr2=0x55a155af6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.846+0000 7f68a9400640 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155af6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.847+0000 7f68a9400640 1 -- 172.21.15.116:0/270159504 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1558347d0 con 0x55a155af66b0 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.847+0000 7f68a9e00640 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155af6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.847+0000 7f68a9400640 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155af8b40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f689800bce0 tx=0x7f689800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffd0d479cf1bb1ce server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.847+0000 7f68aa800640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f68900a1d80 0x7f68900a4240 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55a155af1da0 tx=0x7f68a40095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.847+0000 7f6897400640 1 -- 172.21.15.116:0/270159504 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898015070 con 0x55a155af66b0 2024-03-04T07:44:22.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.847+0000 7f6897400640 1 -- 172.21.15.116:0/270159504 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f689801a940 con 0x55a155af66b0 2024-03-04T07:44:22.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.847+0000 7f6897400640 1 -- 172.21.15.116:0/270159504 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68980119c0 con 0x55a155af66b0 2024-03-04T07:44:22.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 -- 172.21.15.116:0/270159504 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 msgr2=0x55a155af8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155af8b40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f689800bce0 tx=0x7f689800f730 comp rx=0 tx=0).stop 2024-03-04T07:44:22.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 -- 172.21.15.116:0/270159504 shutdown_connections 2024-03-04T07:44:22.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155af8b40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155af6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/270159504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155af0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 -- 172.21.15.116:0/270159504 >> 172.21.15.116:0/270159504 conn(0x55a155aec0c0 msgr2=0x55a155aec4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 -- 172.21.15.116:0/270159504 shutdown_connections 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 -- 172.21.15.116:0/270159504 wait complete. 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.848+0000 7f68b3e2b740 1 -- start start 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 1 Processor -- start 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af0b50 0x55a155afaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155afb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af66b0 0x55a155b02450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1559f40c0 con 0x55a155af1540 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a155a51dd0 con 0x55a155af0b50 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a155af9660 con 0x55a155af66b0 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68b3e2b740 10 client.0 _ll_register_callbacks cb 0x55a155a11860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:22.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68a9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af0b50 0x55a155afaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68a9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af0b50 0x55a155afaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49892/0 (socket says 172.21.15.116:49892) 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68a9e00640 1 -- 172.21.15.116:0/1162191117 learned_addr learned my addr 172.21.15.116:0/1162191117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.849+0000 7f68a9400640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155afb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68aa800640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af66b0 0x55a155b02450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68a9e00640 1 -- 172.21.15.116:0/1162191117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af66b0 msgr2=0x55a155b02450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68a9e00640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af66b0 0x55a155b02450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68a9e00640 1 -- 172.21.15.116:0/1162191117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 msgr2=0x55a155afb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68a9e00640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155afb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68a9e00640 1 -- 172.21.15.116:0/1162191117 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a15576a3f0 con 0x55a155af0b50 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68aa800640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af66b0 0x55a155b02450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68a9e00640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af0b50 0x55a155afaed0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f689c034940 tx=0x7f689c08b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1350b611627feb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f688be00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f688be00640 1 -- 172.21.15.116:0/1162191117 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c0bc230 con 0x55a155af0b50 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f688be00640 1 -- 172.21.15.116:0/1162191117 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f689c0326d0 con 0x55a155af0b50 2024-03-04T07:44:22.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f689801a940 con 0x55a155af0b50 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f688be00640 1 -- 172.21.15.116:0/1162191117 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f689c0bbdc0 con 0x55a155af0b50 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68b3e2b740 10 client.26165 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.850+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a15572a0f0 con 0x55a155af0b50 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f688be00640 1 -- 172.21.15.116:0/1162191117 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f689801a940 con 0x55a155af0b50 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f688be00640 1 -- 172.21.15.116:0/1162191117 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f689c03c030 con 0x55a155af0b50 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f68b3e2b740 10 client.26165 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f68b3e2b740 10 client.26165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f68b3e2b740 20 client.26165 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b06940 0x55a155b08db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f68b3e2b740 4 client.26165 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a155a63270 con 0x55a155b06940 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.851+0000 7f68a9400640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b06940 0x55a155b08db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.852+0000 7f68a9400640 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b06940 0x55a155b08db0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=151af820e6c4f3d7 server_cookie=6c54e01b3f38502b in_seq=0 out_seq=0 2024-03-04T07:44:22.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.852+0000 7f688be00640 10 client.26165 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f688be00640 1 -- 172.21.15.116:0/1162191117 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a155a63270 con 0x55a155b06940 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f688be00640 10 client.26165 handle_command_reply: tid=0 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 client.26165 shutdown 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af0b50 msgr2=0x55a155afaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af0b50 0x55a155afaed0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f689c034940 tx=0x7f689c08b650 comp rx=0 tx=0).stop 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 shutdown_connections 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b06940 0x55a155b08db0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af66b0 0x55a155b02450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155afb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/1162191117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af0b50 0x55a155afaed0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f688be00640 0 client.26165 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 >> 172.21.15.116:0/1162191117 conn(0x55a155aec0c0 msgr2=0x55a155aeb360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 shutdown_connections 2024-03-04T07:44:22.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 1 -- 172.21.15.116:0/1162191117 wait complete. 2024-03-04T07:44:22.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.853+0000 7f68b3e2b740 20 client.26165 trim_cache size 0 max 16384 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.870+0000 7f68b3e2b740 1 Processor -- start 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68b3e2b740 1 -- start start 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af0b50 0x55a155afb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af1540 0x55a155b02410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155aefa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1a8ca0 con 0x55a155af0b50 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1ab280 con 0x55a155af66b0 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1a92a0 con 0x55a155af1540 2024-03-04T07:44:22.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68aa800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155aefa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68aa800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155aefa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49896/0 (socket says 172.21.15.116:49896) 2024-03-04T07:44:22.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68aa800640 1 -- 172.21.15.116:0/623439884 learned_addr learned my addr 172.21.15.116:0/623439884 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68a9400640 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af1540 0x55a155b02410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.871+0000 7f68a9e00640 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af0b50 0x55a155afb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68aa800640 1 -- 172.21.15.116:0/623439884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af1540 msgr2=0x55a155b02410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68aa800640 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af1540 0x55a155b02410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68aa800640 1 -- 172.21.15.116:0/623439884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af0b50 msgr2=0x55a155afb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68aa800640 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af0b50 0x55a155afb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68aa800640 1 -- 172.21.15.116:0/623439884 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a155af1920 con 0x55a155af66b0 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68a9400640 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af1540 0x55a155b02410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68a9e00640 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af0b50 0x55a155afb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f68aa800640 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155aefa30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f68a4003a20 tx=0x7f68a4008000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22fda9a6eb23e4d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f6896a00640 1 -- 172.21.15.116:0/623439884 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a4004050 con 0x55a155af66b0 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f6896a00640 1 -- 172.21.15.116:0/623439884 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68a402c070 con 0x55a155af66b0 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.872+0000 7f6896a00640 1 -- 172.21.15.116:0/623439884 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a4027d10 con 0x55a155af66b0 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 -- 172.21.15.116:0/623439884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 msgr2=0x55a155aefa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155aefa30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f68a4003a20 tx=0x7f68a4008000 comp rx=0 tx=0).stop 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 -- 172.21.15.116:0/623439884 shutdown_connections 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155aefa30 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af1540 0x55a155b02410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/623439884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af0b50 0x55a155afb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 -- 172.21.15.116:0/623439884 >> 172.21.15.116:0/623439884 conn(0x55a155aec0c0 msgr2=0x55a155aef320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 -- 172.21.15.116:0/623439884 shutdown_connections 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 -- 172.21.15.116:0/623439884 wait complete. 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.873+0000 7f68b3e2b740 1 -- start start 2024-03-04T07:44:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 1 Processor -- start 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155b00570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155b00ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155b01900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1a9900 con 0x55a155af1540 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1aac20 con 0x55a155af66b0 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1a9f60 con 0x55a155af0b50 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68b3e2b740 10 client.0 _ll_register_callbacks cb 0x7f689c0044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68a9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155b00ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68a9400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155b00ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44830/0 (socket says 172.21.15.116:44830) 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68a9400640 1 -- 172.21.15.116:0/2329964594 learned_addr learned my addr 172.21.15.116:0/2329964594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68aa800640 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155b01900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.874+0000 7f68a9e00640 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155b00570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f68a9400640 1 -- 172.21.15.116:0/2329964594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 msgr2=0x55a155b00570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f68a9400640 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155b00570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f68a9400640 1 -- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 msgr2=0x55a155b01900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f68a9400640 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155b01900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f68a9400640 1 -- 172.21.15.116:0/2329964594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a155b01fa0 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f68a9400640 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155b00ab0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f689800bce0 tx=0x7f6898005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bfcef27e5b5036 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f688be00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f688be00640 1 -- 172.21.15.116:0/2329964594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898015040 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68a402c070 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.875+0000 7f688be00640 1 -- 172.21.15.116:0/2329964594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f6898005040 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f688be00640 1 -- 172.21.15.116:0/2329964594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68980282f0 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 10 client.16050 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a155b6b3e0 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f688be00640 1 -- 172.21.15.116:0/2329964594 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68a402c070 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f688be00640 1 -- 172.21.15.116:0/2329964594 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6898004680 con 0x55a155af1540 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 10 client.16050 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 20 client.16050 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b6bc80 0x55a155b6e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 4 client.16050 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a155a5d5c0 con 0x55a155b6bc80 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.876+0000 7f68a9e00640 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b6bc80 0x55a155b6e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68a9e00640 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b6bc80 0x55a155b6e0a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82fb1944da85ad32 server_cookie=64ab90556474e713 in_seq=0 out_seq=0 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f688be00640 10 client.16050 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f688be00640 1 -- 172.21.15.116:0/2329964594 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a155a5d5c0 con 0x55a155b6bc80 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f688be00640 10 client.16050 handle_command_reply: tid=0 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 client.16050 shutdown 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 msgr2=0x55a155b00ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155b00ab0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f689800bce0 tx=0x7f6898005ed0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 shutdown_connections 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a155b6bc80 0x55a155b6e0a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a155af66b0 0x55a155b01900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a155af1540 0x55a155b00ab0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 --2- 172.21.15.116:0/2329964594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a155af0b50 0x55a155b00570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f688be00640 0 client.16050 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.877+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 >> 172.21.15.116:0/2329964594 conn(0x55a155aec0c0 msgr2=0x55a155afb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 shutdown_connections 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b3e2b740 1 -- 172.21.15.116:0/2329964594 wait complete. 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b3e2b740 20 client.16050 trim_cache size 0 max 16384 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:44:22.883 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:22.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f68900a1d80 msgr2=0x7f68900a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f68900a1d80 0x7f68900a4240 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55a155af1da0 tx=0x7f68a40095c0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 msgr2=0x7f68ac199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:22.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac199b40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f689c0349e0 tx=0x7f689c0042f0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 shutdown_connections 2024-03-04T07:44:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68ac103190 0x7f68ac1a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f68900a1d80 0x7f68900a4240 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68ac1027d0 0x7f68ac19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 --2- 172.21.15.116:0/465133970 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f68ac1015e0 0x7f68ac199b40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 >> 172.21.15.116:0/465133970 conn(0x7f68ac0fcb70 msgr2=0x7f68ac107f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.878+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 shutdown_connections 2024-03-04T07:44:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:22.879+0000 7f68b1200640 1 -- 172.21.15.116:0/465133970 wait complete. 2024-03-04T07:44:22.896 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:44:22.897 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:44:22.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:22 smithi116 ceph-mon[32150]: pgmap v509: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:44:22.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:22 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:44:22.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:22 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:22.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:22 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:44:22.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:22 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:22.970 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:22 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:44:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:22 smithi133 ceph-mon[36505]: pgmap v509: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:44:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:22 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:44:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:22 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:22 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:44:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:22 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:22 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:44:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.100+0000 7f4f3d200640 1 -- 172.21.15.116:0/2723273349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38075610 msgr2=0x7f4f38073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.100+0000 7f4f3d200640 1 --2- 172.21.15.116:0/2723273349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38075610 0x7f4f38073a60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009930 tx=0x7f4f2802bc20 comp rx=0 tx=0).stop 2024-03-04T07:44:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.100+0000 7f4f3d200640 1 -- 172.21.15.116:0/2723273349 shutdown_connections 2024-03-04T07:44:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.100+0000 7f4f3d200640 1 --2- 172.21.15.116:0/2723273349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f3810dbc0 0x7f4f3810ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.100+0000 7f4f3d200640 1 --2- 172.21.15.116:0/2723273349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f38074030 0x7f4f380744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.100+0000 7f4f3d200640 1 --2- 172.21.15.116:0/2723273349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38075610 0x7f4f38073a60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.100+0000 7f4f3d200640 1 -- 172.21.15.116:0/2723273349 >> 172.21.15.116:0/2723273349 conn(0x7f4f380fcb80 msgr2=0x7f4f380fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.101+0000 7f4f3d200640 1 -- 172.21.15.116:0/2723273349 shutdown_connections 2024-03-04T07:44:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.101+0000 7f4f3d200640 1 -- 172.21.15.116:0/2723273349 wait complete. 2024-03-04T07:44:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.101+0000 7f4f3d200640 1 Processor -- start 2024-03-04T07:44:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f3d200640 1 -- start start 2024-03-04T07:44:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f3d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38074030 0x7f4f3819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f3d200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f38075610 0x7f4f3819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f3d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f3810dbc0 0x7f4f381a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f3d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38113890 con 0x7f4f3810dbc0 2024-03-04T07:44:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f3d200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38113710 con 0x7f4f38074030 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f3d200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38113a10 con 0x7f4f38075610 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f35e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38074030 0x7f4f3819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f35400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f38075610 0x7f4f3819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f35e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38074030 0x7f4f3819df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49914/0 (socket says 172.21.15.116:49914) 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f35e00640 1 -- 172.21.15.116:0/4261968255 learned_addr learned my addr 172.21.15.116:0/4261968255 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.102+0000 7f4f36800640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f3810dbc0 0x7f4f381a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f35e00640 1 -- 172.21.15.116:0/4261968255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f38075610 msgr2=0x7f4f3819e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f35e00640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f38075610 0x7f4f3819e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f35e00640 1 -- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f3810dbc0 msgr2=0x7f4f381a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f35e00640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f3810dbc0 0x7f4f381a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f35e00640 1 -- 172.21.15.116:0/4261968255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f381a5b60 con 0x7f4f38074030 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f35400640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f38075610 0x7f4f3819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f35e00640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38074030 0x7f4f3819df40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4f28000c00 tx=0x7f4f28005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f2ea00640 1 -- 172.21.15.116:0/4261968255 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28043020 con 0x7f4f38074030 2024-03-04T07:44:23.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f2ea00640 1 -- 172.21.15.116:0/4261968255 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f280054e0 con 0x7f4f38074030 2024-03-04T07:44:23.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f381a5df0 con 0x7f4f38074030 2024-03-04T07:44:23.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f2ea00640 1 -- 172.21.15.116:0/4261968255 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28003d10 con 0x7f4f38074030 2024-03-04T07:44:23.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.103+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f381a62b0 con 0x7f4f38074030 2024-03-04T07:44:23.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.104+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38103d90 con 0x7f4f38074030 2024-03-04T07:44:23.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.106+0000 7f4f2ea00640 1 -- 172.21.15.116:0/4261968255 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4f28046020 con 0x7f4f38074030 2024-03-04T07:44:23.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.107+0000 7f4f2ea00640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f1c0a1e00 0x7f4f1c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.108+0000 7f4f2ea00640 1 -- 172.21.15.116:0/4261968255 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4f280efec0 con 0x7f4f38074030 2024-03-04T07:44:23.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.108+0000 7f4f35400640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f1c0a1e00 0x7f4f1c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.108+0000 7f4f35400640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f1c0a1e00 0x7f4f1c0a42c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f3819f450 tx=0x7f4f2400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.111+0000 7f4f2ea00640 1 -- 172.21.15.116:0/4261968255 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4f2804d020 con 0x7f4f38074030 2024-03-04T07:44:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:22 smithi152 ceph-mon[37154]: pgmap v509: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:44:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:22 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:44:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:22 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:22 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:44:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:22 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:22 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:44:23.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.348+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f38062200 con 0x7f4f38074030 2024-03-04T07:44:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.350+0000 7f4f2ea00640 1 -- 172.21.15.116:0/4261968255 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f4f280bd2b0 con 0x7f4f38074030 2024-03-04T07:44:23.351 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:44:23.351 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:44:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:44:23.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f1c0a1e00 msgr2=0x7f4f1c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f1c0a1e00 0x7f4f1c0a42c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4f3819f450 tx=0x7f4f2400b390 comp rx=0 tx=0).stop 2024-03-04T07:44:23.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38074030 msgr2=0x7f4f3819df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38074030 0x7f4f3819df40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4f28000c00 tx=0x7f4f28005d50 comp rx=0 tx=0).stop 2024-03-04T07:44:23.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 shutdown_connections 2024-03-04T07:44:23.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f3810dbc0 0x7f4f381a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f1c0a1e00 0x7f4f1c0a42c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f38075610 0x7f4f3819e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 --2- 172.21.15.116:0/4261968255 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f38074030 0x7f4f3819df40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 >> 172.21.15.116:0/4261968255 conn(0x7f4f380fcb80 msgr2=0x7f4f38100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 shutdown_connections 2024-03-04T07:44:23.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.352+0000 7f4f3d200640 1 -- 172.21.15.116:0/4261968255 wait complete. 2024-03-04T07:44:23.373 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:44:23.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:44:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.586+0000 7f9ec6c00640 1 -- 172.21.15.116:0/2948693376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 msgr2=0x7f9ec0109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.586+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/2948693376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec0109160 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009900 tx=0x7f9eac02bc30 comp rx=0 tx=0).stop 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.587+0000 7f9ec6c00640 1 -- 172.21.15.116:0/2948693376 shutdown_connections 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.587+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/2948693376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01096a0 0x7f9ec010ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.587+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/2948693376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec0109160 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.587+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/2948693376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ec0107af0 0x7f9ec0107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.587+0000 7f9ec6c00640 1 -- 172.21.15.116:0/2948693376 >> 172.21.15.116:0/2948693376 conn(0x7f9ec0076aa0 msgr2=0x7f9ec0078ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.587+0000 7f9ec6c00640 1 -- 172.21.15.116:0/2948693376 shutdown_connections 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.587+0000 7f9ec6c00640 1 -- 172.21.15.116:0/2948693376 wait complete. 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 Processor -- start 2024-03-04T07:44:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 -- start start 2024-03-04T07:44:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ec0107af0 0x7f9ec019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec019e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01096a0 0x7f9ec01a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01137c0 con 0x7f9ec01096a0 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0113640 con 0x7f9ec0108ce0 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0113940 con 0x7f9ec0107af0 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec6200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01096a0 0x7f9ec01a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec4e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec019e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec5800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ec0107af0 0x7f9ec019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec4e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec019e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49932/0 (socket says 172.21.15.116:49932) 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.588+0000 7f9ec4e00640 1 -- 172.21.15.116:0/820333777 learned_addr learned my addr 172.21.15.116:0/820333777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ec4e00640 1 -- 172.21.15.116:0/820333777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ec0107af0 msgr2=0x7f9ec019ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ec0107af0 0x7f9ec019ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ec4e00640 1 -- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01096a0 msgr2=0x7f9ec01a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01096a0 0x7f9ec01a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ec4e00640 1 -- 172.21.15.116:0/820333777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec01a59f0 con 0x7f9ec0108ce0 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec019e310 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9eac005ab0 tx=0x7f9eac02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ebe000640 1 -- 172.21.15.116:0/820333777 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac043020 con 0x7f9ec0108ce0 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ebe000640 1 -- 172.21.15.116:0/820333777 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eac0054e0 con 0x7f9ec0108ce0 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ebe000640 1 -- 172.21.15.116:0/820333777 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac003ce0 con 0x7f9ec0108ce0 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.589+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec01a5c80 con 0x7f9ec0108ce0 2024-03-04T07:44:23.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.590+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec01a61d0 con 0x7f9ec0108ce0 2024-03-04T07:44:23.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.592+0000 7f9ecaf66740 1 Processor -- start 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ebe000640 1 -- 172.21.15.116:0/820333777 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9eac046020 con 0x7f9ec0108ce0 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ecaf66740 1 -- start start 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083d3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083d8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d89d0 0x561a083dae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a08337d60 con 0x561a083d89d0 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a082e0cf0 con 0x561a083d2e70 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a08333e00 con 0x561a083d3860 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ec6200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083d3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ec6200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083d3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49944/0 (socket says 172.21.15.116:49944) 2024-03-04T07:44:23.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ec4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d89d0 0x561a083dae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ec5800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083d8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.593+0000 7f9ec6200640 1 -- 172.21.15.116:0/2628218464 learned_addr learned my addr 172.21.15.116:0/2628218464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ec6200640 1 -- 172.21.15.116:0/2628218464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 msgr2=0x561a083d8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ec6200640 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083d8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ec6200640 1 -- 172.21.15.116:0/2628218464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d89d0 msgr2=0x561a083dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ec6200640 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d89d0 0x561a083dae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ec6200640 1 -- 172.21.15.116:0/2628218464 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a0804c3f0 con 0x561a083d2e70 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ec5800640 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083d8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ec6200640 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083d3290 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8010370 tx=0x7f9eb8010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a333dd6cb764845 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9eb2a00640 1 -- 172.21.15.116:0/2628218464 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8016070 con 0x561a083d2e70 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9eb2a00640 1 -- 172.21.15.116:0/2628218464 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eb8004d20 con 0x561a083d2e70 2024-03-04T07:44:23.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ebe000640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9ea80a1e00 0x7f9ea80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9eb2a00640 1 -- 172.21.15.116:0/2628218464 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8005480 con 0x561a083d2e70 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.594+0000 7f9ebe000640 1 -- 172.21.15.116:0/820333777 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9eac0efda0 con 0x7f9ec0108ce0 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ec5800640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9ea80a1e00 0x7f9ea80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2628218464 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 msgr2=0x561a083d3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083d3290 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8010370 tx=0x7f9eb8010830 comp rx=0 tx=0).stop 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2628218464 shutdown_connections 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d89d0 0x561a083dae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083d8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2628218464 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083d3290 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2628218464 >> 172.21.15.116:0/2628218464 conn(0x561a083ce3e0 msgr2=0x561a083ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2628218464 shutdown_connections 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2628218464 wait complete. 2024-03-04T07:44:23.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.595+0000 7f9ec5800640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9ea80a1e00 0x7f9ea80a42c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x561a083d3c60 tx=0x7f9eb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 -- start start 2024-03-04T07:44:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 Processor -- start 2024-03-04T07:44:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083dd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083e4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a083db980 con 0x561a083d2e70 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a082d6220 con 0x561a083d89d0 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.596+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a083dbb00 con 0x561a083d3860 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ecaf66740 10 client.0 _ll_register_callbacks cb 0x561a0804bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083e4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083e4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49950/0 (socket says 172.21.15.116:49950) 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec4e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083dd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44884/0 (socket says 172.21.15.116:44884) 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 -- 172.21.15.116:0/49047180 learned_addr learned my addr 172.21.15.116:0/49047180 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 -- 172.21.15.116:0/49047180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 msgr2=0x561a083dd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083dd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 -- 172.21.15.116:0/49047180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d2e70 msgr2=0x561a083dd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 -- 172.21.15.116:0/49047180 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a081167d0 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ec6200640 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083e4980 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8009880 tx=0x7f9eb8004f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d89259db7e726cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.597+0000 7f9ea7400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ea7400640 1 -- 172.21.15.116:0/49047180 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8029060 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ea7400640 1 -- 172.21.15.116:0/49047180 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9eb8003e00 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9eb8004d20 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ea7400640 1 -- 172.21.15.116:0/49047180 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9eb8016070 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ecaf66740 10 client.26189 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561a083e7d40 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ea7400640 1 -- 172.21.15.116:0/49047180 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eb8004d20 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ea7400640 1 -- 172.21.15.116:0/49047180 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9eb802e070 con 0x561a083d89d0 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ecaf66740 10 client.26189 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ecaf66740 10 client.26189 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.598+0000 7f9ecaf66740 20 client.26189 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.599+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a083e9140 0x561a083eb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.599+0000 7f9ecaf66740 4 client.26189 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.599+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a083452a0 con 0x561a083e9140 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.599+0000 7f9ec5800640 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a083e9140 0x561a083eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.599+0000 7f9ec5800640 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a083e9140 0x561a083eb530 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55523e233f937ee1 server_cookie=6505c5a427cfbd46 in_seq=0 out_seq=0 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.599+0000 7f9ea7400640 10 client.26189 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.600+0000 7f9ea7400640 1 -- 172.21.15.116:0/49047180 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561a083452a0 con 0x561a083e9140 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.600+0000 7f9ea7400640 10 client.26189 handle_command_reply: tid=0 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.600+0000 7f9ecaf66740 1 client.26189 shutdown 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 msgr2=0x561a083e4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083e4980 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8009880 tx=0x7f9eb8004f40 comp rx=0 tx=0).stop 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 shutdown_connections 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a083e9140 0x561a083eb530 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083e4980 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d3860 0x561a083dd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/49047180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 >> 172.21.15.116:0/49047180 conn(0x561a083ce3e0 msgr2=0x561a083cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ea7400640 0 client.26189 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 shutdown_connections 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 1 -- 172.21.15.116:0/49047180 wait complete. 2024-03-04T07:44:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.601+0000 7f9ecaf66740 20 client.26189 trim_cache size 0 max 16384 2024-03-04T07:44:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.614+0000 7f9ecaf66740 1 Processor -- start 2024-03-04T07:44:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.614+0000 7f9ecaf66740 1 -- start start 2024-03-04T07:44:23.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083dd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01acdf0 con 0x561a083d3860 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01af3d0 con 0x561a083d89d0 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01ad3f0 con 0x561a083d2e70 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44888/0 (socket says 172.21.15.116:44888) 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec4e00640 1 -- 172.21.15.116:0/3678695021 learned_addr learned my addr 172.21.15.116:0/3678695021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec6200640 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083dd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec5800640 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec4e00640 1 -- 172.21.15.116:0/3678695021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d2e70 msgr2=0x561a083dd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec4e00640 1 -- 172.21.15.116:0/3678695021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 msgr2=0x561a083dd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.615+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083dd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.616+0000 7f9ec4e00640 1 -- 172.21.15.116:0/3678695021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a083cee30 con 0x561a083d3860 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.616+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e4940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0f0080 tx=0x7f9eac0f3350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ba335aa41bc9e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.616+0000 7f9eb2000640 1 -- 172.21.15.116:0/3678695021 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0b8d70 con 0x561a083d3860 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.616+0000 7f9eb2000640 1 -- 172.21.15.116:0/3678695021 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eac004c80 con 0x561a083d3860 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.616+0000 7f9eb2000640 1 -- 172.21.15.116:0/3678695021 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0bc430 con 0x561a083d3860 2024-03-04T07:44:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 -- 172.21.15.116:0/3678695021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 msgr2=0x561a083e4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e4940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0f0080 tx=0x7f9eac0f3350 comp rx=0 tx=0).stop 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 -- 172.21.15.116:0/3678695021 shutdown_connections 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d89d0 0x561a083dd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e4940 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/3678695021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d2e70 0x561a083dd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 -- 172.21.15.116:0/3678695021 >> 172.21.15.116:0/3678695021 conn(0x561a083ce3e0 msgr2=0x561a083d4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 -- 172.21.15.116:0/3678695021 shutdown_connections 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 -- 172.21.15.116:0/3678695021 wait complete. 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 -- start start 2024-03-04T07:44:23.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.617+0000 7f9ecaf66740 1 Processor -- start 2024-03-04T07:44:23.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083e2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ecaf66740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d89d0 0x561a083e38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01ada50 con 0x561a083d3860 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01aed70 con 0x561a083d2e70 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ecaf66740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01ae0b0 con 0x561a083d89d0 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ecaf66740 10 client.0 _ll_register_callbacks cb 0x7f9eac0f36f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083e2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083e2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49968/0 (socket says 172.21.15.116:49968) 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 -- 172.21.15.116:0/2814776115 learned_addr learned my addr 172.21.15.116:0/2814776115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec6200640 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d89d0 0x561a083e38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 -- 172.21.15.116:0/2814776115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d89d0 msgr2=0x561a083e38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d89d0 0x561a083e38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 -- 172.21.15.116:0/2814776115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 msgr2=0x561a083e2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.618+0000 7f9ec5800640 1 -- 172.21.15.116:0/2814776115 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a083e3ec0 con 0x561a083d2e70 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ec5800640 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083e2420 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x561a083d42d0 tx=0x7f9eb4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=232cb5e70c027127 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ea7400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ea7400640 1 -- 172.21.15.116:0/2814776115 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb402f070 con 0x561a083d2e70 2024-03-04T07:44:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9eac004c80 con 0x561a083d2e70 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ea7400640 1 -- 172.21.15.116:0/2814776115 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9eb401d870 con 0x561a083d2e70 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ea7400640 1 -- 172.21.15.116:0/2814776115 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9eb403b050 con 0x561a083d2e70 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ecaf66740 10 client.26195 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.619+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561a08444bc0 con 0x561a083d2e70 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ea7400640 1 -- 172.21.15.116:0/2814776115 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eac004c80 con 0x561a083d2e70 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ea7400640 1 -- 172.21.15.116:0/2814776115 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9eb40035f0 con 0x561a083d2e70 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ecaf66740 10 client.26195 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ecaf66740 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ecaf66740 20 client.26195 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a08445550 0x561a08447970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ecaf66740 4 client.26195 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:44:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561a0833f5f0 con 0x561a08445550 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a08445550 0x561a08447970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ec4e00640 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a08445550 0x561a08447970 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c98c08c4eb1e9e0 server_cookie=490b3309585bad1f in_seq=0 out_seq=0 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.620+0000 7f9ea7400640 10 client.26195 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ea7400640 1 -- 172.21.15.116:0/2814776115 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561a0833f5f0 con 0x561a08445550 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ea7400640 10 client.26195 handle_command_reply: tid=0 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 client.26195 shutdown 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 msgr2=0x561a083e2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083e2420 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x561a083d42d0 tx=0x7f9eb4017000 comp rx=0 tx=0).stop 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 shutdown_connections 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a08445550 0x561a08447970 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a083d89d0 0x561a083e38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a083d3860 0x561a083e2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 --2- 172.21.15.116:0/2814776115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a083d2e70 0x561a083e2420 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ea7400640 0 client.26195 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:23.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 >> 172.21.15.116:0/2814776115 conn(0x561a083ce3e0 msgr2=0x561a083dd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 shutdown_connections 2024-03-04T07:44:23.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 1 -- 172.21.15.116:0/2814776115 wait complete. 2024-03-04T07:44:23.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.621+0000 7f9ecaf66740 20 client.26195 trim_cache size 0 max 16384 2024-03-04T07:44:23.630 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:44:23.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9ea80a1e00 msgr2=0x7f9ea80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9ea80a1e00 0x7f9ea80a42c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x561a083d3c60 tx=0x7f9eb4005f90 comp rx=0 tx=0).stop 2024-03-04T07:44:23.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 msgr2=0x7f9ec019e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:23.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec019e310 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9eac005ab0 tx=0x7f9eac02c6c0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 shutdown_connections 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01096a0 0x7f9ec01a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9ec0108ce0 0x7f9ec019e310 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9ea80a1e00 0x7f9ea80a42c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 --2- 172.21.15.116:0/820333777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ec0107af0 0x7f9ec019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 >> 172.21.15.116:0/820333777 conn(0x7f9ec0076aa0 msgr2=0x7f9ec010bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 shutdown_connections 2024-03-04T07:44:23.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:23.622+0000 7f9ec6c00640 1 -- 172.21.15.116:0/820333777 wait complete. 2024-03-04T07:44:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:23 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4261968255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:24.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:23 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4261968255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:23 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4261968255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:24.642 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:44:24.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:44:24.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:44:24.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 -- 172.21.15.116:0/2929741304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 msgr2=0x7f4958106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 --2- 172.21.15.116:0/2929741304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958106d80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4944009900 tx=0x7f494402bc40 comp rx=0 tx=0).stop 2024-03-04T07:44:24.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 -- 172.21.15.116:0/2929741304 shutdown_connections 2024-03-04T07:44:24.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 --2- 172.21.15.116:0/2929741304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49581072c0 0x7f4958109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 --2- 172.21.15.116:0/2929741304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958106d80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 --2- 172.21.15.116:0/2929741304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49580ff240 0x7f49580ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 -- 172.21.15.116:0/2929741304 >> 172.21.15.116:0/2929741304 conn(0x7f49580fad90 msgr2=0x7f49580fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:24.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 -- 172.21.15.116:0/2929741304 shutdown_connections 2024-03-04T07:44:24.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.871+0000 7f495ca00640 1 -- 172.21.15.116:0/2929741304 wait complete. 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 Processor -- start 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 -- start start 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49580ff240 0x7f4958102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49581072c0 0x7f4958101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495810d2a0 con 0x7f49580ff240 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495810d120 con 0x7f49580ffc30 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f495ca00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495810d420 con 0x7f49581072c0 2024-03-04T07:44:24.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f4955400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f4955e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49580ff240 0x7f4958102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f4955400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49986/0 (socket says 172.21.15.116:49986) 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.872+0000 7f4955400640 1 -- 172.21.15.116:0/1434095635 learned_addr learned my addr 172.21.15.116:0/1434095635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4956800640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49581072c0 0x7f4958101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4955400640 1 -- 172.21.15.116:0/1434095635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49581072c0 msgr2=0x7f4958101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4955400640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49581072c0 0x7f4958101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4955400640 1 -- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49580ff240 msgr2=0x7f4958102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4955400640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49580ff240 0x7f4958102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4955400640 1 -- 172.21.15.116:0/1434095635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4958101a50 con 0x7f49580ffc30 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4956800640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49581072c0 0x7f4958101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f4955400640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958100d10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f494402c8f0 tx=0x7f4944005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.873+0000 7f494ea00640 1 -- 172.21.15.116:0/1434095635 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944041070 con 0x7f49580ffc30 2024-03-04T07:44:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.874+0000 7f494ea00640 1 -- 172.21.15.116:0/1434095635 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49440044d0 con 0x7f49580ffc30 2024-03-04T07:44:24.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.874+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49581a1a30 con 0x7f49580ffc30 2024-03-04T07:44:24.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.874+0000 7f494ea00640 1 -- 172.21.15.116:0/1434095635 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944034440 con 0x7f49580ffc30 2024-03-04T07:44:24.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.874+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49581a1f40 con 0x7f49580ffc30 2024-03-04T07:44:24.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.877+0000 7f494ea00640 1 -- 172.21.15.116:0/1434095635 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4944044070 con 0x7f49580ffc30 2024-03-04T07:44:24.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.877+0000 7f495f7ae740 1 Processor -- start 2024-03-04T07:44:24.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.877+0000 7f495f7ae740 1 -- start start 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c940d0 0x562c35c944f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c995c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c99c30 0x562c35c9c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f495f7ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c35ba7e20 con 0x562c35c940d0 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f495f7ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c35bf9a20 con 0x562c35c94ac0 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f495f7ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c35ba2b00 con 0x562c35c99c30 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4956800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c940d0 0x562c35c944f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4955e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4956800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c940d0 0x562c35c944f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44916/0 (socket says 172.21.15.116:44916) 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4956800640 1 -- 172.21.15.116:0/1949709229 learned_addr learned my addr 172.21.15.116:0/1949709229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4955400640 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c99c30 0x562c35c9c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4955e00640 1 -- 172.21.15.116:0/1949709229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c99c30 msgr2=0x562c35c9c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4955e00640 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c99c30 0x562c35c9c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4955e00640 1 -- 172.21.15.116:0/1949709229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c940d0 msgr2=0x562c35c944f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4955e00640 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c940d0 0x562c35c944f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.878+0000 7f4955e00640 1 -- 172.21.15.116:0/1949709229 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c35bb7e10 con 0x562c35c94ac0 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.879+0000 7f494ea00640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f493c0a1e00 0x7f493c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.879+0000 7f4955e00640 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c995c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f494800bbf0 tx=0x7f49480107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6523963c722c12fa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.879+0000 7f494ea00640 1 -- 172.21.15.116:0/1434095635 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f49440ef980 con 0x7f49580ffc30 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.879+0000 7f4956800640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f493c0a1e00 0x7f493c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.879+0000 7f4943000640 1 -- 172.21.15.116:0/1949709229 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948004030 con 0x562c35c94ac0 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.879+0000 7f4943000640 1 -- 172.21.15.116:0/1949709229 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49480041d0 con 0x562c35c94ac0 2024-03-04T07:44:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.879+0000 7f4943000640 1 -- 172.21.15.116:0/1949709229 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49480116c0 con 0x562c35c94ac0 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 -- 172.21.15.116:0/1949709229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 msgr2=0x562c35c995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f4956800640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f493c0a1e00 0x7f493c0a42c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4950009fd0 tx=0x7f4950009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c995c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f494800bbf0 tx=0x7f49480107a0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 -- 172.21.15.116:0/1949709229 shutdown_connections 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c99c30 0x562c35c9c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c995c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 --2- 172.21.15.116:0/1949709229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c940d0 0x562c35c944f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 -- 172.21.15.116:0/1949709229 >> 172.21.15.116:0/1949709229 conn(0x562c35c8f640 msgr2=0x562c35c8fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 -- 172.21.15.116:0/1949709229 shutdown_connections 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 -- 172.21.15.116:0/1949709229 wait complete. 2024-03-04T07:44:24.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.880+0000 7f495f7ae740 1 -- start start 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 1 Processor -- start 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35c9e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35c9e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c99c30 0x562c35ca5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c35bb5f40 con 0x562c35c94ac0 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c35bf5ac0 con 0x562c35c99c30 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c35c9cbe0 con 0x562c35c940d0 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f495f7ae740 10 client.0 _ll_register_callbacks cb 0x562c35bf4aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f4955e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35c9e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f4955e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35c9e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53930/0 (socket says 172.21.15.116:53930) 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f4955e00640 1 -- 172.21.15.116:0/2821916813 learned_addr learned my addr 172.21.15.116:0/2821916813 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f4956800640 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c99c30 0x562c35ca5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.881+0000 7f4955400640 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35c9e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4955e00640 1 -- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c99c30 msgr2=0x562c35ca5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4955e00640 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c99c30 0x562c35ca5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4955e00640 1 -- 172.21.15.116:0/2821916813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 msgr2=0x562c35c9e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4955e00640 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35c9e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4955e00640 1 -- 172.21.15.116:0/2821916813 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c359d67e0 con 0x562c35c940d0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4955e00640 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35c9e490 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4948011010 tx=0x7f494800bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=115e3a9ebd3ca26e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4937400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4937400640 1 -- 172.21.15.116:0/2821916813 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948005240 con 0x562c35c940d0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4937400640 1 -- 172.21.15.116:0/2821916813 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f49480053e0 con 0x562c35c940d0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49480041d0 con 0x562c35c940d0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.882+0000 7f4937400640 1 -- 172.21.15.116:0/2821916813 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49480055e0 con 0x562c35c940d0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 10 client.26149 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562c35a36f70 con 0x562c35c940d0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f4937400640 1 -- 172.21.15.116:0/2821916813 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49480041d0 con 0x562c35c940d0 2024-03-04T07:44:24.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f4937400640 1 -- 172.21.15.116:0/2821916813 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4948004c50 con 0x562c35c940d0 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 10 client.26149 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 10 client.26149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 20 client.26149 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35ca9fb0 0x562c35cac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 4 client.26149 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c35c06f60 con 0x562c35ca9fb0 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.883+0000 7f4955400640 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35ca9fb0 0x562c35cac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.884+0000 7f4955400640 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35ca9fb0 0x562c35cac3a0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da803c16c7d49444 server_cookie=c4d7df3b5da51274 in_seq=0 out_seq=0 2024-03-04T07:44:24.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.884+0000 7f4937400640 10 client.26149 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.884+0000 7f4937400640 1 -- 172.21.15.116:0/2821916813 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562c35c06f60 con 0x562c35ca9fb0 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.884+0000 7f4937400640 10 client.26149 handle_command_reply: tid=0 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.884+0000 7f495f7ae740 1 client.26149 shutdown 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 msgr2=0x562c35c9e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35c9e490 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4948011010 tx=0x7f494800bfd0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 shutdown_connections 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35ca9fb0 0x562c35cac3a0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c99c30 0x562c35ca5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35c9e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2821916813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35c9e490 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f4937400640 0 client.26149 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 >> 172.21.15.116:0/2821916813 conn(0x562c35c8f640 msgr2=0x562c35c90890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 shutdown_connections 2024-03-04T07:44:24.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 1 -- 172.21.15.116:0/2821916813 wait complete. 2024-03-04T07:44:24.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.885+0000 7f495f7ae740 20 client.26149 trim_cache size 0 max 16384 2024-03-04T07:44:24.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 Processor -- start 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 -- start start 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c8f640 0x562c35c8ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35ca59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c92fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581a8c50 con 0x562c35c8f640 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581ab230 con 0x562c35c94ac0 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f495f7ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581a9250 con 0x562c35c940d0 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f4955e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c8f640 0x562c35c8ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.903+0000 7f4955e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c8f640 0x562c35c8ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44936/0 (socket says 172.21.15.116:44936) 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4956800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c92fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955e00640 1 -- 172.21.15.116:0/2739665338 learned_addr learned my addr 172.21.15.116:0/2739665338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:24.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955400640 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35ca59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955e00640 1 -- 172.21.15.116:0/2739665338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 msgr2=0x562c35ca59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955e00640 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35ca59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955e00640 1 -- 172.21.15.116:0/2739665338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 msgr2=0x562c35c92fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955e00640 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c92fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955e00640 1 -- 172.21.15.116:0/2739665338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c358ce170 con 0x562c35c8f640 2024-03-04T07:44:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955400640 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35ca59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:24.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4955e00640 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c8f640 0x562c35c8ea60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f49480107e0 tx=0x7f4948005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65142a8609988ca9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:24.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.904+0000 7f4942600640 1 -- 172.21.15.116:0/2739665338 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948011ab0 con 0x562c35c8f640 2024-03-04T07:44:24.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f4942600640 1 -- 172.21.15.116:0/2739665338 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4948029070 con 0x562c35c8f640 2024-03-04T07:44:24.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f4942600640 1 -- 172.21.15.116:0/2739665338 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494801be60 con 0x562c35c8f640 2024-03-04T07:44:24.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 -- 172.21.15.116:0/2739665338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c8f640 msgr2=0x562c35c8ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c8f640 0x562c35c8ea60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f49480107e0 tx=0x7f4948005c50 comp rx=0 tx=0).stop 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 -- 172.21.15.116:0/2739665338 shutdown_connections 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c94ac0 0x562c35c92fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c940d0 0x562c35ca59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 --2- 172.21.15.116:0/2739665338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c8f640 0x562c35c8ea60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 -- 172.21.15.116:0/2739665338 >> 172.21.15.116:0/2739665338 conn(0x562c35c958a0 msgr2=0x562c35c928a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 -- 172.21.15.116:0/2739665338 shutdown_connections 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.905+0000 7f495f7ae740 1 -- 172.21.15.116:0/2739665338 wait complete. 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 -- start start 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 Processor -- start 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c8f640 0x562c35ca3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c940d0 0x562c35ca3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35ca4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581a98b0 con 0x562c35c94ac0 2024-03-04T07:44:24.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581aabd0 con 0x562c35c940d0 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581a9f10 con 0x562c35c8f640 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.906+0000 7f495f7ae740 10 client.0 _ll_register_callbacks cb 0x7f4950009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c940d0 0x562c35ca3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c940d0 0x562c35ca3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50034/0 (socket says 172.21.15.116:50034) 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 -- 172.21.15.116:0/293813029 learned_addr learned my addr 172.21.15.116:0/293813029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c8f640 0x562c35ca3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4956800640 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35ca4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 -- 172.21.15.116:0/293813029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c8f640 msgr2=0x562c35ca3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c8f640 0x562c35ca3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 -- 172.21.15.116:0/293813029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 msgr2=0x562c35ca4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35ca4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 -- 172.21.15.116:0/293813029 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c35ca51e0 con 0x562c35c940d0 2024-03-04T07:44:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4955400640 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c940d0 0x562c35ca3cd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f49440b8ad0 tx=0x7f49440a7040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=459a27d500f2d090 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4937400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4937400640 1 -- 172.21.15.116:0/293813029 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49440d5060 con 0x562c35c940d0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4948029070 con 0x562c35c940d0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.907+0000 7f4937400640 1 -- 172.21.15.116:0/293813029 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f49440ed550 con 0x562c35c940d0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f4937400640 1 -- 172.21.15.116:0/293813029 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f494404b040 con 0x562c35c940d0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 10 client.26213 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562c35d05db0 con 0x562c35c940d0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f4937400640 1 -- 172.21.15.116:0/293813029 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4948029070 con 0x562c35c940d0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f4937400640 1 -- 172.21.15.116:0/293813029 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f49440b8ce0 con 0x562c35c940d0 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 10 client.26213 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 20 client.26213 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35c99c30 0x562c35d08630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 4 client.26213 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:44:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562c35c012b0 con 0x562c35c99c30 2024-03-04T07:44:24.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.908+0000 7f4955e00640 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35c99c30 0x562c35d08630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.909+0000 7f4955e00640 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35c99c30 0x562c35d08630 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a74e4aa2ecd5b4ec server_cookie=d0779b1b76f54ec6 in_seq=0 out_seq=0 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.909+0000 7f4937400640 10 client.26213 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f4937400640 1 -- 172.21.15.116:0/293813029 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562c35c012b0 con 0x562c35c99c30 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f4937400640 10 client.26213 handle_command_reply: tid=0 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 client.26213 shutdown 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c940d0 msgr2=0x562c35ca3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c940d0 0x562c35ca3cd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f49440b8ad0 tx=0x7f49440a7040 comp rx=0 tx=0).stop 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 shutdown_connections 2024-03-04T07:44:24.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562c35c99c30 0x562c35d08630 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562c35c94ac0 0x562c35ca4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562c35c940d0 0x562c35ca3cd0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 --2- 172.21.15.116:0/293813029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c35c8f640 0x562c35ca3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f4937400640 0 client.26213 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.910+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 >> 172.21.15.116:0/293813029 conn(0x562c35c958a0 msgr2=0x562c35c93130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.911+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 shutdown_connections 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.911+0000 7f495f7ae740 1 -- 172.21.15.116:0/293813029 wait complete. 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.911+0000 7f495f7ae740 20 client.26213 trim_cache size 0 max 16384 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:24.915 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:44:24.916 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "fc3ed6cb-74d9-449c-bdfe-86f7d16513a5", 2024-03-04T07:44:24.916 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:44:24.916 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.911+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f493c0a1e00 msgr2=0x7f493c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.911+0000 7f495ca00640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f493c0a1e00 0x7f493c0a42c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4950009fd0 tx=0x7f4950009500 comp rx=0 tx=0).stop 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.911+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 msgr2=0x7f4958100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958100d10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f494402c8f0 tx=0x7f4944005d50 comp rx=0 tx=0).stop 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 shutdown_connections 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f493c0a1e00 0x7f493c0a42c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49581072c0 0x7f4958101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49580ffc30 0x7f4958100d10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 --2- 172.21.15.116:0/1434095635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49580ff240 0x7f4958102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 >> 172.21.15.116:0/1434095635 conn(0x7f49580fad90 msgr2=0x7f4958102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 shutdown_connections 2024-03-04T07:44:24.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:24.912+0000 7f495ca00640 1 -- 172.21.15.116:0/1434095635 wait complete. 2024-03-04T07:44:24.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:44:24.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:44:24.998 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:24 smithi116 ceph-mon[32150]: pgmap v510: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:25.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:24 smithi133 ceph-mon[36505]: pgmap v510: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 -- 172.21.15.116:0/2162395277 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 msgr2=0x7fde70102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 --2- 172.21.15.116:0/2162395277 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 0x7fde70102c70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fde58009ca0 tx=0x7fde5802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 -- 172.21.15.116:0/2162395277 shutdown_connections 2024-03-04T07:44:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 --2- 172.21.15.116:0/2162395277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde701031b0 0x7fde70109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 --2- 172.21.15.116:0/2162395277 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 0x7fde70102c70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 --2- 172.21.15.116:0/2162395277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 0x7fde70101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 -- 172.21.15.116:0/2162395277 >> 172.21.15.116:0/2162395277 conn(0x7fde700fcb70 msgr2=0x7fde700fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 -- 172.21.15.116:0/2162395277 shutdown_connections 2024-03-04T07:44:25.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.134+0000 7fde76c00640 1 -- 172.21.15.116:0/2162395277 wait complete. 2024-03-04T07:44:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.135+0000 7fde76c00640 1 Processor -- start 2024-03-04T07:44:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.135+0000 7fde76c00640 1 -- start start 2024-03-04T07:44:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.135+0000 7fde76c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 0x7fde70199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.135+0000 7fde76c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 0x7fde7019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde76c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde701031b0 0x7fde701a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde76c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7010d280 con 0x7fde70101600 2024-03-04T07:44:25.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde76c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7010d100 con 0x7fde701027f0 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde76c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7010d400 con 0x7fde701031b0 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 0x7fde70199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 0x7fde7019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 0x7fde70199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44964/0 (socket says 172.21.15.116:44964) 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6fe00640 1 -- 172.21.15.116:0/1554321089 learned_addr learned my addr 172.21.15.116:0/1554321089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde74c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde701031b0 0x7fde701a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6f400640 1 -- 172.21.15.116:0/1554321089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde701031b0 msgr2=0x7fde701a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6f400640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde701031b0 0x7fde701a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6f400640 1 -- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 msgr2=0x7fde70199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6f400640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 0x7fde70199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6f400640 1 -- 172.21.15.116:0/1554321089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde701a1720 con 0x7fde701027f0 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6fe00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 0x7fde70199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:25.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde74c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde701031b0 0x7fde701a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:25.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.136+0000 7fde6f400640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 0x7fde7019a040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fde58009bf0 tx=0x7fde58032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:25.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.137+0000 7fde6cc00640 1 -- 172.21.15.116:0/1554321089 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58041020 con 0x7fde701027f0 2024-03-04T07:44:25.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.137+0000 7fde6cc00640 1 -- 172.21.15.116:0/1554321089 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde58034cd0 con 0x7fde701027f0 2024-03-04T07:44:25.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.137+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde701a19b0 con 0x7fde701027f0 2024-03-04T07:44:25.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.137+0000 7fde6cc00640 1 -- 172.21.15.116:0/1554321089 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58039040 con 0x7fde701027f0 2024-03-04T07:44:25.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.137+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde701a1ec0 con 0x7fde701027f0 2024-03-04T07:44:25.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 Processor -- start 2024-03-04T07:44:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 -- start start 2024-03-04T07:44:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6d7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d8540 0x55f95f6dd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6dfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde6cc00640 1 -- 172.21.15.116:0/1554321089 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fde58044020 con 0x7fde701027f0 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95f605090 con 0x55f95f6dd6b0 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95f63cd30 con 0x55f95f6d8540 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.141+0000 7fde799ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95f5e5a30 con 0x55f95f6d7b50 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6dfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde74c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6d7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d8540 0x55f95f6dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6dfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44968/0 (socket says 172.21.15.116:44968) 2024-03-04T07:44:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6f400640 1 -- 172.21.15.116:0/198832251 learned_addr learned my addr 172.21.15.116:0/198832251 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6fe00640 1 -- 172.21.15.116:0/198832251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 msgr2=0x55f95f6d7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6fe00640 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6d7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6fe00640 1 -- 172.21.15.116:0/198832251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 msgr2=0x55f95f6dfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6fe00640 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6dfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6fe00640 1 -- 172.21.15.116:0/198832251 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f95f41b7d0 con 0x55f95f6d8540 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.142+0000 7fde6f400640 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6dfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde6fe00640 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d8540 0x55f95f6dd040 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fde600114c0 tx=0x7fde60011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d08bc1f9ccac66ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde65200640 1 -- 172.21.15.116:0/198832251 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60005020 con 0x55f95f6d8540 2024-03-04T07:44:25.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde65200640 1 -- 172.21.15.116:0/198832251 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde600047a0 con 0x55f95f6d8540 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde65200640 1 -- 172.21.15.116:0/198832251 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6000f980 con 0x55f95f6d8540 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde6cc00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde500a1e00 0x7fde500a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde6cc00640 1 -- 172.21.15.116:0/1554321089 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fde580efa40 con 0x7fde701027f0 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde74c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde500a1e00 0x7fde500a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde799ab740 1 -- 172.21.15.116:0/198832251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d8540 msgr2=0x55f95f6dd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.143+0000 7fde799ab740 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d8540 0x55f95f6dd040 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fde600114c0 tx=0x7fde60011980 comp rx=0 tx=0).stop 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 -- 172.21.15.116:0/198832251 shutdown_connections 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6dfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d8540 0x55f95f6dd040 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 --2- 172.21.15.116:0/198832251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6d7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 -- 172.21.15.116:0/198832251 >> 172.21.15.116:0/198832251 conn(0x55f95f6d30c0 msgr2=0x55f95f6d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 -- 172.21.15.116:0/198832251 shutdown_connections 2024-03-04T07:44:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 -- 172.21.15.116:0/198832251 wait complete. 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde74c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde500a1e00 0x7fde500a42c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55f95f6d8da0 tx=0x7fde68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.144+0000 7fde799ab740 1 -- start start 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 1 Processor -- start 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95f5db0c0 con 0x55f95f6d8540 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95f638dd0 con 0x55f95f6d7b50 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95f6e0660 con 0x55f95f6dd6b0 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde799ab740 10 client.0 _ll_register_callbacks cb 0x55f95f5f8860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6e1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50066/0 (socket says 172.21.15.116:50066) 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde6fe00640 1 -- 172.21.15.116:0/1076198995 learned_addr learned my addr 172.21.15.116:0/1076198995 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde6f400640 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.145+0000 7fde74c00640 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde6fe00640 1 -- 172.21.15.116:0/1076198995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6dd6b0 msgr2=0x55f95f6e9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde6fe00640 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde6fe00640 1 -- 172.21.15.116:0/1076198995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 msgr2=0x55f95f6e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde6fe00640 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde6fe00640 1 -- 172.21.15.116:0/1076198995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f95f3513f0 con 0x55f95f6d7b50 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde6fe00640 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6e1ec0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fde60009350 tx=0x7fde600051e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=152a82aa5f275c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde5d600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde5d600640 1 -- 172.21.15.116:0/1076198995 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60028020 con 0x55f95f6d7b50 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde600047a0 con 0x55f95f6d7b50 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde5d600640 1 -- 172.21.15.116:0/1076198995 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fde600053d0 con 0x55f95f6d7b50 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde5d600640 1 -- 172.21.15.116:0/1076198995 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde6001d040 con 0x55f95f6d7b50 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde799ab740 10 client.26225 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.146+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f95f3110f0 con 0x55f95f6d7b50 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde5d600640 1 -- 172.21.15.116:0/1076198995 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde600047a0 con 0x55f95f6d7b50 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde5d600640 1 -- 172.21.15.116:0/1076198995 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fde600236d0 con 0x55f95f6d7b50 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde799ab740 10 client.26225 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde799ab740 10 client.26225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde799ab740 20 client.26225 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde799ab740 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f6ed6f0 0x55f95f6efb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde799ab740 4 client.26225 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f95f64a270 con 0x55f95f6ed6f0 2024-03-04T07:44:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.147+0000 7fde6f400640 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f6ed6f0 0x55f95f6efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.148+0000 7fde5d600640 10 client.26225 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:25.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.148+0000 7fde6f400640 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f6ed6f0 0x55f95f6efb20 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f019e64c28593e63 server_cookie=c35cad61e5865b90 in_seq=0 out_seq=0 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.148+0000 7fde5d600640 1 -- 172.21.15.116:0/1076198995 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f95f64a270 con 0x55f95f6ed6f0 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.148+0000 7fde5d600640 10 client.26225 handle_command_reply: tid=0 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.148+0000 7fde799ab740 1 client.26225 shutdown 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.148+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 msgr2=0x55f95f6e1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.148+0000 7fde799ab740 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6e1ec0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fde60009350 tx=0x7fde600051e0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 shutdown_connections 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f6ed6f0 0x55f95f6efb20 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 --2- 172.21.15.116:0/1076198995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6e1ec0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 >> 172.21.15.116:0/1076198995 conn(0x55f95f6d30c0 msgr2=0x55f95f6d2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde5d600640 0 client.26225 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 shutdown_connections 2024-03-04T07:44:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 1 -- 172.21.15.116:0/1076198995 wait complete. 2024-03-04T07:44:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.149+0000 7fde799ab740 20 client.26225 trim_cache size 0 max 16384 2024-03-04T07:44:25.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.166+0000 7fde799ab740 1 Processor -- start 2024-03-04T07:44:25.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.166+0000 7fde799ab740 1 -- start start 2024-03-04T07:44:25.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde799ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde799ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d8540 0x55f95f6e9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde799ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde799ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701a8c10 con 0x55f95f6dd6b0 2024-03-04T07:44:25.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde799ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701ab1f0 con 0x55f95f6d7b50 2024-03-04T07:44:25.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde799ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701a9210 con 0x55f95f6d8540 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50068/0 (socket says 172.21.15.116:50068) 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 -- 172.21.15.116:0/1780938302 learned_addr learned my addr 172.21.15.116:0/1780938302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6f400640 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d8540 0x55f95f6e9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde74c00640 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 -- 172.21.15.116:0/1780938302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d8540 msgr2=0x55f95f6e9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d8540 0x55f95f6e9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 -- 172.21.15.116:0/1780938302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 msgr2=0x55f95f6f48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6f48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 -- 172.21.15.116:0/1780938302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f95f6d3e30 con 0x55f95f6d7b50 2024-03-04T07:44:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.167+0000 7fde6fe00640 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6f43b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fde60031040 tx=0x7fde60011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221b1828b9325b7d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde5fe00640 1 -- 172.21.15.116:0/1780938302 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60028070 con 0x55f95f6d7b50 2024-03-04T07:44:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde5fe00640 1 -- 172.21.15.116:0/1780938302 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde60023ce0 con 0x55f95f6d7b50 2024-03-04T07:44:25.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde5fe00640 1 -- 172.21.15.116:0/1780938302 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6000f8b0 con 0x55f95f6d7b50 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 -- 172.21.15.116:0/1780938302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 msgr2=0x55f95f6f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6f43b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fde60031040 tx=0x7fde60011980 comp rx=0 tx=0).stop 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 -- 172.21.15.116:0/1780938302 shutdown_connections 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6dd6b0 0x55f95f6f48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d8540 0x55f95f6e9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 --2- 172.21.15.116:0/1780938302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6d7b50 0x55f95f6f43b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 -- 172.21.15.116:0/1780938302 >> 172.21.15.116:0/1780938302 conn(0x55f95f6d30c0 msgr2=0x55f95f6d6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 -- 172.21.15.116:0/1780938302 shutdown_connections 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.168+0000 7fde799ab740 1 -- 172.21.15.116:0/1780938302 wait complete. 2024-03-04T07:44:25.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 -- start start 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 Processor -- start 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6e70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701a9870 con 0x55f95f6d8540 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701aab90 con 0x55f95f6dd6b0 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701a9ed0 con 0x55f95f6d7b50 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.169+0000 7fde799ab740 10 client.0 _ll_register_callbacks cb 0x7fde60009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde74c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde74c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50070/0 (socket says 172.21.15.116:50070) 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde74c00640 1 -- 172.21.15.116:0/4205961528 learned_addr learned my addr 172.21.15.116:0/4205961528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6fe00640 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6e70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6f400640 1 -- 172.21.15.116:0/4205961528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 msgr2=0x55f95f6e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6f400640 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6e70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6f400640 1 -- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6dd6b0 msgr2=0x55f95f6e8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6f400640 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6f400640 1 -- 172.21.15.116:0/4205961528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f95f6e8c00 con 0x55f95f6d8540 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde74c00640 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde6f400640 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e7600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fde5802c130 tx=0x7fde58000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d8dca13399e877 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.170+0000 7fde5d600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde5d600640 1 -- 172.21.15.116:0/4205961528 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde580bc340 con 0x55f95f6d8540 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde5d600640 1 -- 172.21.15.116:0/4205961528 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fde580b8e70 con 0x55f95f6d8540 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde60023ce0 con 0x55f95f6d8540 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde5d600640 1 -- 172.21.15.116:0/4205961528 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde580bc160 con 0x55f95f6d8540 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 10 client.16110 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f95f751e30 con 0x55f95f6d8540 2024-03-04T07:44:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde5d600640 1 -- 172.21.15.116:0/4205961528 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde60023ce0 con 0x55f95f6d8540 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde5d600640 1 -- 172.21.15.116:0/4205961528 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fde5803c030 con 0x55f95f6d8540 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 10 client.16110 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 20 client.16110 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f7527e0 0x55f95f754c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 4 client.16110 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.171+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f95f6445c0 con 0x55f95f7527e0 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.172+0000 7fde6fe00640 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f7527e0 0x55f95f754c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.172+0000 7fde6fe00640 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f7527e0 0x55f95f754c00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da2ef83e9a9f6f81 server_cookie=95f3323b4b035d0e in_seq=0 out_seq=0 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.172+0000 7fde5d600640 10 client.16110 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.172+0000 7fde5d600640 1 -- 172.21.15.116:0/4205961528 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f95f6445c0 con 0x55f95f7527e0 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.172+0000 7fde5d600640 10 client.16110 handle_command_reply: tid=0 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.172+0000 7fde799ab740 1 client.16110 shutdown 2024-03-04T07:44:25.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 msgr2=0x55f95f6e7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e7600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fde5802c130 tx=0x7fde58000f40 comp rx=0 tx=0).stop 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 shutdown_connections 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f95f7527e0 0x55f95f754c00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f95f6dd6b0 0x55f95f6e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f95f6d8540 0x55f95f6e7600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 --2- 172.21.15.116:0/4205961528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f95f6d7b50 0x55f95f6e70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 >> 172.21.15.116:0/4205961528 conn(0x55f95f6d30c0 msgr2=0x55f95f6d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde5d600640 0 client.16110 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 shutdown_connections 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 1 -- 172.21.15.116:0/4205961528 wait complete. 2024-03-04T07:44:25.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.173+0000 7fde799ab740 20 client.16110 trim_cache size 0 max 16384 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "fc3ed6cb-74d9-449c-bdfe-86f7d16513a5": { 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "fc3ed6cb-74d9-449c-bdfe-86f7d16513a5", 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:44:25.179 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde500a1e00 msgr2=0x7fde500a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde500a1e00 0x7fde500a42c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55f95f6d8da0 tx=0x7fde68005f90 comp rx=0 tx=0).stop 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 msgr2=0x7fde7019a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 0x7fde7019a040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fde58009bf0 tx=0x7fde58032ec0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 shutdown_connections 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fde500a1e00 0x7fde500a42c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde701031b0 0x7fde701a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fde701027f0 0x7fde7019a040 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 --2- 172.21.15.116:0/1554321089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde70101600 0x7fde70199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:25.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 >> 172.21.15.116:0/1554321089 conn(0x7fde700fcb70 msgr2=0x7fde70073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:25.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 shutdown_connections 2024-03-04T07:44:25.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:25.174+0000 7fde76c00640 1 -- 172.21.15.116:0/1554321089 wait complete. 2024-03-04T07:44:25.193 INFO:tasks.cephfs.filesystem:scrub status for tag:fc3ed6cb-74d9-449c-bdfe-86f7d16513a5 - {'path': '/', 'tag': 'fc3ed6cb-74d9-449c-bdfe-86f7d16513a5', 'options': 'recursive,force'} 2024-03-04T07:44:25.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:24 smithi152 ceph-mon[37154]: pgmap v510: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:25 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:44:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:25 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:25 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:44:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:25 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:25 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:44:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:25 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:25 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:44:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:25 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:25 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:44:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:25 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:25 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:44:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:25 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:26 smithi116 ceph-mon[32150]: pgmap v511: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:44:27.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:26 smithi152 ceph-mon[37154]: pgmap v511: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:44:27.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:26 smithi133 ceph-mon[36505]: pgmap v511: 97 pgs: 97 active+clean; 2.5 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:44:28.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:27 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:44:28.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:27 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:44:28.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:27 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:44:29.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:28 smithi116 ceph-mon[32150]: pgmap v512: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-03-04T07:44:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:28 smithi152 ceph-mon[37154]: pgmap v512: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-03-04T07:44:29.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:28 smithi133 ceph-mon[36505]: pgmap v512: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-03-04T07:44:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 -- 172.21.15.116:0/1952749888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 msgr2=0x7fe218108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 --2- 172.21.15.116:0/1952749888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe218108980 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe210009be0 tx=0x7fe21002bef0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 -- 172.21.15.116:0/1952749888 shutdown_connections 2024-03-04T07:44:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 --2- 172.21.15.116:0/1952749888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe218108980 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 --2- 172.21.15.116:0/1952749888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218101730 0x7fe218101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 --2- 172.21.15.116:0/1952749888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe218100540 0x7fe218100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 -- 172.21.15.116:0/1952749888 >> 172.21.15.116:0/1952749888 conn(0x7fe2180fbab0 msgr2=0x7fe2180fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 -- 172.21.15.116:0/1952749888 shutdown_connections 2024-03-04T07:44:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.341+0000 7fe21f200640 1 -- 172.21.15.116:0/1952749888 wait complete. 2024-03-04T07:44:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.342+0000 7fe21f200640 1 Processor -- start 2024-03-04T07:44:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.342+0000 7fe21f200640 1 -- start start 2024-03-04T07:44:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21f200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218100540 0x7fe218199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe218101730 0x7fe218199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21f200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe2181a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21810c1d0 con 0x7fe218101730 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21f200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21810c050 con 0x7fe2181020f0 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21f200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21810c350 con 0x7fe218100540 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe2181a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe217400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe218101730 0x7fe218199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe2181a0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59360/0 (socket says 172.21.15.116:59360) 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe217400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe218101730 0x7fe218199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33708/0 (socket says 172.21.15.116:33708) 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe21d200640 1 -- 172.21.15.116:0/1621185060 learned_addr learned my addr 172.21.15.116:0/1621185060 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.343+0000 7fe217e00640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218100540 0x7fe218199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe21d200640 1 -- 172.21.15.116:0/1621185060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218100540 msgr2=0x7fe218199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe21d200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218100540 0x7fe218199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe21d200640 1 -- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe218101730 msgr2=0x7fe218199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe21d200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe218101730 0x7fe218199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe21d200640 1 -- 172.21.15.116:0/1621185060 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2181a1650 con 0x7fe2181020f0 2024-03-04T07:44:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe21d200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe2181a0fe0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe210036040 tx=0x7fe2100337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe214c00640 1 -- 172.21.15.116:0/1621185060 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210043020 con 0x7fe2181020f0 2024-03-04T07:44:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe214c00640 1 -- 172.21.15.116:0/1621185060 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe21002ca60 con 0x7fe2181020f0 2024-03-04T07:44:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe214c00640 1 -- 172.21.15.116:0/1621185060 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210034700 con 0x7fe2181020f0 2024-03-04T07:44:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.344+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2181a18e0 con 0x7fe2181020f0 2024-03-04T07:44:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.345+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2181a1de0 con 0x7fe2181020f0 2024-03-04T07:44:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe214c00640 1 -- 172.21.15.116:0/1621185060 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe210046020 con 0x7fe2181020f0 2024-03-04T07:44:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 Processor -- start 2024-03-04T07:44:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 -- start start 2024-03-04T07:44:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b1850 0x56065d7b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b2240 0x56065d7b6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7b9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065d717370 con 0x56065d7b73b0 2024-03-04T07:44:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065d6c0550 con 0x56065d7b1850 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.348+0000 7fe221fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065d713410 con 0x56065d7b2240 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b1850 0x56065d7b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b2240 0x56065d7b6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b1850 0x56065d7b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59362/0 (socket says 172.21.15.116:59362) 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217400640 1 -- 172.21.15.116:0/1077779864 learned_addr learned my addr 172.21.15.116:0/1077779864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217400640 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe214c00640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1fc0a1f90 0x7fe1fc0a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217e00640 1 -- 172.21.15.116:0/1077779864 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b1850 msgr2=0x56065d7b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217e00640 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b1850 0x56065d7b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217e00640 1 -- 172.21.15.116:0/1077779864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 msgr2=0x56065d7b9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217e00640 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7b9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217e00640 1 -- 172.21.15.116:0/1077779864 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56065d6d43c0 con 0x56065d7b2240 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe217400640 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe214c00640 1 -- 172.21.15.116:0/1621185060 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe2100f0290 con 0x7fe2181020f0 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.349+0000 7fe21d200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1fc0a1f90 0x7fe1fc0a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.350+0000 7fe217e00640 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b2240 0x56065d7b6d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe218101590 tx=0x7fe208011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f488fe8ac9e3618 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.350+0000 7fe20d600640 1 -- 172.21.15.116:0/1077779864 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20801e070 con 0x56065d7b2240 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.350+0000 7fe20d600640 1 -- 172.21.15.116:0/1077779864 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe208019910 con 0x56065d7b2240 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.350+0000 7fe20d600640 1 -- 172.21.15.116:0/1077779864 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20800ea10 con 0x56065d7b2240 2024-03-04T07:44:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.350+0000 7fe21d200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1fc0a1f90 0x7fe1fc0a4450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe210009370 tx=0x7fe210004770 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 -- 172.21.15.116:0/1077779864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b2240 msgr2=0x56065d7b6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b2240 0x56065d7b6d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe218101590 tx=0x7fe208011810 comp rx=0 tx=0).stop 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 -- 172.21.15.116:0/1077779864 shutdown_connections 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b2240 0x56065d7b6d40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 --2- 172.21.15.116:0/1077779864 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b1850 0x56065d7b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 -- 172.21.15.116:0/1077779864 >> 172.21.15.116:0/1077779864 conn(0x56065d7acdc0 msgr2=0x56065d7ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 -- 172.21.15.116:0/1077779864 shutdown_connections 2024-03-04T07:44:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 -- 172.21.15.116:0/1077779864 wait complete. 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.351+0000 7fe221fab740 1 -- start start 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 1 Processor -- start 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7bbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7bc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7c3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065d7ba360 con 0x56065d7b2240 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065d6d39b0 con 0x56065d7b73b0 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56065d7ba4e0 con 0x56065d7b1850 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe221fab740 10 client.0 _ll_register_callbacks cb 0x56065d70f340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe217e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7bc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe217e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7bc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33734/0 (socket says 172.21.15.116:33734) 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe217e00640 1 -- 172.21.15.116:0/1482268152 learned_addr learned my addr 172.21.15.116:0/1482268152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.352+0000 7fe217400640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217400640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7bbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217e00640 1 -- 172.21.15.116:0/1482268152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 msgr2=0x56065d7bbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217e00640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7bbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217e00640 1 -- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 msgr2=0x56065d7c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217e00640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217e00640 1 -- 172.21.15.116:0/1482268152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56065d42c210 con 0x56065d7b2240 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217400640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe217e00640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7bc0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe208009820 tx=0x7fe20800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8fd1c12feb21f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe202000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe202000640 1 -- 172.21.15.116:0/1482268152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208028070 con 0x56065d7b2240 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe208019910 con 0x56065d7b2240 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.353+0000 7fe202000640 1 -- 172.21.15.116:0/1482268152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe208005040 con 0x56065d7b2240 2024-03-04T07:44:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe202000640 1 -- 172.21.15.116:0/1482268152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe208034050 con 0x56065d7b2240 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 10 client.16122 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56065d4f8620 con 0x56065d7b2240 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe202000640 1 -- 172.21.15.116:0/1482268152 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe208019910 con 0x56065d7b2240 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe202000640 1 -- 172.21.15.116:0/1482268152 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe2080046c0 con 0x56065d7b2240 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 10 client.16122 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 10 client.16122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 20 client.16122 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d7c76f0 0x56065d7c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 4 client.16122 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.354+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56065d7248b0 con 0x56065d7c76f0 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.355+0000 7fe217400640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d7c76f0 0x56065d7c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.355+0000 7fe217400640 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d7c76f0 0x56065d7c9b10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1163f5a0ad967562 server_cookie=3d3e7d442e2b056c in_seq=0 out_seq=0 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.355+0000 7fe202000640 10 client.16122 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe202000640 1 -- 172.21.15.116:0/1482268152 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56065d7248b0 con 0x56065d7c76f0 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe202000640 10 client.16122 handle_command_reply: tid=0 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 client.16122 shutdown 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 msgr2=0x56065d7bc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7bc0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe208009820 tx=0x7fe20800e4b0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 shutdown_connections 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d7c76f0 0x56065d7c9b10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7bc0c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe202000640 0 client.16122 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 --2- 172.21.15.116:0/1482268152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7bbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 >> 172.21.15.116:0/1482268152 conn(0x56065d7acdc0 msgr2=0x56065d7abfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 shutdown_connections 2024-03-04T07:44:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 1 -- 172.21.15.116:0/1482268152 wait complete. 2024-03-04T07:44:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.356+0000 7fe221fab740 20 client.16122 trim_cache size 0 max 16384 2024-03-04T07:44:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.371+0000 7fe221fab740 1 Processor -- start 2024-03-04T07:44:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.371+0000 7fe221fab740 1 -- start start 2024-03-04T07:44:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe221fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe221fab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe221fab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe221fab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2181a8b80 con 0x56065d7b2240 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe221fab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2181ab160 con 0x56065d7b73b0 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe221fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2181a9180 con 0x56065d7b1850 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:44258/0 (socket says 172.21.15.116:44258) 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 -- 172.21.15.116:0/2228582408 learned_addr learned my addr 172.21.15.116:0/2228582408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217e00640 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 -- 172.21.15.116:0/2228582408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 msgr2=0x56065d7ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 -- 172.21.15.116:0/2228582408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 msgr2=0x56065d7c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.372+0000 7fe217400640 1 -- 172.21.15.116:0/2228582408 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56065d7a73c0 con 0x56065d7b1850 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.373+0000 7fe217400640 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7abe60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe204004700 tx=0x7fe204004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0e50b55d7050b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.373+0000 7fe20cc00640 1 -- 172.21.15.116:0/2228582408 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2040108e0 con 0x56065d7b1850 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.373+0000 7fe20cc00640 1 -- 172.21.15.116:0/2228582408 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe204005020 con 0x56065d7b1850 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.373+0000 7fe20cc00640 1 -- 172.21.15.116:0/2228582408 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204004b90 con 0x56065d7b1850 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 -- 172.21.15.116:0/2228582408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 msgr2=0x56065d7abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7abe60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe204004700 tx=0x7fe204004730 comp rx=0 tx=0).stop 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 -- 172.21.15.116:0/2228582408 shutdown_connections 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b73b0 0x56065d7ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b2240 0x56065d7c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 --2- 172.21.15.116:0/2228582408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7abe60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 -- 172.21.15.116:0/2228582408 >> 172.21.15.116:0/2228582408 conn(0x56065d7acdc0 msgr2=0x56065d7ab750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 -- 172.21.15.116:0/2228582408 shutdown_connections 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 -- 172.21.15.116:0/2228582408 wait complete. 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 -- start start 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.374+0000 7fe221fab740 1 Processor -- start 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe221fab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7c1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe221fab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b2240 0x56065d7c1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe221fab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7c2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe221fab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2181a97e0 con 0x56065d7b73b0 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe221fab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2181aab00 con 0x56065d7b2240 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe221fab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2181a9e40 con 0x56065d7b1850 2024-03-04T07:44:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe221fab740 10 client.0 _ll_register_callbacks cb 0x7fe20801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b2240 0x56065d7c1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b2240 0x56065d7c1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59392/0 (socket says 172.21.15.116:59392) 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 -- 172.21.15.116:0/2508705493 learned_addr learned my addr 172.21.15.116:0/2508705493 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217400640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7c1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 -- 172.21.15.116:0/2508705493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 msgr2=0x56065d7c1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217400640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7c1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 -- 172.21.15.116:0/2508705493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 msgr2=0x56065d7c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7c2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217e00640 1 -- 172.21.15.116:0/2508705493 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56065d7c2c00 con 0x56065d7b2240 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.375+0000 7fe217400640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7c2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe217e00640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b2240 0x56065d7c1670 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x56065d7bd060 tx=0x7fe20800e420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb0a4dea36edb21 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe202000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe202000640 1 -- 172.21.15.116:0/2508705493 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20803d060 con 0x56065d7b2240 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe202000640 1 -- 172.21.15.116:0/2508705493 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe208019840 con 0x56065d7b2240 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe204005020 con 0x56065d7b2240 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe202000640 1 -- 172.21.15.116:0/2508705493 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe208028050 con 0x56065d7b2240 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe221fab740 10 client.26249 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56065d823870 con 0x56065d7b2240 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.376+0000 7fe202000640 1 -- 172.21.15.116:0/2508705493 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe204005020 con 0x56065d7b2240 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe202000640 1 -- 172.21.15.116:0/2508705493 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe208042070 con 0x56065d7b2240 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe221fab740 10 client.26249 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe221fab740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe221fab740 20 client.26249 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe221fab740 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d8240e0 0x56065d826500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe221fab740 4 client.26249 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:44:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56065d7bb970 con 0x56065d8240e0 2024-03-04T07:44:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe217400640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d8240e0 0x56065d826500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe202000640 10 client.26249 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.377+0000 7fe217400640 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d8240e0 0x56065d826500 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8c6b61f18a54caf server_cookie=f11821e18a53f91f in_seq=0 out_seq=0 2024-03-04T07:44:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.391+0000 7fe202000640 1 -- 172.21.15.116:0/2508705493 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56065d7bb970 con 0x56065d8240e0 2024-03-04T07:44:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.391+0000 7fe202000640 10 client.26249 handle_command_reply: tid=0 2024-03-04T07:44:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.391+0000 7fe221fab740 1 client.26249 shutdown 2024-03-04T07:44:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.391+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b2240 msgr2=0x56065d7c1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.391+0000 7fe221fab740 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b2240 0x56065d7c1670 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x56065d7bd060 tx=0x7fe20800e420 comp rx=0 tx=0).stop 2024-03-04T07:44:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.391+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 shutdown_connections 2024-03-04T07:44:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.391+0000 7fe221fab740 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56065d8240e0 0x56065d826500 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe221fab740 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56065d7b73b0 0x56065d7c2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe221fab740 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56065d7b2240 0x56065d7c1670 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe221fab740 1 --2- 172.21.15.116:0/2508705493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56065d7b1850 0x56065d7c1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 >> 172.21.15.116:0/2508705493 conn(0x56065d7acdc0 msgr2=0x56065d7b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe202000640 0 client.26249 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 shutdown_connections 2024-03-04T07:44:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe221fab740 1 -- 172.21.15.116:0/2508705493 wait complete. 2024-03-04T07:44:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.392+0000 7fe221fab740 20 client.26249 trim_cache size 0 max 16384 2024-03-04T07:44:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:44:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:44:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:44:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1fc0a1f90 msgr2=0x7fe1fc0a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1fc0a1f90 0x7fe1fc0a4450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe210009370 tx=0x7fe210004770 comp rx=0 tx=0).stop 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 msgr2=0x7fe2181a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe2181a0fe0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe210036040 tx=0x7fe2100337b0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 shutdown_connections 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe2181020f0 0x7fe2181a0fe0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe218101730 0x7fe218199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe1fc0a1f90 0x7fe1fc0a4450 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 --2- 172.21.15.116:0/1621185060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218100540 0x7fe218199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 >> 172.21.15.116:0/1621185060 conn(0x7fe2180fbab0 msgr2=0x7fe2180fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 shutdown_connections 2024-03-04T07:44:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:29.393+0000 7fe21f200640 1 -- 172.21.15.116:0/1621185060 wait complete. 2024-03-04T07:44:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:30 smithi116 ceph-mon[32150]: pgmap v513: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:44:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:30 smithi152 ceph-mon[37154]: pgmap v513: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:44:31.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:30 smithi133 ceph-mon[36505]: pgmap v513: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:44:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:32 smithi116 ceph-mon[32150]: pgmap v514: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:44:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:32 smithi152 ceph-mon[37154]: pgmap v514: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:44:33.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:32 smithi133 ceph-mon[36505]: pgmap v514: 97 pgs: 97 active+clean; 2.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:44:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:34 smithi116 ceph-mon[32150]: pgmap v515: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:44:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:34 smithi152 ceph-mon[37154]: pgmap v515: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:44:35.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:34 smithi133 ceph-mon[36505]: pgmap v515: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:44:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:36 smithi116 ceph-mon[32150]: pgmap v516: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:44:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:36 smithi152 ceph-mon[37154]: pgmap v516: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:44:37.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:36 smithi133 ceph-mon[36505]: pgmap v516: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:44:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:37 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:44:38.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:37 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:44:38.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:37 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:44:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:38 smithi116 ceph-mon[32150]: pgmap v517: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:44:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:38 smithi152 ceph-mon[37154]: pgmap v517: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:44:39.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:38 smithi133 ceph-mon[36505]: pgmap v517: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:44:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:39 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:44:40.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:39 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:44:40.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:39 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:44:40.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:39 smithi116 ceph-mon[32150]: pgmap v518: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:40.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:39 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:44:40.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:39 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:44:40.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:39 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:44:40.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:39 smithi152 ceph-mon[37154]: pgmap v518: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:40.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:39 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:44:40.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:39 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:44:40.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:39 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:44:40.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:39 smithi133 ceph-mon[36505]: pgmap v518: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.536+0000 7f7ad6800640 1 -- 172.21.15.116:0/3804591948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad0108e20 msgr2=0x7f7ad01092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.536+0000 7f7ad6800640 1 --2- 172.21.15.116:0/3804591948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad0108e20 0x7f7ad01092a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009900 tx=0x7f7ab802bc30 comp rx=0 tx=0).stop 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.537+0000 7f7ad6800640 1 -- 172.21.15.116:0/3804591948 shutdown_connections 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.537+0000 7f7ad6800640 1 --2- 172.21.15.116:0/3804591948 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad01097e0 0x7f7ad0110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.537+0000 7f7ad6800640 1 --2- 172.21.15.116:0/3804591948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad0108e20 0x7f7ad01092a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.537+0000 7f7ad6800640 1 --2- 172.21.15.116:0/3804591948 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ad0107c30 0x7f7ad0108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.537+0000 7f7ad6800640 1 -- 172.21.15.116:0/3804591948 >> 172.21.15.116:0/3804591948 conn(0x7f7ad0076be0 msgr2=0x7f7ad0079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.537+0000 7f7ad6800640 1 -- 172.21.15.116:0/3804591948 shutdown_connections 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.537+0000 7f7ad6800640 1 -- 172.21.15.116:0/3804591948 wait complete. 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 Processor -- start 2024-03-04T07:44:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 -- start start 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ad0107c30 0x7f7ad019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad0108e20 0x7f7ad019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad01097e0 0x7f7ad01a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01138b0 con 0x7f7ad01097e0 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0113730 con 0x7f7ad0108e20 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ad6800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0113a30 con 0x7f7ad0107c30 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7acfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad01097e0 0x7f7ad01a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.538+0000 7f7ac7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad0108e20 0x7f7ad019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7acfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad01097e0 0x7f7ad01a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52622/0 (socket says 172.21.15.116:52622) 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ac7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad0108e20 0x7f7ad019e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:53852/0 (socket says 172.21.15.116:53852) 2024-03-04T07:44:42.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7acfe00640 1 -- 172.21.15.116:0/39173680 learned_addr learned my addr 172.21.15.116:0/39173680 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7acf400640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ad0107c30 0x7f7ad019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ac7e00640 1 -- 172.21.15.116:0/39173680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ad0107c30 msgr2=0x7f7ad019df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ad0107c30 0x7f7ad019df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ac7e00640 1 -- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad01097e0 msgr2=0x7f7ad01a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad01097e0 0x7f7ad01a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ac7e00640 1 -- 172.21.15.116:0/39173680 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad01a5b20 con 0x7f7ad0108e20 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7acf400640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ad0107c30 0x7f7ad019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad0108e20 0x7f7ad019e440 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8000c00 tx=0x7f7ab802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7accc00640 1 -- 172.21.15.116:0/39173680 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8041070 con 0x7f7ad0108e20 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7accc00640 1 -- 172.21.15.116:0/39173680 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ab8004450 con 0x7f7ad0108e20 2024-03-04T07:44:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.539+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad01a5db0 con 0x7f7ad0108e20 2024-03-04T07:44:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.540+0000 7f7accc00640 1 -- 172.21.15.116:0/39173680 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8034440 con 0x7f7ad0108e20 2024-03-04T07:44:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.540+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad01a6260 con 0x7f7ad0108e20 2024-03-04T07:44:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7accc00640 1 -- 172.21.15.116:0/39173680 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f7ab8044070 con 0x7f7ad0108e20 2024-03-04T07:44:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7ad95ad740 1 Processor -- start 2024-03-04T07:44:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7ad95ad740 1 -- start start 2024-03-04T07:44:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e16c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e1bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e1e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab48d7c370 con 0x55ab48e17240 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.543+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab48d25550 con 0x55ab48e1c3b0 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab48d78410 con 0x55ab48e16850 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e1bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e1bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52630/0 (socket says 172.21.15.116:52630) 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 -- 172.21.15.116:0/2651489937 learned_addr learned my addr 172.21.15.116:0/2651489937 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e1e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acfe00640 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e16c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 -- 172.21.15.116:0/2651489937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 msgr2=0x55ab48e16c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e16c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 -- 172.21.15.116:0/2651489937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 msgr2=0x55ab48e1e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e1e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 -- 172.21.15.116:0/2651489937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab48d393c0 con 0x55ab48e17240 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.544+0000 7f7acf400640 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e1bd40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009cf0 tx=0x7f7ac00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1652e6e0601bf6f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.545+0000 7f7accc00640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7ab40a1ea0 0x7f7ab40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.545+0000 7f7ac4c00640 1 -- 172.21.15.116:0/2651489937 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0012450 con 0x55ab48e17240 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.545+0000 7f7accc00640 1 -- 172.21.15.116:0/39173680 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7ab80ef860 con 0x7f7ad0108e20 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.545+0000 7f7acfe00640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7ab40a1ea0 0x7f7ab40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.545+0000 7f7ac4c00640 1 -- 172.21.15.116:0/2651489937 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ac0012ad0 con 0x55ab48e17240 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.545+0000 7f7ac4c00640 1 -- 172.21.15.116:0/2651489937 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000fb90 con 0x55ab48e17240 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2651489937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 msgr2=0x55ab48e1bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e1bd40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009cf0 tx=0x7f7ac00117b0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7acfe00640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7ab40a1ea0 0x7f7ab40a4360 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55ab48e17aa0 tx=0x7f7ac8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2651489937 shutdown_connections 2024-03-04T07:44:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e1e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e1bd40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2651489937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e16c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2651489937 >> 172.21.15.116:0/2651489937 conn(0x55ab48e11dc0 msgr2=0x55ab48e121c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2651489937 shutdown_connections 2024-03-04T07:44:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2651489937 wait complete. 2024-03-04T07:44:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 -- start start 2024-03-04T07:44:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.546+0000 7f7ad95ad740 1 Processor -- start 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e20b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e21060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e1c3b0 0x55ab48e280a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab48e1f360 con 0x55ab48e17240 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab48d389b0 con 0x55ab48e16850 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab48e1f4e0 con 0x55ab48e1c3b0 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ad95ad740 10 client.0 _ll_register_callbacks cb 0x55ab48d74340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7acf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e20b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7acf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e20b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:53884/0 (socket says 172.21.15.116:53884) 2024-03-04T07:44:42.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7acf400640 1 -- 172.21.15.116:0/3409603021 learned_addr learned my addr 172.21.15.116:0/3409603021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e21060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.547+0000 7f7acfe00640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e1c3b0 0x55ab48e280a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7acf400640 1 -- 172.21.15.116:0/3409603021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e1c3b0 msgr2=0x55ab48e280a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7acf400640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e1c3b0 0x55ab48e280a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7acf400640 1 -- 172.21.15.116:0/3409603021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 msgr2=0x55ab48e21060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7acf400640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e21060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7acf400640 1 -- 172.21.15.116:0/3409603021 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab48a91210 con 0x55ab48e16850 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e21060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7acf400640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e20b20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0005830 tx=0x7f7ac0012d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93894e4f68c5b45c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7abd600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7abd600640 1 -- 172.21.15.116:0/3409603021 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0028020 con 0x55ab48e16850 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7abd600640 1 -- 172.21.15.116:0/3409603021 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7ac0004250 con 0x55ab48e16850 2024-03-04T07:44:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.548+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ac0012ad0 con 0x55ab48e16850 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7abd600640 1 -- 172.21.15.116:0/3409603021 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ac0004450 con 0x55ab48e16850 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 10 client.26261 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ab48b5d620 con 0x55ab48e16850 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7abd600640 1 -- 172.21.15.116:0/3409603021 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ac0012ad0 con 0x55ab48e16850 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7abd600640 1 -- 172.21.15.116:0/3409603021 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7ac000f890 con 0x55ab48e16850 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 10 client.26261 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 10 client.26261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 20 client.26261 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e2c650 0x55ab48e2ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 4 client.26261 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab48d898b0 con 0x55ab48e2c650 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.549+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e2c650 0x55ab48e2ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.550+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e2c650 0x55ab48e2ea40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d2bf1b66b64a1d4 server_cookie=6689004c3ec2ab63 in_seq=0 out_seq=0 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.550+0000 7f7abd600640 10 client.26261 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.551+0000 7f7abd600640 1 -- 172.21.15.116:0/3409603021 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ab48d898b0 con 0x55ab48e2c650 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.551+0000 7f7abd600640 10 client.26261 handle_command_reply: tid=0 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.551+0000 7f7ad95ad740 1 client.26261 shutdown 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.551+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 msgr2=0x55ab48e20b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.551+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e20b20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0005830 tx=0x7f7ac0012d00 comp rx=0 tx=0).stop 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 shutdown_connections 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e2c650 0x55ab48e2ea40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e1c3b0 0x55ab48e280a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e21060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7abd600640 0 client.26261 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3409603021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e20b20 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 >> 172.21.15.116:0/3409603021 conn(0x55ab48e11dc0 msgr2=0x55ab48e10fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 shutdown_connections 2024-03-04T07:44:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3409603021 wait complete. 2024-03-04T07:44:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.552+0000 7f7ad95ad740 20 client.26261 trim_cache size 0 max 16384 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 Processor -- start 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 -- start start 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e10e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e28060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e33290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01acfa0 con 0x55ab48e17240 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01af580 con 0x55ab48e1c3b0 2024-03-04T07:44:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.567+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01ad5a0 con 0x55ab48e16850 2024-03-04T07:44:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e33290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7ac7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e28060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e33290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:53886/0 (socket says 172.21.15.116:53886) 2024-03-04T07:44:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 -- 172.21.15.116:0/2089603978 learned_addr learned my addr 172.21.15.116:0/2089603978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acf400640 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e10e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 -- 172.21.15.116:0/2089603978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 msgr2=0x55ab48e10e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e10e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 -- 172.21.15.116:0/2089603978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 msgr2=0x55ab48e28060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e28060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 -- 172.21.15.116:0/2089603978 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab48e2f070 con 0x55ab48e1c3b0 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.568+0000 7f7acfe00640 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e33290 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800f640 tx=0x7f7ac801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=123f9f3ed4e19995 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7abfe00640 1 -- 172.21.15.116:0/2089603978 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8004b20 con 0x55ab48e1c3b0 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7abfe00640 1 -- 172.21.15.116:0/2089603978 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ac8004cc0 con 0x55ab48e1c3b0 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7abfe00640 1 -- 172.21.15.116:0/2089603978 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8005560 con 0x55ab48e1c3b0 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2089603978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 msgr2=0x55ab48e33290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e33290 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800f640 tx=0x7f7ac801a000 comp rx=0 tx=0).stop 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2089603978 shutdown_connections 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e1c3b0 0x55ab48e33290 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e17240 0x55ab48e28060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/2089603978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e16850 0x55ab48e10e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2089603978 >> 172.21.15.116:0/2089603978 conn(0x55ab48e11dc0 msgr2=0x55ab48e10750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2089603978 shutdown_connections 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.569+0000 7f7ad95ad740 1 -- 172.21.15.116:0/2089603978 wait complete. 2024-03-04T07:44:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 -- start start 2024-03-04T07:44:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 Processor -- start 2024-03-04T07:44:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e25ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e17240 0x55ab48e261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e1c3b0 0x55ab48e27180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01adc00 con 0x55ab48e1c3b0 2024-03-04T07:44:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01aef20 con 0x55ab48e16850 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01ae260 con 0x55ab48e17240 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.570+0000 7f7ad95ad740 10 client.0 _ll_register_callbacks cb 0x7f7ab80f5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e25ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e25ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:53892/0 (socket says 172.21.15.116:53892) 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 -- 172.21.15.116:0/3270791957 learned_addr learned my addr 172.21.15.116:0/3270791957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acfe00640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e1c3b0 0x55ab48e27180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e17240 0x55ab48e261e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 -- 172.21.15.116:0/3270791957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e17240 msgr2=0x55ab48e261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e17240 0x55ab48e261e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 -- 172.21.15.116:0/3270791957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e1c3b0 msgr2=0x55ab48e27180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e1c3b0 0x55ab48e27180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 -- 172.21.15.116:0/3270791957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab48e11710 con 0x55ab48e16850 2024-03-04T07:44:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e17240 0x55ab48e261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7acf400640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e25ca0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0005860 tx=0x7f7ac000fef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76be64bc40975609 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7abd600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.571+0000 7f7abd600640 1 -- 172.21.15.116:0/3270791957 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0028040 con 0x55ab48e16850 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7abd600640 1 -- 172.21.15.116:0/3270791957 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7ac0005040 con 0x55ab48e16850 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ac8004cc0 con 0x55ab48e16850 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7abd600640 1 -- 172.21.15.116:0/3270791957 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ac0005280 con 0x55ab48e16850 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 10 client.26267 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ab48e881b0 con 0x55ab48e16850 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7abd600640 1 -- 172.21.15.116:0/3270791957 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ac8004cc0 con 0x55ab48e16850 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7abd600640 1 -- 172.21.15.116:0/3270791957 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7ac0041030 con 0x55ab48e16850 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 10 client.26267 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 10 client.26267 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 20 client.26267 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e88ad0 0x55ab48e8aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 4 client.26267 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:44:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.572+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ab48d83c00 con 0x55ab48e88ad0 2024-03-04T07:44:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.573+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e88ad0 0x55ab48e8aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.573+0000 7f7ac7e00640 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e88ad0 0x55ab48e8aef0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ead7d9bc56d42904 server_cookie=2bed63192c552095 in_seq=0 out_seq=0 2024-03-04T07:44:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.573+0000 7f7abd600640 10 client.26267 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:42.590 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7abd600640 1 -- 172.21.15.116:0/3270791957 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ab48d83c00 con 0x55ab48e88ad0 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7abd600640 10 client.26267 handle_command_reply: tid=0 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 client.26267 shutdown 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 msgr2=0x55ab48e25ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e25ca0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0005860 tx=0x7f7ac000fef0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 shutdown_connections 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ab48e88ad0 0x55ab48e8aef0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab48e1c3b0 0x55ab48e27180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab48e17240 0x55ab48e261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 --2- 172.21.15.116:0/3270791957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ab48e16850 0x55ab48e25ca0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7abd600640 0 client.26267 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 >> 172.21.15.116:0/3270791957 conn(0x55ab48e11dc0 msgr2=0x55ab48e10fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 shutdown_connections 2024-03-04T07:44:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.589+0000 7f7ad95ad740 1 -- 172.21.15.116:0/3270791957 wait complete. 2024-03-04T07:44:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad95ad740 20 client.26267 trim_cache size 0 max 16384 2024-03-04T07:44:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:44:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:44:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:44:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7ab40a1ea0 msgr2=0x7f7ab40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7ab40a1ea0 0x7f7ab40a4360 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55ab48e17aa0 tx=0x7f7ac8009500 comp rx=0 tx=0).stop 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad0108e20 msgr2=0x7f7ad019e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad0108e20 0x7f7ad019e440 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8000c00 tx=0x7f7ab802c640 comp rx=0 tx=0).stop 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 shutdown_connections 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad01097e0 0x7f7ad01a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7ad0108e20 0x7f7ad019e440 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.590+0000 7f7ad6800640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7ab40a1ea0 0x7f7ab40a4360 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.591+0000 7f7ad6800640 1 --2- 172.21.15.116:0/39173680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ad0107c30 0x7f7ad019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.591+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 >> 172.21.15.116:0/39173680 conn(0x7f7ad0076be0 msgr2=0x7f7ad010c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.591+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 shutdown_connections 2024-03-04T07:44:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:42.591+0000 7f7ad6800640 1 -- 172.21.15.116:0/39173680 wait complete. 2024-03-04T07:44:43.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:42 smithi116 ceph-mon[32150]: pgmap v519: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:43.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:42 smithi152 ceph-mon[37154]: pgmap v519: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:43.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:42 smithi133 ceph-mon[36505]: pgmap v519: 97 pgs: 97 active+clean; 2.7 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:44:45.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:44 smithi116 ceph-mon[32150]: pgmap v520: 97 pgs: 97 active+clean; 2.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-03-04T07:44:45.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:44 smithi152 ceph-mon[37154]: pgmap v520: 97 pgs: 97 active+clean; 2.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-03-04T07:44:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:44 smithi133 ceph-mon[36505]: pgmap v520: 97 pgs: 97 active+clean; 2.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 10 op/s 2024-03-04T07:44:47.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:46 smithi116 ceph-mon[32150]: pgmap v521: 97 pgs: 97 active+clean; 2.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:44:47.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:46 smithi152 ceph-mon[37154]: pgmap v521: 97 pgs: 97 active+clean; 2.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:44:47.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:46 smithi133 ceph-mon[36505]: pgmap v521: 97 pgs: 97 active+clean; 2.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:44:49.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:48 smithi116 ceph-mon[32150]: pgmap v522: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:49.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:48 smithi152 ceph-mon[37154]: pgmap v522: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:48 smithi133 ceph-mon[36505]: pgmap v522: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:51.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:50 smithi116 ceph-mon[32150]: pgmap v523: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:51.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:50 smithi152 ceph-mon[37154]: pgmap v523: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:51.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:50 smithi133 ceph-mon[36505]: pgmap v523: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:53.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:52 smithi116 ceph-mon[32150]: pgmap v524: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:44:53.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:52 smithi152 ceph-mon[37154]: pgmap v524: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:44:53.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:52 smithi133 ceph-mon[36505]: pgmap v524: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:44:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:53 smithi116 ceph-mon[32150]: pgmap v525: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:54.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:53 smithi152 ceph-mon[37154]: pgmap v525: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:53 smithi133 ceph-mon[36505]: pgmap v525: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:44:55.194 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:44:55.195 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:44:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 -- 172.21.15.116:0/4269969656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df80771d0 msgr2=0x7f3df8077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 --2- 172.21.15.116:0/4269969656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df80771d0 0x7f3df8077630 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009ca0 tx=0x7f3de002c100 comp rx=0 tx=0).stop 2024-03-04T07:44:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 -- 172.21.15.116:0/4269969656 shutdown_connections 2024-03-04T07:44:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 --2- 172.21.15.116:0/4269969656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3df8077b70 0x7f3df8078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 --2- 172.21.15.116:0/4269969656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df80771d0 0x7f3df8077630 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 --2- 172.21.15.116:0/4269969656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df8075f90 0x7f3df80763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 -- 172.21.15.116:0/4269969656 >> 172.21.15.116:0/4269969656 conn(0x7f3df80fed80 msgr2=0x7f3df81011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 -- 172.21.15.116:0/4269969656 shutdown_connections 2024-03-04T07:44:55.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.410+0000 7f3dfe800640 1 -- 172.21.15.116:0/4269969656 wait complete. 2024-03-04T07:44:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.411+0000 7f3dfe800640 1 Processor -- start 2024-03-04T07:44:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3dfe800640 1 -- start start 2024-03-04T07:44:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3dfe800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3df8075f90 0x7f3df819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3dfe800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80771d0 0x7f3df819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3dfe800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df8077b70 0x7f3df81a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3dfe800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8113860 con 0x7f3df80771d0 2024-03-04T07:44:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3dfe800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81136e0 con 0x7f3df8077b70 2024-03-04T07:44:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3dfe800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81139e0 con 0x7f3df8075f90 2024-03-04T07:44:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3deea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80771d0 0x7f3df819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3df7400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3df8075f90 0x7f3df819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3df7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df8077b70 0x7f3df81a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3deea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80771d0 0x7f3df819e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40778/0 (socket says 172.21.15.116:40778) 2024-03-04T07:44:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.412+0000 7f3deea00640 1 -- 172.21.15.116:0/1422651888 learned_addr learned my addr 172.21.15.116:0/1422651888 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3deea00640 1 -- 172.21.15.116:0/1422651888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3df8075f90 msgr2=0x7f3df819df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3deea00640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3df8075f90 0x7f3df819df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3deea00640 1 -- 172.21.15.116:0/1422651888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df8077b70 msgr2=0x7f3df81a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3deea00640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df8077b70 0x7f3df81a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3deea00640 1 -- 172.21.15.116:0/1422651888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df81a5b80 con 0x7f3df80771d0 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3df7e00640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df8077b70 0x7f3df81a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3df7400640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3df8075f90 0x7f3df819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3deea00640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80771d0 0x7f3df819e4a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3de00325f0 tx=0x7f3de0035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.413+0000 7f3df4c00640 1 -- 172.21.15.116:0/1422651888 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0039040 con 0x7f3df80771d0 2024-03-04T07:44:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.414+0000 7f3df4c00640 1 -- 172.21.15.116:0/1422651888 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3de0044070 con 0x7f3df80771d0 2024-03-04T07:44:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.414+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df81a5e10 con 0x7f3df80771d0 2024-03-04T07:44:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.414+0000 7f3df4c00640 1 -- 172.21.15.116:0/1422651888 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0004540 con 0x7f3df80771d0 2024-03-04T07:44:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.414+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df81a6270 con 0x7f3df80771d0 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 Processor -- start 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 -- start start 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3df4c00640 1 -- 172.21.15.116:0/1422651888 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3de0032890 con 0x7f3df80771d0 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188772bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772c540 0x561887731040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x561887733b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561887659090 con 0x56188772bb50 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561887690d30 con 0x56188772c540 2024-03-04T07:44:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.418+0000 7f3e015b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561887639a30 con 0x5618877316b0 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772c540 0x561887731040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772c540 0x561887731040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50954/0 (socket says 172.21.15.116:50954) 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 -- 172.21.15.116:0/2491389716 learned_addr learned my addr 172.21.15.116:0/2491389716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7e00640 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188772bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3deea00640 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x561887733b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 -- 172.21.15.116:0/2491389716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 msgr2=0x561887733b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x561887733b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 -- 172.21.15.116:0/2491389716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 msgr2=0x56188772bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188772bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 -- 172.21.15.116:0/2491389716 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56188746f7d0 con 0x56188772c540 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3deea00640 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x561887733b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3df7400640 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772c540 0x561887731040 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3de80115f0 tx=0x7f3de8011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4e893932ec29e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.419+0000 7f3decc00640 1 -- 172.21.15.116:0/2491389716 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de801e070 con 0x56188772c540 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3decc00640 1 -- 172.21.15.116:0/2491389716 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3de800fb20 con 0x56188772c540 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3decc00640 1 -- 172.21.15.116:0/2491389716 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8019c40 con 0x56188772c540 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3df4c00640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3ddc0a1d80 0x7f3ddc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3df4c00640 1 -- 172.21.15.116:0/1422651888 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3de00f03f0 con 0x7f3df80771d0 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3df7e00640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3ddc0a1d80 0x7f3ddc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3e015b9740 1 -- 172.21.15.116:0/2491389716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772c540 msgr2=0x561887731040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3e015b9740 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772c540 0x561887731040 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3de80115f0 tx=0x7f3de8011ab0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3e015b9740 1 -- 172.21.15.116:0/2491389716 shutdown_connections 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3e015b9740 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x561887733b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3e015b9740 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772c540 0x561887731040 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3e015b9740 1 --2- 172.21.15.116:0/2491389716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188772bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.420+0000 7f3e015b9740 1 -- 172.21.15.116:0/2491389716 >> 172.21.15.116:0/2491389716 conn(0x5618877270c0 msgr2=0x5618877274c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.421+0000 7f3e015b9740 1 -- 172.21.15.116:0/2491389716 shutdown_connections 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.421+0000 7f3df7e00640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3ddc0a1d80 0x7f3ddc0a4240 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x56188772cda0 tx=0x7f3df0005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.421+0000 7f3e015b9740 1 -- 172.21.15.116:0/2491389716 wait complete. 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.421+0000 7f3e015b9740 1 -- start start 2024-03-04T07:44:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.421+0000 7f3e015b9740 1 Processor -- start 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887735f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772c540 0x561887736480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x56188773d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3e015b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56188762f0c0 con 0x56188772c540 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3e015b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56188768cdd0 con 0x56188772bb50 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3e015b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561887734660 con 0x5618877316b0 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3e015b9740 10 client.0 _ll_register_callbacks cb 0x56188764c860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887735f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3deea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772c540 0x561887736480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887735f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50958/0 (socket says 172.21.15.116:50958) 2024-03-04T07:44:55.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 -- 172.21.15.116:0/3395253667 learned_addr learned my addr 172.21.15.116:0/3395253667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7e00640 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x56188773d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 -- 172.21.15.116:0/3395253667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 msgr2=0x56188773d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x56188773d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 -- 172.21.15.116:0/3395253667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772c540 msgr2=0x561887736480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772c540 0x561887736480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.422+0000 7f3df7400640 1 -- 172.21.15.116:0/3395253667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618873a53f0 con 0x56188772bb50 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3df7400640 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887735f40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3de80115c0 tx=0x7f3de800f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9c32aebf163e78b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3de5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3de5600640 1 -- 172.21.15.116:0/3395253667 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8027060 con 0x56188772bb50 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3de800fb20 con 0x56188772bb50 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3de5600640 1 -- 172.21.15.116:0/3395253667 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3de80199c0 con 0x56188772bb50 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3de5600640 1 -- 172.21.15.116:0/3395253667 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3de801e040 con 0x56188772bb50 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3e015b9740 10 client.26279 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:55.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5618873650f0 con 0x56188772bb50 2024-03-04T07:44:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.423+0000 7f3de5600640 1 -- 172.21.15.116:0/3395253667 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3de800fb20 con 0x56188772bb50 2024-03-04T07:44:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3de5600640 1 -- 172.21.15.116:0/3395253667 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3de802c070 con 0x56188772bb50 2024-03-04T07:44:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3e015b9740 10 client.26279 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3e015b9740 10 client.26279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3e015b9740 20 client.26279 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561887741770 0x561887743b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3e015b9740 4 client.26279 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56188769e270 con 0x561887741770 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3deea00640 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561887741770 0x561887743b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3deea00640 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561887741770 0x561887743b60 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ecc8788107a6834 server_cookie=b3958c0d6fe829cd in_seq=0 out_seq=0 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.424+0000 7f3de5600640 10 client.26279 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3de5600640 1 -- 172.21.15.116:0/3395253667 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56188769e270 con 0x561887741770 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3de5600640 10 client.26279 handle_command_reply: tid=0 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 client.26279 shutdown 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 msgr2=0x561887735f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887735f40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3de80115c0 tx=0x7f3de800f870 comp rx=0 tx=0).stop 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 shutdown_connections 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561887741770 0x561887743b60 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5618877316b0 0x56188773d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772c540 0x561887736480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3395253667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887735f40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3de5600640 0 client.26279 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 >> 172.21.15.116:0/3395253667 conn(0x5618877270c0 msgr2=0x561887726360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 shutdown_connections 2024-03-04T07:44:55.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 1 -- 172.21.15.116:0/3395253667 wait complete. 2024-03-04T07:44:55.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.426+0000 7f3e015b9740 20 client.26279 trim_cache size 0 max 16384 2024-03-04T07:44:55.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.443+0000 7f3e015b9740 1 Processor -- start 2024-03-04T07:44:55.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.443+0000 7f3e015b9740 1 -- start start 2024-03-04T07:44:55.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887748290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5618877316b0 0x5618877487d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3e015b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81ad040 con 0x5618877316b0 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3e015b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81af620 con 0x56188772bb50 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3e015b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81ad640 con 0x56188772c540 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887748290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887748290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50960/0 (socket says 172.21.15.116:50960) 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 -- 172.21.15.116:0/3025651013 learned_addr learned my addr 172.21.15.116:0/3025651013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7e00640 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5618877316b0 0x5618877487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3deea00640 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 -- 172.21.15.116:0/3025651013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 msgr2=0x56188773d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 -- 172.21.15.116:0/3025651013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5618877316b0 msgr2=0x5618877487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5618877316b0 0x5618877487d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7400640 1 -- 172.21.15.116:0/3025651013 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56188772d510 con 0x56188772bb50 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.444+0000 7f3df7e00640 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5618877316b0 0x5618877487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3df7400640 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887748290 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3de8004740 tx=0x7f3de80059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d5b43312fcee5d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3de7e00640 1 -- 172.21.15.116:0/3025651013 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8032070 con 0x56188772bb50 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3de7e00640 1 -- 172.21.15.116:0/3025651013 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3de8004510 con 0x56188772bb50 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3de7e00640 1 -- 172.21.15.116:0/3025651013 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8027030 con 0x56188772bb50 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3e015b9740 1 -- 172.21.15.116:0/3025651013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 msgr2=0x561887748290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887748290 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3de8004740 tx=0x7f3de80059b0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3e015b9740 1 -- 172.21.15.116:0/3025651013 shutdown_connections 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5618877316b0 0x5618877487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3e015b9740 1 --2- 172.21.15.116:0/3025651013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56188772bb50 0x561887748290 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.445+0000 7f3e015b9740 1 -- 172.21.15.116:0/3025651013 >> 172.21.15.116:0/3025651013 conn(0x5618877270c0 msgr2=0x56188772a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.446+0000 7f3e015b9740 1 -- 172.21.15.116:0/3025651013 shutdown_connections 2024-03-04T07:44:55.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.446+0000 7f3e015b9740 1 -- 172.21.15.116:0/3025651013 wait complete. 2024-03-04T07:44:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.446+0000 7f3e015b9740 1 -- start start 2024-03-04T07:44:55.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.446+0000 7f3e015b9740 1 Processor -- start 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188773b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3e015b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5618877316b0 0x56188773c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3e015b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81adca0 con 0x56188772bb50 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3e015b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81aefc0 con 0x5618877316b0 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3e015b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81ae300 con 0x56188772c540 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3e015b9740 10 client.0 _ll_register_callbacks cb 0x7f3de8009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188773b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5618877316b0 0x56188773c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188773b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40804/0 (socket says 172.21.15.116:40804) 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7400640 1 -- 172.21.15.116:0/1803823218 learned_addr learned my addr 172.21.15.116:0/1803823218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3deea00640 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7e00640 1 -- 172.21.15.116:0/1803823218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 msgr2=0x56188773b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7e00640 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7e00640 1 -- 172.21.15.116:0/1803823218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 msgr2=0x56188773b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7e00640 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188773b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7e00640 1 -- 172.21.15.116:0/1803823218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56188773cb90 con 0x5618877316b0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.447+0000 7f3df7400640 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188773b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3df7e00640 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5618877316b0 0x56188773c5a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3df000bfd0 tx=0x7f3df00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccd6bb9b1815ee68 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3de5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3de5600640 1 -- 172.21.15.116:0/1803823218 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df00109c0 con 0x5618877316b0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3de5600640 1 -- 172.21.15.116:0/1803823218 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3df0010b60 con 0x5618877316b0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3de8004510 con 0x5618877316b0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3de5600640 1 -- 172.21.15.116:0/1803823218 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3df0010d60 con 0x5618877316b0 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3e015b9740 10 client.26285 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:55.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56188779d360 con 0x5618877316b0 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.448+0000 7f3de5600640 1 -- 172.21.15.116:0/1803823218 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3de8004510 con 0x5618877316b0 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3de5600640 1 -- 172.21.15.116:0/1803823218 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3df001b040 con 0x5618877316b0 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3e015b9740 10 client.26285 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3e015b9740 10 client.26285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3e015b9740 20 client.26285 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3e015b9740 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56188779dd30 0x5618877a0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3e015b9740 4 client.26285 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5618876985c0 con 0x56188779dd30 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3df7400640 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56188779dd30 0x5618877a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3de5600640 10 client.26285 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.449+0000 7f3df7400640 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56188779dd30 0x5618877a0150 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=604ae8e72ad732ca server_cookie=2e185681c74df883 in_seq=0 out_seq=0 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3de5600640 1 -- 172.21.15.116:0/1803823218 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5618876985c0 con 0x56188779dd30 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3de5600640 10 client.26285 handle_command_reply: tid=0 2024-03-04T07:44:55.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 client.26285 shutdown 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5618877316b0 msgr2=0x56188773c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5618877316b0 0x56188773c5a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3df000bfd0 tx=0x7f3df00046c0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 shutdown_connections 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56188779dd30 0x5618877a0150 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5618877316b0 0x56188773c5a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3de5600640 0 client.26285 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56188772c540 0x56188773b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 --2- 172.21.15.116:0/1803823218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56188772bb50 0x56188773b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 >> 172.21.15.116:0/1803823218 conn(0x5618877270c0 msgr2=0x561887725ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 shutdown_connections 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 1 -- 172.21.15.116:0/1803823218 wait complete. 2024-03-04T07:44:55.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.450+0000 7f3e015b9740 20 client.26285 trim_cache size 0 max 16384 2024-03-04T07:44:55.456 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:55.456 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:44:55.456 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:44:55.456 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3ddc0a1d80 msgr2=0x7f3ddc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3ddc0a1d80 0x7f3ddc0a4240 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x56188772cda0 tx=0x7f3df0005cc0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80771d0 msgr2=0x7f3df819e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80771d0 0x7f3df819e4a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3de00325f0 tx=0x7f3de0035d10 comp rx=0 tx=0).stop 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 shutdown_connections 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3df8077b70 0x7f3df81a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3df80771d0 0x7f3df819e4a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3ddc0a1d80 0x7f3ddc0a4240 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 --2- 172.21.15.116:0/1422651888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3df8075f90 0x7f3df819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 >> 172.21.15.116:0/1422651888 conn(0x7f3df80fed80 msgr2=0x7f3df81004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 shutdown_connections 2024-03-04T07:44:55.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.451+0000 7f3dfe800640 1 -- 172.21.15.116:0/1422651888 wait complete. 2024-03-04T07:44:55.468 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:44:55.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:44:55.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 -- 172.21.15.116:0/3950859863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c104a20 msgr2=0x7f2f1c106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 --2- 172.21.15.116:0/3950859863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c104a20 0x7f2f1c106e10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009900 tx=0x7f2f0802bc40 comp rx=0 tx=0).stop 2024-03-04T07:44:55.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 -- 172.21.15.116:0/3950859863 shutdown_connections 2024-03-04T07:44:55.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 --2- 172.21.15.116:0/3950859863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c107350 0x7f2f1c1097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 --2- 172.21.15.116:0/3950859863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c104a20 0x7f2f1c106e10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 --2- 172.21.15.116:0/3950859863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 0x7f2f1c1044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 -- 172.21.15.116:0/3950859863 >> 172.21.15.116:0/3950859863 conn(0x7f2f1c0fbb20 msgr2=0x7f2f1c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 -- 172.21.15.116:0/3950859863 shutdown_connections 2024-03-04T07:44:55.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.679+0000 7f2f23e00640 1 -- 172.21.15.116:0/3950859863 wait complete. 2024-03-04T07:44:55.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.680+0000 7f2f23e00640 1 Processor -- start 2024-03-04T07:44:55.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.680+0000 7f2f23e00640 1 -- start start 2024-03-04T07:44:55.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f23e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 0x7f2f1c197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f23e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c104a20 0x7f2f1c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f23e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c107350 0x7f2f1c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f23e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c10d2d0 con 0x7f2f1c104a20 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f23e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c10d150 con 0x7f2f1c107350 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f23e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c10d450 con 0x7f2f1c1020f0 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f20a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c104a20 0x7f2f1c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f21400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 0x7f2f1c197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f20a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c104a20 0x7f2f1c197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40824/0 (socket says 172.21.15.116:40824) 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f20a00640 1 -- 172.21.15.116:0/1422226823 learned_addr learned my addr 172.21.15.116:0/1422226823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f21400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 0x7f2f1c197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:34434/0 (socket says 172.21.15.116:34434) 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.681+0000 7f2f21e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c107350 0x7f2f1c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f21400640 1 -- 172.21.15.116:0/1422226823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c107350 msgr2=0x7f2f1c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f21400640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c107350 0x7f2f1c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f21400640 1 -- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c104a20 msgr2=0x7f2f1c197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f21400640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c104a20 0x7f2f1c197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f21400640 1 -- 172.21.15.116:0/1422226823 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c19f530 con 0x7f2f1c1020f0 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f20a00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c104a20 0x7f2f1c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f21e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c107350 0x7f2f1c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f21400640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 0x7f2f1c197910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000e9f0 tx=0x7f2f1000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f1a000640 1 -- 172.21.15.116:0/1422226823 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1000ccc0 con 0x7f2f1c1020f0 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1c19f820 con 0x7f2f1c1020f0 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f1a000640 1 -- 172.21.15.116:0/1422226823 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f10005020 con 0x7f2f1c1020f0 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f1a000640 1 -- 172.21.15.116:0/1422226823 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10004a00 con 0x7f2f1c1020f0 2024-03-04T07:44:55.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.682+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1c19fd40 con 0x7f2f1c1020f0 2024-03-04T07:44:55.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.684+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1c0fd610 con 0x7f2f1c1020f0 2024-03-04T07:44:55.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.686+0000 7f2f1a000640 1 -- 172.21.15.116:0/1422226823 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2f100051d0 con 0x7f2f1c1020f0 2024-03-04T07:44:55.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.687+0000 7f2f1a000640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2f040a1e00 0x7f2f040a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.687+0000 7f2f1a000640 1 -- 172.21.15.116:0/1422226823 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2f100cdc90 con 0x7f2f1c1020f0 2024-03-04T07:44:55.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.687+0000 7f2f20a00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2f040a1e00 0x7f2f040a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.688+0000 7f2f20a00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2f040a1e00 0x7f2f040a42c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f08005ab0 tx=0x7f2f0802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.691+0000 7f2f1a000640 1 -- 172.21.15.116:0/1422226823 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2f1009b080 con 0x7f2f1c1020f0 2024-03-04T07:44:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 -- 172.21.15.116:0/3729930879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94101ea0 msgr2=0x7fce94104290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 --2- 172.21.15.116:0/3729930879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94101ea0 0x7fce94104290 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fce84009900 tx=0x7fce8402bc30 comp rx=0 tx=0).stop 2024-03-04T07:44:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 -- 172.21.15.116:0/3729930879 shutdown_connections 2024-03-04T07:44:55.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 --2- 172.21.15.116:0/3729930879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce94107100 0x7fce94109550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 --2- 172.21.15.116:0/3729930879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce941047d0 0x7fce94106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 --2- 172.21.15.116:0/3729930879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94101ea0 0x7fce94104290 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 -- 172.21.15.116:0/3729930879 >> 172.21.15.116:0/3729930879 conn(0x7fce940fb8b0 msgr2=0x7fce940fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 -- 172.21.15.116:0/3729930879 shutdown_connections 2024-03-04T07:44:55.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.730+0000 7fce99400640 1 -- 172.21.15.116:0/3729930879 wait complete. 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.731+0000 7fce99400640 1 Processor -- start 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.731+0000 7fce99400640 1 -- start start 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce99400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce94101ea0 0x7fce94199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce99400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce941047d0 0x7fce94199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce99400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94107100 0x7fce941a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce99400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9410d100 con 0x7fce941047d0 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce99400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9410cf80 con 0x7fce94107100 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce99400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9410d280 con 0x7fce94101ea0 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce93400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce941047d0 0x7fce94199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94107100 0x7fce941a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94107100 0x7fce941a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51002/0 (socket says 172.21.15.116:51002) 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce93e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce94101ea0 0x7fce94199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 -- 172.21.15.116:0/173490596 learned_addr learned my addr 172.21.15.116:0/173490596 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 -- 172.21.15.116:0/173490596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce94101ea0 msgr2=0x7fce94199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce94101ea0 0x7fce94199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 -- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce941047d0 msgr2=0x7fce94199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce941047d0 0x7fce94199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.732+0000 7fce98a00640 1 -- 172.21.15.116:0/173490596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce941a14e0 con 0x7fce94107100 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.733+0000 7fce98a00640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94107100 0x7fce941a0e70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00eb10 tx=0x7fce8c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.733+0000 7fce90c00640 1 -- 172.21.15.116:0/173490596 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c004b20 con 0x7fce94107100 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.733+0000 7fce90c00640 1 -- 172.21.15.116:0/173490596 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce8c004cc0 con 0x7fce94107100 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.733+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce941a17d0 con 0x7fce94107100 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.733+0000 7fce90c00640 1 -- 172.21.15.116:0/173490596 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c010760 con 0x7fce94107100 2024-03-04T07:44:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.733+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce941a1d70 con 0x7fce94107100 2024-03-04T07:44:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.737+0000 7fce9d76b740 1 Processor -- start 2024-03-04T07:44:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce90c00640 1 -- 172.21.15.116:0/173490596 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fce8c005020 con 0x7fce94107100 2024-03-04T07:44:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce9d76b740 1 -- start start 2024-03-04T07:44:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce9d76b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd032370 con 0x55bcbd0cd400 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce9d76b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbcfdb550 con 0x55bcbd0cca10 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce9d76b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd02e410 con 0x55bcbd0d2570 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51006/0 (socket says 172.21.15.116:51006) 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce93400640 1 -- 172.21.15.116:0/3638507280 learned_addr learned my addr 172.21.15.116:0/3638507280 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce93e00640 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.738+0000 7fce93400640 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce93400640 1 -- 172.21.15.116:0/3638507280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 msgr2=0x55bcbd0d4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce93400640 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0d4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce93400640 1 -- 172.21.15.116:0/3638507280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cd400 msgr2=0x55bcbd0d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce93400640 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce93400640 1 -- 172.21.15.116:0/3638507280 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbcfef3c0 con 0x55bcbd0cca10 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce90c00640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce780a1df0 0x7fce780a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce90c00640 1 -- 172.21.15.116:0/173490596 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fce8c0ce400 con 0x7fce94107100 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce93400640 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cce30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce800058c0 tx=0x7fce80005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49c9e439a45b496 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.739+0000 7fce93400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce780a1df0 0x7fce780a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce89200640 1 -- 172.21.15.116:0/3638507280 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80004030 con 0x55bcbd0cca10 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce89200640 1 -- 172.21.15.116:0/3638507280 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce8000fc70 con 0x55bcbd0cca10 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce89200640 1 -- 172.21.15.116:0/3638507280 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80003cd0 con 0x55bcbd0cca10 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce9d76b740 1 -- 172.21.15.116:0/3638507280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 msgr2=0x55bcbd0cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce93400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce780a1df0 0x7fce780a42b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fce8001e000 tx=0x7fce80003f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce9d76b740 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cce30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce800058c0 tx=0x7fce80005850 comp rx=0 tx=0).stop 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce9d76b740 1 -- 172.21.15.116:0/3638507280 shutdown_connections 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce9d76b740 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0d4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce9d76b740 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce9d76b740 1 --2- 172.21.15.116:0/3638507280 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cce30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.740+0000 7fce9d76b740 1 -- 172.21.15.116:0/3638507280 >> 172.21.15.116:0/3638507280 conn(0x55bcbd0c7f80 msgr2=0x55bcbd0c8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.741+0000 7fce9d76b740 1 -- 172.21.15.116:0/3638507280 shutdown_connections 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.741+0000 7fce9d76b740 1 -- 172.21.15.116:0/3638507280 wait complete. 2024-03-04T07:44:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.741+0000 7fce9d76b740 1 -- start start 2024-03-04T07:44:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.741+0000 7fce9d76b740 1 Processor -- start 2024-03-04T07:44:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0d6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0de240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce9d76b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd0d5520 con 0x55bcbd0cca10 2024-03-04T07:44:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce9d76b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbcfee9b0 con 0x55bcbd0cd400 2024-03-04T07:44:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce9d76b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd0d56a0 con 0x55bcbd0d2570 2024-03-04T07:44:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce9d76b740 10 client.0 _ll_register_callbacks cb 0x55bcbd02a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0d6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51020/0 (socket says 172.21.15.116:51020) 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 -- 172.21.15.116:0/2448765755 learned_addr learned my addr 172.21.15.116:0/2448765755 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce98a00640 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 -- 172.21.15.116:0/2448765755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 msgr2=0x55bcbd0de240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0de240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 -- 172.21.15.116:0/2448765755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 msgr2=0x55bcbd0d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0d6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.742+0000 7fce93400640 1 -- 172.21.15.116:0/2448765755 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcbcd47210 con 0x55bcbd0cd400 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce93400640 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d7200 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fce80003d40 tx=0x7fce80016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb660afc032181f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce7d600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce7d600640 1 -- 172.21.15.116:0/2448765755 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8000fe20 con 0x55bcbd0cd400 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce7d600640 1 -- 172.21.15.116:0/2448765755 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fce80028030 con 0x55bcbd0cd400 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce8000fc70 con 0x55bcbd0cd400 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce7d600640 1 -- 172.21.15.116:0/2448765755 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce8000f790 con 0x55bcbd0cd400 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce9d76b740 10 client.26303 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bcbce13620 con 0x55bcbd0cd400 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.743+0000 7fce7d600640 1 -- 172.21.15.116:0/2448765755 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce8000fc70 con 0x55bcbd0cd400 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce7d600640 1 -- 172.21.15.116:0/2448765755 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fce800050d0 con 0x55bcbd0cd400 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce9d76b740 10 client.26303 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce9d76b740 10 client.26303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce9d76b740 20 client.26303 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd0e2830 0x55bcbd0e4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce9d76b740 4 client.26303 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcbd03f8b0 con 0x55bcbd0e2830 2024-03-04T07:44:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.744+0000 7fce93e00640 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd0e2830 0x55bcbd0e4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.745+0000 7fce7d600640 10 client.26303 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.745+0000 7fce93e00640 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd0e2830 0x55bcbd0e4c80 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c22a64e251e426fb server_cookie=d6dee8842d6e70b2 in_seq=0 out_seq=0 2024-03-04T07:44:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.745+0000 7fce7d600640 1 -- 172.21.15.116:0/2448765755 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bcbd03f8b0 con 0x55bcbd0e2830 2024-03-04T07:44:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.745+0000 7fce7d600640 10 client.26303 handle_command_reply: tid=0 2024-03-04T07:44:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.745+0000 7fce9d76b740 1 client.26303 shutdown 2024-03-04T07:44:55.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.745+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 msgr2=0x55bcbd0d7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.745+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d7200 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fce80003d40 tx=0x7fce80016000 comp rx=0 tx=0).stop 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 shutdown_connections 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd0e2830 0x55bcbd0e4c80 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0de240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0d7200 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce7d600640 0 client.26303 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2448765755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0d6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 >> 172.21.15.116:0/2448765755 conn(0x55bcbd0c7f80 msgr2=0x55bcbd0c7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 shutdown_connections 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 1 -- 172.21.15.116:0/2448765755 wait complete. 2024-03-04T07:44:55.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.746+0000 7fce9d76b740 20 client.26303 trim_cache size 0 max 16384 2024-03-04T07:44:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 Processor -- start 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 -- start start 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0de200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941a8b10 con 0x55bcbd0cca10 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941ab0f0 con 0x55bcbd0cd400 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce9d76b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941a9110 con 0x55bcbd0d2570 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0de200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce93e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0de200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51022/0 (socket says 172.21.15.116:51022) 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce93400640 1 -- 172.21.15.116:0/1521415473 learned_addr learned my addr 172.21.15.116:0/1521415473 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.761+0000 7fce98a00640 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce93400640 1 -- 172.21.15.116:0/1521415473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 msgr2=0x55bcbd0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce93400640 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce93400640 1 -- 172.21.15.116:0/1521415473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 msgr2=0x55bcbd0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce93400640 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce93400640 1 -- 172.21.15.116:0/1521415473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbd0d1970 con 0x55bcbd0cd400 2024-03-04T07:44:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce93400640 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0de200 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fce80004fd0 tx=0x7fce80009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ed31c79f0e535 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce7fe00640 1 -- 172.21.15.116:0/1521415473 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800035f0 con 0x55bcbd0cd400 2024-03-04T07:44:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce7fe00640 1 -- 172.21.15.116:0/1521415473 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce80032070 con 0x55bcbd0cd400 2024-03-04T07:44:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.762+0000 7fce7fe00640 1 -- 172.21.15.116:0/1521415473 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800222a0 con 0x55bcbd0cd400 2024-03-04T07:44:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 -- 172.21.15.116:0/1521415473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 msgr2=0x55bcbd0de200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0de200 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fce80004fd0 tx=0x7fce80009ee0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 -- 172.21.15.116:0/1521415473 shutdown_connections 2024-03-04T07:44:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0d2570 0x55bcbd0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cd400 0x55bcbd0de200 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 --2- 172.21.15.116:0/1521415473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0cca10 0x55bcbd0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 -- 172.21.15.116:0/1521415473 >> 172.21.15.116:0/1521415473 conn(0x55bcbd0c7f80 msgr2=0x55bcbd0ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 -- 172.21.15.116:0/1521415473 shutdown_connections 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 -- 172.21.15.116:0/1521415473 wait complete. 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.763+0000 7fce9d76b740 1 -- start start 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 1 Processor -- start 2024-03-04T07:44:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0db9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0cd400 0x55bcbd0dbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0d2570 0x55bcbd0dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941a9770 con 0x55bcbd0d2570 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941aaa90 con 0x55bcbd0cca10 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941a9dd0 con 0x55bcbd0cd400 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce9d76b740 10 client.0 _ll_register_callbacks cb 0x7fce8c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce93e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0db9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce93e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0db9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51036/0 (socket says 172.21.15.116:51036) 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce93e00640 1 -- 172.21.15.116:0/2485633372 learned_addr learned my addr 172.21.15.116:0/2485633372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce98a00640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0d2570 0x55bcbd0dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.764+0000 7fce93400640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0cd400 0x55bcbd0dbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce93e00640 1 -- 172.21.15.116:0/2485633372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0cd400 msgr2=0x55bcbd0dbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce93e00640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0cd400 0x55bcbd0dbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce93e00640 1 -- 172.21.15.116:0/2485633372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0d2570 msgr2=0x55bcbd0dceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce93e00640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0d2570 0x55bcbd0dceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce93e00640 1 -- 172.21.15.116:0/2485633372 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcbd0dd4a0 con 0x55bcbd0cca10 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce98a00640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0d2570 0x55bcbd0dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce93e00640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0db9d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fce84005ab0 tx=0x7fce8402cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2265928a6b59a078 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce7d600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce7d600640 1 -- 172.21.15.116:0/2485633372 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8404d280 con 0x55bcbd0cca10 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce7d600640 1 -- 172.21.15.116:0/2485633372 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fce84004040 con 0x55bcbd0cca10 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce80032070 con 0x55bcbd0cca10 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce7d600640 1 -- 172.21.15.116:0/2485633372 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce84004240 con 0x55bcbd0cca10 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce9d76b740 10 client.26309 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.765+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bcbd13e1c0 con 0x55bcbd0cca10 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce7d600640 1 -- 172.21.15.116:0/2485633372 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce80032070 con 0x55bcbd0cca10 2024-03-04T07:44:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce7d600640 1 -- 172.21.15.116:0/2485633372 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fce84004fd0 con 0x55bcbd0cca10 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce9d76b740 10 client.26309 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce9d76b740 10 client.26309 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce9d76b740 20 client.26309 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd13eaf0 0x55bcbd140f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce9d76b740 4 client.26309 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bcbd039c00 con 0x55bcbd13eaf0 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.766+0000 7fce93400640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd13eaf0 0x55bcbd140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.767+0000 7fce93400640 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd13eaf0 0x55bcbd140f10 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=411ed4591994361f server_cookie=93d581c97d2403ee in_seq=0 out_seq=0 2024-03-04T07:44:55.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.767+0000 7fce7d600640 10 client.26309 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.781+0000 7fce7d600640 1 -- 172.21.15.116:0/2485633372 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bcbd039c00 con 0x55bcbd13eaf0 2024-03-04T07:44:55.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.781+0000 7fce7d600640 10 client.26309 handle_command_reply: tid=0 2024-03-04T07:44:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.781+0000 7fce9d76b740 1 client.26309 shutdown 2024-03-04T07:44:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 msgr2=0x55bcbd0db9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0db9d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fce84005ab0 tx=0x7fce8402cca0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 shutdown_connections 2024-03-04T07:44:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bcbd13eaf0 0x55bcbd140f10 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bcbd0d2570 0x55bcbd0dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bcbd0cd400 0x55bcbd0dbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 --2- 172.21.15.116:0/2485633372 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bcbd0cca10 0x55bcbd0db9d0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 >> 172.21.15.116:0/2485633372 conn(0x55bcbd0c7f80 msgr2=0x55bcbd0cb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce7d600640 0 client.26309 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 shutdown_connections 2024-03-04T07:44:55.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 1 -- 172.21.15.116:0/2485633372 wait complete. 2024-03-04T07:44:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.782+0000 7fce9d76b740 20 client.26309 trim_cache size 0 max 16384 2024-03-04T07:44:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:44:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:44:55.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:44:55.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:44:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce780a1df0 msgr2=0x7fce780a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce780a1df0 0x7fce780a42b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fce8001e000 tx=0x7fce80003f30 comp rx=0 tx=0).stop 2024-03-04T07:44:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94107100 msgr2=0x7fce941a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94107100 0x7fce941a0e70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00eb10 tx=0x7fce8c00c2d0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 shutdown_connections 2024-03-04T07:44:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fce94107100 0x7fce941a0e70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce941047d0 0x7fce94199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce780a1df0 0x7fce780a42b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 --2- 172.21.15.116:0/173490596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fce94101ea0 0x7fce94199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 >> 172.21.15.116:0/173490596 conn(0x7fce940fb8b0 msgr2=0x7fce94101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 shutdown_connections 2024-03-04T07:44:55.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:44:55.783+0000 7fce99400640 1 -- 172.21.15.116:0/173490596 wait complete. 2024-03-04T07:44:55.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.938+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f1c02cff0 con 0x7f2f1c1020f0 2024-03-04T07:44:55.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.940+0000 7f2f1a000640 1 -- 172.21.15.116:0/1422226823 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f2f1009a980 con 0x7f2f1c1020f0 2024-03-04T07:44:55.940 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:44:55.940 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:44:55.940 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.942+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2f040a1e00 msgr2=0x7f2f040a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.942+0000 7f2f23e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2f040a1e00 0x7f2f040a42c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f08005ab0 tx=0x7f2f0802cf90 comp rx=0 tx=0).stop 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.942+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 msgr2=0x7f2f1c197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.942+0000 7f2f23e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 0x7f2f1c197910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000e9f0 tx=0x7f2f1000edf0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 shutdown_connections 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2f1c107350 0x7f2f1c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f1c104a20 0x7f2f1c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2f040a1e00 0x7f2f040a42c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 --2- 172.21.15.116:0/1422226823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2f1c1020f0 0x7f2f1c197910 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 >> 172.21.15.116:0/1422226823 conn(0x7f2f1c0fbb20 msgr2=0x7f2f1c102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 shutdown_connections 2024-03-04T07:44:55.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:55.943+0000 7f2f23e00640 1 -- 172.21.15.116:0/1422226823 wait complete. 2024-03-04T07:44:55.964 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:44:55.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:44:56.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 -- 172.21.15.116:0/577390375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 msgr2=0x7faae4110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 --2- 172.21.15.116:0/577390375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 0x7faae4110030 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faadc009be0 tx=0x7faadc02bef0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 -- 172.21.15.116:0/577390375 shutdown_connections 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 --2- 172.21.15.116:0/577390375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 0x7faae4110030 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 --2- 172.21.15.116:0/577390375 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 0x7faae4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 --2- 172.21.15.116:0/577390375 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4107bf0 0x7faae4108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 -- 172.21.15.116:0/577390375 >> 172.21.15.116:0/577390375 conn(0x7faae4076b90 msgr2=0x7faae4078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 -- 172.21.15.116:0/577390375 shutdown_connections 2024-03-04T07:44:56.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.178+0000 7faaea600640 1 -- 172.21.15.116:0/577390375 wait complete. 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.179+0000 7faaea600640 1 Processor -- start 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.179+0000 7faaea600640 1 -- start start 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.179+0000 7faaea600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4107bf0 0x7faae419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faaea600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 0x7faae419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faaea600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 0x7faae41a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faaea600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4113870 con 0x7faae41097a0 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faaea600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41136f0 con 0x7faae4108de0 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faaea600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41139f0 con 0x7faae4107bf0 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 0x7faae419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 0x7faae419e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59082/0 (socket says 172.21.15.116:59082) 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 -- 172.21.15.116:0/144676176 learned_addr learned my addr 172.21.15.116:0/144676176 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faae3e00640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 0x7faae41a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faae3400640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4107bf0 0x7faae419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 -- 172.21.15.116:0/144676176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4107bf0 msgr2=0x7faae419de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4107bf0 0x7faae419de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 -- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 msgr2=0x7faae41a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 0x7faae41a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 -- 172.21.15.116:0/144676176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae41a5ab0 con 0x7faae4108de0 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.180+0000 7faadbe00640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 0x7faae419e3d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faacc00e9c0 tx=0x7faacc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.181+0000 7faae0c00640 1 -- 172.21.15.116:0/144676176 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc00ccc0 con 0x7faae4108de0 2024-03-04T07:44:56.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.181+0000 7faae0c00640 1 -- 172.21.15.116:0/144676176 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faacc005500 con 0x7faae4108de0 2024-03-04T07:44:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.181+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae41a5da0 con 0x7faae4108de0 2024-03-04T07:44:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.181+0000 7faae0c00640 1 -- 172.21.15.116:0/144676176 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc010430 con 0x7faae4108de0 2024-03-04T07:44:56.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.181+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae41a6260 con 0x7faae4108de0 2024-03-04T07:44:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 Processor -- start 2024-03-04T07:44:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 -- start start 2024-03-04T07:44:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faae0c00640 1 -- 172.21.15.116:0/144676176 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7faacc0046c0 con 0x7faae4108de0 2024-03-04T07:44:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d845ca0 0x556b2d8460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d84b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d84dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2d7aad60 con 0x556b2d846690 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2d753cf0 con 0x556b2d84b800 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.184+0000 7faaed3bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2d7a6e00 con 0x556b2d845ca0 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d845ca0 0x556b2d8460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d84b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faadbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d84dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d845ca0 0x556b2d8460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:44666/0 (socket says 172.21.15.116:44666) 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 -- 172.21.15.116:0/1274616042 learned_addr learned my addr 172.21.15.116:0/1274616042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 -- 172.21.15.116:0/1274616042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 msgr2=0x556b2d84dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d84dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 -- 172.21.15.116:0/1274616042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 msgr2=0x556b2d84b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d84b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faae3e00640 1 -- 172.21.15.116:0/1274616042 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b2d4bf3f0 con 0x556b2d845ca0 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.185+0000 7faadbe00640 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d84dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.186+0000 7faae3e00640 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d845ca0 0x556b2d8460c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faadc009be0 tx=0x7faadc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b4c3d4ae7a6cd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.186+0000 7faae0c00640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faac40a1d80 0x7faac40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.186+0000 7faad8c00640 1 -- 172.21.15.116:0/1274616042 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc043030 con 0x556b2d845ca0 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.186+0000 7faad8c00640 1 -- 172.21.15.116:0/1274616042 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faadc033b90 con 0x556b2d845ca0 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.186+0000 7faae0c00640 1 -- 172.21.15.116:0/144676176 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7faacc0ce150 con 0x7faae4108de0 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.186+0000 7faad8c00640 1 -- 172.21.15.116:0/1274616042 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc02c890 con 0x556b2d845ca0 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.186+0000 7faae3400640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faac40a1d80 0x7faac40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 -- 172.21.15.116:0/1274616042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d845ca0 msgr2=0x556b2d8460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d845ca0 0x556b2d8460c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faadc009be0 tx=0x7faadc009310 comp rx=0 tx=0).stop 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faae3400640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faac40a1d80 0x7faac40a4240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x556b2d846a90 tx=0x7faad4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 -- 172.21.15.116:0/1274616042 shutdown_connections 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d84dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d84b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 --2- 172.21.15.116:0/1274616042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d845ca0 0x556b2d8460c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 -- 172.21.15.116:0/1274616042 >> 172.21.15.116:0/1274616042 conn(0x556b2d841210 msgr2=0x556b2d841610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 -- 172.21.15.116:0/1274616042 shutdown_connections 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.187+0000 7faaed3bd740 1 -- 172.21.15.116:0/1274616042 wait complete. 2024-03-04T07:44:56.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 -- start start 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 Processor -- start 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d8501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8506f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d857730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2d84e750 con 0x556b2d846690 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2d749220 con 0x556b2d845ca0 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b2d84e8d0 con 0x556b2d84b800 2024-03-04T07:44:56.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.188+0000 7faaed3bd740 10 client.0 _ll_register_callbacks cb 0x556b2d4bebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:56.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d8501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d8501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59092/0 (socket says 172.21.15.116:59092) 2024-03-04T07:44:56.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 -- 172.21.15.116:0/278960447 learned_addr learned my addr 172.21.15.116:0/278960447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:56.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faadbe00640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3e00640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d857730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 -- 172.21.15.116:0/278960447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 msgr2=0x556b2d857730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d857730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 -- 172.21.15.116:0/278960447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 msgr2=0x556b2d8506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 -- 172.21.15.116:0/278960447 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b2d5897d0 con 0x556b2d845ca0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faadbe00640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faae3400640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d8501b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faad4004270 tx=0x7faad4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd3f87c2b6d9dd46 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.189+0000 7faad1600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faad1600640 1 -- 172.21.15.116:0/278960447 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad40044e0 con 0x556b2d845ca0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faad1600640 1 -- 172.21.15.116:0/278960447 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7faad4003d00 con 0x556b2d845ca0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faadc033b90 con 0x556b2d845ca0 2024-03-04T07:44:56.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faad1600640 1 -- 172.21.15.116:0/278960447 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad402a420 con 0x556b2d845ca0 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faaed3bd740 10 client.26321 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556b2d85ab60 con 0x556b2d845ca0 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faad1600640 1 -- 172.21.15.116:0/278960447 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faadc033b90 con 0x556b2d845ca0 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.190+0000 7faad1600640 1 -- 172.21.15.116:0/278960447 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7faad4020640 con 0x556b2d845ca0 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faaed3bd740 10 client.26321 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faaed3bd740 10 client.26321 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faaed3bd740 20 client.26321 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faaed3bd740 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d85c230 0x556b2d85e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faaed3bd740 4 client.26321 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b2d7b82a0 con 0x556b2d85c230 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faadbe00640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d85c230 0x556b2d85e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faad1600640 10 client.26321 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.191+0000 7faadbe00640 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d85c230 0x556b2d85e6b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c09d33021e93bef server_cookie=dc7006fa0bf764de in_seq=0 out_seq=0 2024-03-04T07:44:56.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faad1600640 1 -- 172.21.15.116:0/278960447 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556b2d7b82a0 con 0x556b2d85c230 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faad1600640 10 client.26321 handle_command_reply: tid=0 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 client.26321 shutdown 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 msgr2=0x556b2d8501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d8501b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faad4004270 tx=0x7faad4004730 comp rx=0 tx=0).stop 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 shutdown_connections 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d85c230 0x556b2d85e6b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d857730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faad1600640 0 client.26321 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 --2- 172.21.15.116:0/278960447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d8501b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.192+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 >> 172.21.15.116:0/278960447 conn(0x556b2d841210 msgr2=0x556b2d8404b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.193+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 shutdown_connections 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.193+0000 7faaed3bd740 1 -- 172.21.15.116:0/278960447 wait complete. 2024-03-04T07:44:56.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.193+0000 7faaed3bd740 20 client.26321 trim_cache size 0 max 16384 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.206+0000 7faaed3bd740 1 Processor -- start 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.206+0000 7faaed3bd740 1 -- start start 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d850650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d844b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faaed3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41acf80 con 0x556b2d846690 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faaed3bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41af560 con 0x556b2d845ca0 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faaed3bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41ad580 con 0x556b2d84b800 2024-03-04T07:44:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d850650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d850650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59098/0 (socket says 172.21.15.116:59098) 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 -- 172.21.15.116:0/3606699161 learned_addr learned my addr 172.21.15.116:0/3606699161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3e00640 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d844b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faadbe00640 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 -- 172.21.15.116:0/3606699161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 msgr2=0x556b2d844b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d844b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 -- 172.21.15.116:0/3606699161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 msgr2=0x556b2d8576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3400640 1 -- 172.21.15.116:0/3606699161 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b2d84f240 con 0x556b2d845ca0 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faae3e00640 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d844b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.207+0000 7faadbe00640 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.208+0000 7faae3400640 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d850650 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faad4013010 tx=0x7faad40039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e151c153f7ae84a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.208+0000 7faad3e00640 1 -- 172.21.15.116:0/3606699161 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4003d00 con 0x556b2d845ca0 2024-03-04T07:44:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.208+0000 7faad3e00640 1 -- 172.21.15.116:0/3606699161 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faad4025030 con 0x556b2d845ca0 2024-03-04T07:44:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.208+0000 7faad3e00640 1 -- 172.21.15.116:0/3606699161 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad402a9b0 con 0x556b2d845ca0 2024-03-04T07:44:56.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 -- 172.21.15.116:0/3606699161 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 msgr2=0x556b2d850650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d850650 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faad4013010 tx=0x7faad40039c0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 -- 172.21.15.116:0/3606699161 shutdown_connections 2024-03-04T07:44:56.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d84b800 0x556b2d844b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d846690 0x556b2d8576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 --2- 172.21.15.116:0/3606699161 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d845ca0 0x556b2d850650 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 -- 172.21.15.116:0/3606699161 >> 172.21.15.116:0/3606699161 conn(0x556b2d841210 msgr2=0x556b2d844470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 -- 172.21.15.116:0/3606699161 shutdown_connections 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 -- 172.21.15.116:0/3606699161 wait complete. 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 -- start start 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.209+0000 7faaed3bd740 1 Processor -- start 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d845ca0 0x556b2d8554f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d846690 0x556b2d855a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faaed3bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d856960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faaed3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41adbe0 con 0x556b2d845ca0 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faaed3bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41aef00 con 0x556b2d84b800 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faaed3bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae41ae240 con 0x556b2d846690 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faaed3bd740 10 client.0 _ll_register_callbacks cb 0x7faad4009960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d845ca0 0x556b2d8554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d856960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d845ca0 0x556b2d8554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54644/0 (socket says 172.21.15.116:54644) 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 -- 172.21.15.116:0/588155377 learned_addr learned my addr 172.21.15.116:0/588155377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faadbe00640 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d846690 0x556b2d855a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 -- 172.21.15.116:0/588155377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d846690 msgr2=0x556b2d855a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d846690 0x556b2d855a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 -- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 msgr2=0x556b2d856960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d856960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3400640 1 -- 172.21.15.116:0/588155377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b2d856f50 con 0x556b2d845ca0 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.210+0000 7faae3e00640 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d856960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faae3400640 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d845ca0 0x556b2d8554f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7faad4012d40 tx=0x7faad4009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e95c15ef93ff0ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faad1600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faad1600640 1 -- 172.21.15.116:0/588155377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad4042020 con 0x556b2d845ca0 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faad4025030 con 0x556b2d845ca0 2024-03-04T07:44:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faad1600640 1 -- 172.21.15.116:0/588155377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7faad4005050 con 0x556b2d845ca0 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faad1600640 1 -- 172.21.15.116:0/588155377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad4003dc0 con 0x556b2d845ca0 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faaed3bd740 10 client.16200 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.211+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556b2d8b7fe0 con 0x556b2d845ca0 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faad1600640 1 -- 172.21.15.116:0/588155377 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faad4025030 con 0x556b2d845ca0 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faad1600640 1 -- 172.21.15.116:0/588155377 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7faad40045e0 con 0x556b2d845ca0 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faaed3bd740 10 client.16200 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faaed3bd740 10 client.16200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faaed3bd740 20 client.16200 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faaed3bd740 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d8b87f0 0x556b2d8bac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faaed3bd740 4 client.16200 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556b2d7b25f0 con 0x556b2d8b87f0 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.212+0000 7faadbe00640 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d8b87f0 0x556b2d8bac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:56.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faadbe00640 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d8b87f0 0x556b2d8bac10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e0a094251a62260 server_cookie=6682993a762aebce in_seq=0 out_seq=0 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faad1600640 10 client.16200 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faad1600640 1 -- 172.21.15.116:0/588155377 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556b2d7b25f0 con 0x556b2d8b87f0 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faad1600640 10 client.16200 handle_command_reply: tid=0 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 client.16200 shutdown 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d845ca0 msgr2=0x556b2d8554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d845ca0 0x556b2d8554f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7faad4012d40 tx=0x7faad4009240 comp rx=0 tx=0).stop 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 shutdown_connections 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b2d8b87f0 0x556b2d8bac10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b2d84b800 0x556b2d856960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b2d846690 0x556b2d855a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 --2- 172.21.15.116:0/588155377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b2d845ca0 0x556b2d8554f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faad1600640 0 client.16200 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 >> 172.21.15.116:0/588155377 conn(0x556b2d841210 msgr2=0x556b2d850440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 shutdown_connections 2024-03-04T07:44:56.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 1 -- 172.21.15.116:0/588155377 wait complete. 2024-03-04T07:44:56.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.213+0000 7faaed3bd740 20 client.16200 trim_cache size 0 max 16384 2024-03-04T07:44:56.219 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faac40a1d80 msgr2=0x7faac40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faac40a1d80 0x7faac40a4240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x556b2d846a90 tx=0x7faad4005f90 comp rx=0 tx=0).stop 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 msgr2=0x7faae419e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 0x7faae419e3d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faacc00e9c0 tx=0x7faacc00ee80 comp rx=0 tx=0).stop 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 shutdown_connections 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae41097a0 0x7faae41a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faae4108de0 0x7faae419e3d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faac40a1d80 0x7faac40a4240 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 --2- 172.21.15.116:0/144676176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faae4107bf0 0x7faae419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 >> 172.21.15.116:0/144676176 conn(0x7faae4076b90 msgr2=0x7faae410c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 shutdown_connections 2024-03-04T07:44:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:56.214+0000 7faaea600640 1 -- 172.21.15.116:0/144676176 wait complete. 2024-03-04T07:44:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:56 smithi116 ceph-mon[32150]: pgmap v526: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:57.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:56 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1422226823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:57.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:56 smithi152 ceph-mon[37154]: pgmap v526: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:57.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:56 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1422226823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:57.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:56 smithi133 ceph-mon[36505]: pgmap v526: 97 pgs: 97 active+clean; 2.9 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:44:57.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:56 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1422226823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:44:57.233 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:44:57.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:44:57.234 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.459+0000 7f0964600640 1 -- 172.21.15.116:0/3201887107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c104a00 msgr2=0x7f095c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.459+0000 7f0964600640 1 --2- 172.21.15.116:0/3201887107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c104a00 0x7f095c104e80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0948009900 tx=0x7f094802bc40 comp rx=0 tx=0).stop 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.460+0000 7f0964600640 1 -- 172.21.15.116:0/3201887107 shutdown_connections 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.460+0000 7f0964600640 1 --2- 172.21.15.116:0/3201887107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f095c1053c0 0x7f095c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.460+0000 7f0964600640 1 --2- 172.21.15.116:0/3201887107 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c104a00 0x7f095c104e80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.460+0000 7f0964600640 1 --2- 172.21.15.116:0/3201887107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c103810 0x7f095c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.460+0000 7f0964600640 1 -- 172.21.15.116:0/3201887107 >> 172.21.15.116:0/3201887107 conn(0x7f095c0fed80 msgr2=0x7f095c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.460+0000 7f0964600640 1 -- 172.21.15.116:0/3201887107 shutdown_connections 2024-03-04T07:44:57.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.460+0000 7f0964600640 1 -- 172.21.15.116:0/3201887107 wait complete. 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 Processor -- start 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 -- start start 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c103810 0x7f095c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f095c104a00 0x7f095c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c1053c0 0x7f095c1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c10f4c0 con 0x7f095c1053c0 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c10f340 con 0x7f095c103810 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0964600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c10f640 con 0x7f095c104a00 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0961c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c103810 0x7f095c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0961c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c103810 0x7f095c199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59136/0 (socket says 172.21.15.116:59136) 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0961c00640 1 -- 172.21.15.116:0/154651582 learned_addr learned my addr 172.21.15.116:0/154651582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0962600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c1053c0 0x7f095c1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.461+0000 7f0961200640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f095c104a00 0x7f095c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0961c00640 1 -- 172.21.15.116:0/154651582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f095c104a00 msgr2=0x7f095c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0961c00640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f095c104a00 0x7f095c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0961c00640 1 -- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c1053c0 msgr2=0x7f095c1a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0961c00640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c1053c0 0x7f095c1a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0961c00640 1 -- 172.21.15.116:0/154651582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c1a1730 con 0x7f095c103810 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0962600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c1053c0 0x7f095c1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0961c00640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c103810 0x7f095c199b10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f094c00e9c0 tx=0x7f094c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f095a000640 1 -- 172.21.15.116:0/154651582 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c00cd80 con 0x7f095c103810 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f095a000640 1 -- 172.21.15.116:0/154651582 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f094c0035f0 con 0x7f095c103810 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f095a000640 1 -- 172.21.15.116:0/154651582 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c010690 con 0x7f095c103810 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.462+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c1a1a20 con 0x7f095c103810 2024-03-04T07:44:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.463+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c1a1ef0 con 0x7f095c103810 2024-03-04T07:44:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f095a000640 1 -- 172.21.15.116:0/154651582 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f094c0037a0 con 0x7f095c103810 2024-03-04T07:44:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 Processor -- start 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 -- start start 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e98a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e98f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e9920c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040e89de20 con 0x56040e98fc30 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040e8efa20 con 0x56040e98a0d0 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f096722c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040e898b00 con 0x56040e98aac0 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f0962600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e98a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f0962600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e98a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59148/0 (socket says 172.21.15.116:59148) 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f0962600640 1 -- 172.21.15.116:0/3772610478 learned_addr learned my addr 172.21.15.116:0/3772610478 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f0961200640 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e9920c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.466+0000 7f0961c00640 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e98f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0962600640 1 -- 172.21.15.116:0/3772610478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 msgr2=0x56040e98f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0962600640 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e98f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0962600640 1 -- 172.21.15.116:0/3772610478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 msgr2=0x56040e9920c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0962600640 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e9920c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0962600640 1 -- 172.21.15.116:0/3772610478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56040e8ade10 con 0x56040e98a0d0 2024-03-04T07:44:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0961c00640 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e98f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0962600640 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e98a4f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0954010750 tx=0x7f0954010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7fcb1a83df8023f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0952a00640 1 -- 172.21.15.116:0/3772610478 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095400ead0 con 0x56040e98a0d0 2024-03-04T07:44:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0952a00640 1 -- 172.21.15.116:0/3772610478 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0954010e60 con 0x56040e98a0d0 2024-03-04T07:44:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.467+0000 7f0952a00640 1 -- 172.21.15.116:0/3772610478 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954011640 con 0x56040e98a0d0 2024-03-04T07:44:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f095a000640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1e00 0x7f09400a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f095a000640 1 -- 172.21.15.116:0/154651582 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f094c0ce0f0 con 0x7f095c103810 2024-03-04T07:44:57.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f0961c00640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1e00 0x7f09400a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 -- 172.21.15.116:0/3772610478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 msgr2=0x56040e98a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e98a4f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0954010750 tx=0x7f0954010c10 comp rx=0 tx=0).stop 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 -- 172.21.15.116:0/3772610478 shutdown_connections 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e9920c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e98f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 --2- 172.21.15.116:0/3772610478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e98a4f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 -- 172.21.15.116:0/3772610478 >> 172.21.15.116:0/3772610478 conn(0x56040e985640 msgr2=0x56040e985a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 -- 172.21.15.116:0/3772610478 shutdown_connections 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.468+0000 7f096722c740 1 -- 172.21.15.116:0/3772610478 wait complete. 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.469+0000 7f0961c00640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1e00 0x7f09400a42c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f094c097240 tx=0x7f094c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.469+0000 7f096722c740 1 -- start start 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.469+0000 7f096722c740 1 Processor -- start 2024-03-04T07:44:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.469+0000 7f096722c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98a0d0 0x56040e994470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.469+0000 7f096722c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e9949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f096722c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f096722c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040e8abf40 con 0x56040e98fc30 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f096722c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040e8ebac0 con 0x56040e98aac0 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f096722c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040e992be0 con 0x56040e98a0d0 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f096722c740 10 client.0 _ll_register_callbacks cb 0x56040e8eaaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e9949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e9949b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59164/0 (socket says 172.21.15.116:59164) 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 -- 172.21.15.116:0/2554421270 learned_addr learned my addr 172.21.15.116:0/2554421270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0962600640 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0962600640 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98a0d0 0x56040e994470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 -- 172.21.15.116:0/2554421270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98a0d0 msgr2=0x56040e994470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98a0d0 0x56040e994470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 -- 172.21.15.116:0/2554421270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 msgr2=0x56040e99b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.470+0000 7f0961200640 1 -- 172.21.15.116:0/2554421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56040e6cc7e0 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f0961200640 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e9949b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f094802c7c0 tx=0x7f0948004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca6155074ea8891a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f0947400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f0947400640 1 -- 172.21.15.116:0/2554421270 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948005d50 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f0947400640 1 -- 172.21.15.116:0/2554421270 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f09480048c0 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0954010e60 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f0947400640 1 -- 172.21.15.116:0/2554421270 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0948004ac0 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f096722c740 10 client.26339 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56040e72cf70 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.471+0000 7f0947400640 1 -- 172.21.15.116:0/2554421270 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0954010e60 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f0947400640 1 -- 172.21.15.116:0/2554421270 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0948046070 con 0x56040e98aac0 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f096722c740 10 client.26339 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:57.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f096722c740 10 client.26339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f096722c740 20 client.26339 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f096722c740 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9a02d0 0x56040e9a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f096722c740 4 client.26339 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56040e8fcf60 con 0x56040e9a02d0 2024-03-04T07:44:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.472+0000 7f0962600640 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9a02d0 0x56040e9a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.473+0000 7f0962600640 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9a02d0 0x56040e9a26f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8014666a7650278f server_cookie=5bb8ecc4a49ea4a1 in_seq=0 out_seq=0 2024-03-04T07:44:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.473+0000 7f0947400640 10 client.26339 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f0947400640 1 -- 172.21.15.116:0/2554421270 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56040e8fcf60 con 0x56040e9a02d0 2024-03-04T07:44:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f0947400640 10 client.26339 handle_command_reply: tid=0 2024-03-04T07:44:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 client.26339 shutdown 2024-03-04T07:44:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 msgr2=0x56040e9949b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e9949b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f094802c7c0 tx=0x7f0948004250 comp rx=0 tx=0).stop 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 shutdown_connections 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9a02d0 0x56040e9a26f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f0947400640 0 client.26339 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e9949b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 --2- 172.21.15.116:0/2554421270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98a0d0 0x56040e994470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 >> 172.21.15.116:0/2554421270 conn(0x56040e985640 msgr2=0x56040e986890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 shutdown_connections 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 1 -- 172.21.15.116:0/2554421270 wait complete. 2024-03-04T07:44:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.474+0000 7f096722c740 20 client.26339 trim_cache size 0 max 16384 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 Processor -- start 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 -- start start 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98a0d0 0x56040e9841d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e99b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98fc30 0x56040e984710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1a8cf0 con 0x56040e98a0d0 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1ab2d0 con 0x56040e98aac0 2024-03-04T07:44:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.494+0000 7f096722c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1a92f0 con 0x56040e98fc30 2024-03-04T07:44:57.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e99b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0962600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98a0d0 0x56040e9841d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e99b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59176/0 (socket says 172.21.15.116:59176) 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 -- 172.21.15.116:0/3019611411 learned_addr learned my addr 172.21.15.116:0/3019611411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0962600640 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98fc30 0x56040e984710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 -- 172.21.15.116:0/3019611411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98fc30 msgr2=0x56040e984710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98fc30 0x56040e984710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 -- 172.21.15.116:0/3019611411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98a0d0 msgr2=0x56040e9841d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98a0d0 0x56040e9841d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 -- 172.21.15.116:0/3019611411 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56040e5c4170 con 0x56040e98aac0 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0962600640 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98fc30 0x56040e984710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0961200640 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e99b9b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0948006010 tx=0x7f0948003ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b174952af40ae2a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.495+0000 7f0952000640 1 -- 172.21.15.116:0/3019611411 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948005d50 con 0x56040e98aac0 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f0952000640 1 -- 172.21.15.116:0/3019611411 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0948057020 con 0x56040e98aac0 2024-03-04T07:44:57.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f0952000640 1 -- 172.21.15.116:0/3019611411 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09480046c0 con 0x56040e98aac0 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 -- 172.21.15.116:0/3019611411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 msgr2=0x56040e99b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e99b9b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0948006010 tx=0x7f0948003ad0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 -- 172.21.15.116:0/3019611411 shutdown_connections 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98fc30 0x56040e984710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98aac0 0x56040e99b9b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 --2- 172.21.15.116:0/3019611411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98a0d0 0x56040e9841d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 -- 172.21.15.116:0/3019611411 >> 172.21.15.116:0/3019611411 conn(0x56040e985640 msgr2=0x56040e994470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 -- 172.21.15.116:0/3019611411 shutdown_connections 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.496+0000 7f096722c740 1 -- 172.21.15.116:0/3019611411 wait complete. 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 -- start start 2024-03-04T07:44:57.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 Processor -- start 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e99b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e999d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1a9950 con 0x56040e98fc30 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1aac70 con 0x56040e98a0d0 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.497+0000 7f096722c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1a9fb0 con 0x56040e98aac0 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f096722c740 10 client.0 _ll_register_callbacks cb 0x7f0948009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e999d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e999d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:44742/0 (socket says 172.21.15.116:44742) 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 -- 172.21.15.116:0/2229068629 learned_addr learned my addr 172.21.15.116:0/2229068629 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0962600640 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e99b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0962600640 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 -- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 msgr2=0x56040e99b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e99b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 -- 172.21.15.116:0/2229068629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 msgr2=0x56040e99a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 -- 172.21.15.116:0/2229068629 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56040e99a8c0 con 0x56040e98aac0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0961200640 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e999d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f094802c180 tx=0x7f094802cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e06d8844ec6829a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.498+0000 7f0947400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f0947400640 1 -- 172.21.15.116:0/2229068629 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09480046c0 con 0x56040e98aac0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0948057020 con 0x56040e98aac0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f0947400640 1 -- 172.21.15.116:0/2229068629 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0948004860 con 0x56040e98aac0 2024-03-04T07:44:57.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f0947400640 1 -- 172.21.15.116:0/2229068629 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0948004ed0 con 0x56040e98aac0 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 10 client.26275 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56040e9fc950 con 0x56040e98aac0 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f0947400640 1 -- 172.21.15.116:0/2229068629 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0948057020 con 0x56040e98aac0 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f0947400640 1 -- 172.21.15.116:0/2229068629 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0948046040 con 0x56040e98aac0 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 10 client.26275 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 10 client.26275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 20 client.26275 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9fd1b0 0x56040e9ff5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 4 client.26275 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.499+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56040e8f72b0 con 0x56040e9fd1b0 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.500+0000 7f0962600640 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9fd1b0 0x56040e9ff5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.500+0000 7f0962600640 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9fd1b0 0x56040e9ff5d0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94199a9669841676 server_cookie=e665b87c26d2864f in_seq=0 out_seq=0 2024-03-04T07:44:57.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.500+0000 7f0947400640 10 client.26275 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f0947400640 1 -- 172.21.15.116:0/2229068629 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56040e8f72b0 con 0x56040e9fd1b0 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f0947400640 10 client.26275 handle_command_reply: tid=0 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 client.26275 shutdown 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 msgr2=0x56040e999d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e999d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f094802c180 tx=0x7f094802cda0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 shutdown_connections 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56040e9fd1b0 0x56040e9ff5d0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56040e98fc30 0x56040e99a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56040e98aac0 0x56040e999d90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 --2- 172.21.15.116:0/2229068629 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56040e98a0d0 0x56040e99b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 >> 172.21.15.116:0/2229068629 conn(0x56040e985640 msgr2=0x56040e9841d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f0947400640 0 client.26275 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.501+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 shutdown_connections 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f096722c740 1 -- 172.21.15.116:0/2229068629 wait complete. 2024-03-04T07:44:57.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f096722c740 20 client.26275 trim_cache size 0 max 16384 2024-03-04T07:44:57.506 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:57.506 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:44:57.506 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "e010ba7d-7cde-4573-95bc-45aaff9709a0", 2024-03-04T07:44:57.506 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:44:57.506 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1e00 msgr2=0x7f09400a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1e00 0x7f09400a42c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f094c097240 tx=0x7f094c000bc0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c103810 msgr2=0x7f095c199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c103810 0x7f095c199b10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f094c00e9c0 tx=0x7f094c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 shutdown_connections 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f095c1053c0 0x7f095c1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09400a1e00 0x7f09400a42c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f095c104a00 0x7f095c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 --2- 172.21.15.116:0/154651582 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f095c103810 0x7f095c199b10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.502+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 >> 172.21.15.116:0/154651582 conn(0x7f095c0fed80 msgr2=0x7f095c107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.503+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 shutdown_connections 2024-03-04T07:44:57.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.503+0000 7f0964600640 1 -- 172.21.15.116:0/154651582 wait complete. 2024-03-04T07:44:57.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:44:57.520 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 -- 172.21.15.116:0/3410918276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081026a0 msgr2=0x7f1d08102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 --2- 172.21.15.116:0/3410918276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081026a0 0x7f1d08102b20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009900 tx=0x7f1cf802bc50 comp rx=0 tx=0).stop 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 -- 172.21.15.116:0/3410918276 shutdown_connections 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 --2- 172.21.15.116:0/3410918276 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d08103060 0x7f1d081098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 --2- 172.21.15.116:0/3410918276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081026a0 0x7f1d08102b20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 --2- 172.21.15.116:0/3410918276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d081014b0 0x7f1d081018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 -- 172.21.15.116:0/3410918276 >> 172.21.15.116:0/3410918276 conn(0x7f1d080fca40 msgr2=0x7f1d080fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 -- 172.21.15.116:0/3410918276 shutdown_connections 2024-03-04T07:44:57.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.724+0000 7f1d0f200640 1 -- 172.21.15.116:0/3410918276 wait complete. 2024-03-04T07:44:57.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 Processor -- start 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 -- start start 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081014b0 0x7f1d08075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d081026a0 0x7f1d080739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d08103060 0x7f1d08073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0810d190 con 0x7f1d081014b0 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0810d010 con 0x7f1d081026a0 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0f200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0810d310 con 0x7f1d08103060 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081014b0 0x7f1d08075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d081026a0 0x7f1d080739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081014b0 0x7f1d08075380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54710/0 (socket says 172.21.15.116:54710) 2024-03-04T07:44:57.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0de00640 1 -- 172.21.15.116:0/1778593457 learned_addr learned my addr 172.21.15.116:0/1778593457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.725+0000 7f1d0e800640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d08103060 0x7f1d08073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0de00640 1 -- 172.21.15.116:0/1778593457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d08103060 msgr2=0x7f1d08073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d08103060 0x7f1d08073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0de00640 1 -- 172.21.15.116:0/1778593457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d081026a0 msgr2=0x7f1d080739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d081026a0 0x7f1d080739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0de00640 1 -- 172.21.15.116:0/1778593457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08074790 con 0x7f1d081014b0 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0d400640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d081026a0 0x7f1d080739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081014b0 0x7f1d08075380 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00d950 tx=0x7f1cfc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d02a00640 1 -- 172.21.15.116:0/1778593457 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc017070 con 0x7f1d081014b0 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d02a00640 1 -- 172.21.15.116:0/1778593457 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1cfc009c30 con 0x7f1d081014b0 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d02a00640 1 -- 172.21.15.116:0/1778593457 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc00f600 con 0x7f1d081014b0 2024-03-04T07:44:57.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.726+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d081a18b0 con 0x7f1d081014b0 2024-03-04T07:44:57.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.727+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d081a1d80 con 0x7f1d081014b0 2024-03-04T07:44:57.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d02a00640 1 -- 172.21.15.116:0/1778593457 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1cfc01a070 con 0x7f1d081014b0 2024-03-04T07:44:57.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 Processor -- start 2024-03-04T07:44:57.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 -- start start 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645380c2090 con 0x564538194d10 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645380f9d30 con 0x56453819a870 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.730+0000 7f1d13441740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645380a2a30 con 0x564538195700 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59214/0 (socket says 172.21.15.116:59214) 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 -- 172.21.15.116:0/264002294 learned_addr learned my addr 172.21.15.116:0/264002294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0e800640 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0de00640 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 -- 172.21.15.116:0/264002294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 msgr2=0x56453819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 -- 172.21.15.116:0/264002294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 msgr2=0x564538195130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 -- 172.21.15.116:0/264002294 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564537ed87d0 con 0x56453819a870 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d0d400640 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819cd00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1cf802c900 tx=0x7f1cf80095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e46dc96135cc957c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1d02a00640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cf00a1e00 0x7f1cf00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.731+0000 7f1cf7000640 1 -- 172.21.15.116:0/264002294 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8042070 con 0x56453819a870 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1cf7000640 1 -- 172.21.15.116:0/264002294 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1cf802cd50 con 0x56453819a870 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1cf7000640 1 -- 172.21.15.116:0/264002294 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8005230 con 0x56453819a870 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d0e800640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cf00a1e00 0x7f1cf00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d02a00640 1 -- 172.21.15.116:0/1778593457 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1cfc0cd730 con 0x7f1d081014b0 2024-03-04T07:44:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 -- 172.21.15.116:0/264002294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 msgr2=0x56453819cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819cd00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1cf802c900 tx=0x7f1cf80095e0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d0e800640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cf00a1e00 0x7f1cf00a42c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x564538195f60 tx=0x7f1d04009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 -- 172.21.15.116:0/264002294 shutdown_connections 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819cd00 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 --2- 172.21.15.116:0/264002294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 -- 172.21.15.116:0/264002294 >> 172.21.15.116:0/264002294 conn(0x564538190280 msgr2=0x564538190680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.732+0000 7f1d13441740 1 -- 172.21.15.116:0/264002294 shutdown_connections 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.733+0000 7f1d13441740 1 -- 172.21.15.116:0/264002294 wait complete. 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.733+0000 7f1d13441740 1 -- start start 2024-03-04T07:44:57.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.733+0000 7f1d13441740 1 Processor -- start 2024-03-04T07:44:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d13441740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538194d10 0x56453819f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d13441740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d13441740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56453819a870 0x5645381a6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d13441740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645380980c0 con 0x56453819a870 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d13441740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645380f5dd0 con 0x564538194d10 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d13441740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56453819d7c0 con 0x564538195700 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d13441740 10 client.0 _ll_register_callbacks cb 0x5645380b5860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538194d10 0x56453819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538194d10 0x56453819f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59218/0 (socket says 172.21.15.116:59218) 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 -- 172.21.15.116:0/1278243908 learned_addr learned my addr 172.21.15.116:0/1278243908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0e800640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56453819a870 0x5645381a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0d400640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 -- 172.21.15.116:0/1278243908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 msgr2=0x56453819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 -- 172.21.15.116:0/1278243908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56453819a870 msgr2=0x5645381a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56453819a870 0x5645381a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0de00640 1 -- 172.21.15.116:0/1278243908 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564537e0e3f0 con 0x564538194d10 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.734+0000 7f1d0d400640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538194d10 0x56453819f160 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009610 tx=0x7f1cfc09a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ff2a42d26288924 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1ceb400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1ceb400640 1 -- 172.21.15.116:0/1278243908 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc021050 con 0x564538194d10 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1ceb400640 1 -- 172.21.15.116:0/1278243908 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1cfc0b8b90 con 0x564538194d10 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1cf802cd50 con 0x564538194d10 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1ceb400640 1 -- 172.21.15.116:0/1278243908 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cfc0d30b0 con 0x564538194d10 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 10 client.26357 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:57.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564537dce0f0 con 0x564538194d10 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1ceb400640 1 -- 172.21.15.116:0/1278243908 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1cf802cd50 con 0x564538194d10 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1ceb400640 1 -- 172.21.15.116:0/1278243908 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1cfc09e140 con 0x564538194d10 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 10 client.26357 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 10 client.26357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 20 client.26357 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645381acf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 4 client.26357 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.735+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564538107270 con 0x5645381aaa70 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.736+0000 7f1d0d400640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645381acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.736+0000 7f1d0d400640 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645381acf30 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3940365be861889e server_cookie=86d9ceb06fbc6663 in_seq=0 out_seq=0 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.736+0000 7f1ceb400640 10 client.26357 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1ceb400640 1 -- 172.21.15.116:0/1278243908 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564538107270 con 0x5645381aaa70 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1ceb400640 10 client.26357 handle_command_reply: tid=0 2024-03-04T07:44:57.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 client.26357 shutdown 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538194d10 msgr2=0x56453819f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538194d10 0x56453819f160 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009610 tx=0x7f1cfc09a0d0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 shutdown_connections 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645381acf30 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56453819a870 0x5645381a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x56453819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1ceb400640 0 client.26357 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 --2- 172.21.15.116:0/1278243908 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538194d10 0x56453819f160 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 >> 172.21.15.116:0/1278243908 conn(0x564538190280 msgr2=0x56453818f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 shutdown_connections 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 1 -- 172.21.15.116:0/1278243908 wait complete. 2024-03-04T07:44:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.737+0000 7f1d13441740 20 client.26357 trim_cache size 0 max 16384 2024-03-04T07:44:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 Processor -- start 2024-03-04T07:44:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 -- start start 2024-03-04T07:44:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x5645381a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081a89c0 con 0x564538194d10 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081aafa0 con 0x56453819a870 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d13441740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081a8fc0 con 0x564538195700 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d0e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d0e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59228/0 (socket says 172.21.15.116:59228) 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.755+0000 7f1d0e800640 1 -- 172.21.15.116:0/1208763003 learned_addr learned my addr 172.21.15.116:0/1208763003 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0d400640 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x5645381a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0e800640 1 -- 172.21.15.116:0/1208763003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 msgr2=0x5645381a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0e800640 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x5645381a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0e800640 1 -- 172.21.15.116:0/1208763003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 msgr2=0x564538195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0e800640 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0e800640 1 -- 172.21.15.116:0/1208763003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56453819e270 con 0x56453819a870 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0de00640 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.756+0000 7f1d0e800640 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819f480 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1d04003dd0 tx=0x7f1d04009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f3521147560a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1cf6600640 1 -- 172.21.15.116:0/1208763003 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04004540 con 0x56453819a870 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1cf6600640 1 -- 172.21.15.116:0/1208763003 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d040046e0 con 0x56453819a870 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1cf6600640 1 -- 172.21.15.116:0/1208763003 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0401d540 con 0x56453819a870 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1d13441740 1 -- 172.21.15.116:0/1208763003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 msgr2=0x56453819f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1d13441740 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819f480 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1d04003dd0 tx=0x7f1d04009f90 comp rx=0 tx=0).stop 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1d13441740 1 -- 172.21.15.116:0/1208763003 shutdown_connections 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1d13441740 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56453819a870 0x56453819f480 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1d13441740 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564538195700 0x5645381a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1d13441740 1 --2- 172.21.15.116:0/1208763003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x564538195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.757+0000 7f1d13441740 1 -- 172.21.15.116:0/1208763003 >> 172.21.15.116:0/1208763003 conn(0x564538190280 msgr2=0x56453819ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.758+0000 7f1d13441740 1 -- 172.21.15.116:0/1208763003 shutdown_connections 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.758+0000 7f1d13441740 1 -- 172.21.15.116:0/1208763003 wait complete. 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.758+0000 7f1d13441740 1 -- start start 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.758+0000 7f1d13441740 1 Processor -- start 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.758+0000 7f1d13441740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x5645381a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d13441740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538195700 0x5645381a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d13441740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56453819a870 0x5645381a5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d13441740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081a9620 con 0x564538194d10 2024-03-04T07:44:57.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d13441740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081aa940 con 0x564538195700 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d13441740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081a9c80 con 0x56453819a870 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d13441740 10 client.0 _ll_register_callbacks cb 0x7f1cfc096770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538195700 0x5645381a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538195700 0x5645381a42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59234/0 (socket says 172.21.15.116:59234) 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 -- 172.21.15.116:0/795409968 learned_addr learned my addr 172.21.15.116:0/795409968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x5645381a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0e800640 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56453819a870 0x5645381a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 -- 172.21.15.116:0/795409968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56453819a870 msgr2=0x5645381a5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56453819a870 0x5645381a5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 -- 172.21.15.116:0/795409968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 msgr2=0x5645381a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x5645381a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 -- 172.21.15.116:0/795409968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645381a5810 con 0x564538195700 2024-03-04T07:44:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0e800640 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56453819a870 0x5645381a5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1d0d400640 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538195700 0x5645381a42d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1cf802f020 tx=0x7f1cf802bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e15928c9c3c9f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.759+0000 7f1ceb400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1ceb400640 1 -- 172.21.15.116:0/795409968 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8054020 con 0x564538195700 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1ceb400640 1 -- 172.21.15.116:0/795409968 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1cf80050f0 con 0x564538195700 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d040046e0 con 0x564538195700 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1ceb400640 1 -- 172.21.15.116:0/795409968 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cf8042050 con 0x564538195700 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1d13441740 10 client.26363 fetch_fsmap learned FSMap version 19 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564538206420 con 0x564538195700 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1ceb400640 1 -- 172.21.15.116:0/795409968 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d040046e0 con 0x564538195700 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.760+0000 7f1ceb400640 1 -- 172.21.15.116:0/795409968 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1cf8050a30 con 0x564538195700 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d13441740 10 client.26363 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d13441740 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d13441740 20 client.26363 populate_metadata read hostname 'smithi116' 2024-03-04T07:44:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d13441740 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645382089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d13441740 4 client.26363 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5645381015c0 con 0x5645381aaa70 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d0de00640 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645382089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d0de00640 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645382089b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0509000f0fd8975 server_cookie=ce120313c8cb83a6 in_seq=0 out_seq=0 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1ceb400640 10 client.26363 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1ceb400640 1 -- 172.21.15.116:0/795409968 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5645381015c0 con 0x5645381aaa70 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1ceb400640 10 client.26363 handle_command_reply: tid=0 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.761+0000 7f1d13441740 1 client.26363 shutdown 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538195700 msgr2=0x5645381a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538195700 0x5645381a42d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1cf802f020 tx=0x7f1cf802bc50 comp rx=0 tx=0).stop 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 shutdown_connections 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5645381aaa70 0x5645382089b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56453819a870 0x5645381a5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564538195700 0x5645381a42d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 --2- 172.21.15.116:0/795409968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564538194d10 0x5645381a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 >> 172.21.15.116:0/795409968 conn(0x564538190280 msgr2=0x5645381934e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1ceb400640 0 client.26363 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 shutdown_connections 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 1 -- 172.21.15.116:0/795409968 wait complete. 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.762+0000 7f1d13441740 20 client.26363 trim_cache size 0 max 16384 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:44:57.767 INFO:teuthology.orchestra.run.smithi116.stdout: "e010ba7d-7cde-4573-95bc-45aaff9709a0": { 2024-03-04T07:44:57.768 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:44:57.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "e010ba7d-7cde-4573-95bc-45aaff9709a0", 2024-03-04T07:44:57.768 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:44:57.768 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:44:57.768 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:44:57.768 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cf00a1e00 msgr2=0x7f1cf00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cf00a1e00 0x7f1cf00a42c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x564538195f60 tx=0x7f1d04009500 comp rx=0 tx=0).stop 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081014b0 msgr2=0x7f1d08075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081014b0 0x7f1d08075380 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00d950 tx=0x7f1cfc00de10 comp rx=0 tx=0).stop 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 shutdown_connections 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1cf00a1e00 0x7f1cf00a42c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d08103060 0x7f1d08073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1d081026a0 0x7f1d080739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 --2- 172.21.15.116:0/1778593457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d081014b0 0x7f1d08075380 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 >> 172.21.15.116:0/1778593457 conn(0x7f1d080fca40 msgr2=0x7f1d081059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 shutdown_connections 2024-03-04T07:44:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:44:57.763+0000 7f1d0f200640 1 -- 172.21.15.116:0/1778593457 wait complete. 2024-03-04T07:44:57.780 INFO:tasks.cephfs.filesystem:scrub status for tag:e010ba7d-7cde-4573-95bc-45aaff9709a0 - {'path': '/', 'tag': 'e010ba7d-7cde-4573-95bc-45aaff9709a0', 'options': 'recursive,force'} 2024-03-04T07:44:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:58 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:44:59.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:58 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:59.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:58 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:44:59.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:58 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:59.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:58 smithi116 ceph-mon[32150]: pgmap v527: 97 pgs: 97 active+clean; 3.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:44:59.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:44:58 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:44:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:58 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:44:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:58 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:58 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:44:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:58 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:58 smithi152 ceph-mon[37154]: pgmap v527: 97 pgs: 97 active+clean; 3.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:44:59.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:44:58 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:44:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:58 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:44:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:58 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:58 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:44:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:58 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:44:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:58 smithi133 ceph-mon[36505]: pgmap v527: 97 pgs: 97 active+clean; 3.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:44:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:44:58 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:45:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:00 smithi116 ceph-mon[32150]: pgmap v528: 97 pgs: 97 active+clean; 3.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:45:01.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:00 smithi152 ceph-mon[37154]: pgmap v528: 97 pgs: 97 active+clean; 3.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:45:01.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:00 smithi133 ceph-mon[36505]: pgmap v528: 97 pgs: 97 active+clean; 3.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:45:03.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:02 smithi116 ceph-mon[32150]: pgmap v529: 97 pgs: 97 active+clean; 3.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:45:03.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:02 smithi152 ceph-mon[37154]: pgmap v529: 97 pgs: 97 active+clean; 3.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:45:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:02 smithi133 ceph-mon[36505]: pgmap v529: 97 pgs: 97 active+clean; 3.0 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:45:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:04 smithi116 ceph-mon[32150]: pgmap v530: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:45:05.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:04 smithi152 ceph-mon[37154]: pgmap v530: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:45:05.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:04 smithi133 ceph-mon[36505]: pgmap v530: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:45:05.192 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:127/600: open fd = 3 2024-03-04T07:45:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.193 INFO:tasks.workunit.client.0.smithi116.stdout:128/600: open fd = 3 2024-03-04T07:45:05.194 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.194 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.194 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.194 INFO:tasks.workunit.client.0.smithi116.stdout:129/600: open fd = 3 2024-03-04T07:45:05.194 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.194 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:130/600: open fd = 3 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:131/600: open fd = 3 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.195 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:132/600: open fd = 3 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:133/600: open fd = 3 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:134/600: open fd = 3 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.196 INFO:tasks.workunit.client.0.smithi116.stdout:135/600: open fd = 3 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:136/600: open fd = 3 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:137/600: open fd = 3 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.197 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:138/600: open fd = 3 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:139/600: open fd = 3 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:140/600: open fd = 3 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:141/600: open fd = 3 2024-03-04T07:45:05.198 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:142/600: open fd = 3 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:143/600: open fd = 3 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.199 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.200 INFO:tasks.workunit.client.0.smithi116.stdout:144/600: open fd = 3 2024-03-04T07:45:05.200 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.200 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.200 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.200 INFO:tasks.workunit.client.0.smithi116.stdout:145/600: open fd = 3 2024-03-04T07:45:05.200 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.200 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.201 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.201 INFO:tasks.workunit.client.0.smithi116.stdout:146/600: open fd = 3 2024-03-04T07:45:05.201 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.201 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.201 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.201 INFO:tasks.workunit.client.0.smithi116.stdout:147/600: open fd = 3 2024-03-04T07:45:05.202 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.202 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.202 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.202 INFO:tasks.workunit.client.0.smithi116.stdout:148/600: open fd = 3 2024-03-04T07:45:05.202 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.202 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:149/600: open fd = 3 2024-03-04T07:45:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.203 INFO:tasks.workunit.client.0.smithi116.stdout:150/600: open fd = 3 2024-03-04T07:45:05.204 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.204 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.204 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.204 INFO:tasks.workunit.client.0.smithi116.stdout:151/600: open fd = 3 2024-03-04T07:45:05.204 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.204 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.205 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.205 INFO:tasks.workunit.client.0.smithi116.stdout:152/600: open fd = 3 2024-03-04T07:45:05.205 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.205 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.205 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.205 INFO:tasks.workunit.client.0.smithi116.stdout:153/600: open fd = 3 2024-03-04T07:45:05.205 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.206 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.206 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.206 INFO:tasks.workunit.client.0.smithi116.stdout:154/600: open fd = 3 2024-03-04T07:45:05.206 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.206 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.206 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.207 INFO:tasks.workunit.client.0.smithi116.stdout:155/600: open fd = 3 2024-03-04T07:45:05.207 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.207 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.207 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.207 INFO:tasks.workunit.client.0.smithi116.stdout:156/600: open fd = 3 2024-03-04T07:45:05.207 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.208 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.208 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.208 INFO:tasks.workunit.client.0.smithi116.stdout:157/600: open fd = 3 2024-03-04T07:45:05.208 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.208 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.208 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:158/600: open fd = 3 2024-03-04T07:45:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:159/600: open fd = 3 2024-03-04T07:45:05.209 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:160/600: open fd = 3 2024-03-04T07:45:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.210 INFO:tasks.workunit.client.0.smithi116.stdout:161/600: open fd = 3 2024-03-04T07:45:05.211 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.211 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.211 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.211 INFO:tasks.workunit.client.0.smithi116.stdout:162/600: open fd = 3 2024-03-04T07:45:05.211 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.212 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.212 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.212 INFO:tasks.workunit.client.0.smithi116.stdout:163/600: open fd = 3 2024-03-04T07:45:05.212 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.212 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.212 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.213 INFO:tasks.workunit.client.0.smithi116.stdout:164/600: open fd = 3 2024-03-04T07:45:05.213 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.213 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.213 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.213 INFO:tasks.workunit.client.0.smithi116.stdout:165/600: open fd = 3 2024-03-04T07:45:05.213 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:166/600: open fd = 3 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:167/600: open fd = 3 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.214 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.215 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.215 INFO:tasks.workunit.client.0.smithi116.stdout:168/600: open fd = 3 2024-03-04T07:45:05.215 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.215 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.215 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.215 INFO:tasks.workunit.client.0.smithi116.stdout:169/600: open fd = 3 2024-03-04T07:45:05.215 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.216 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.216 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.216 INFO:tasks.workunit.client.0.smithi116.stdout:170/600: open fd = 3 2024-03-04T07:45:05.216 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.216 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.216 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.217 INFO:tasks.workunit.client.0.smithi116.stdout:171/600: open fd = 3 2024-03-04T07:45:05.217 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.217 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.217 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.217 INFO:tasks.workunit.client.0.smithi116.stdout:172/600: open fd = 3 2024-03-04T07:45:05.217 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.218 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.218 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.218 INFO:tasks.workunit.client.0.smithi116.stdout:173/600: open fd = 3 2024-03-04T07:45:05.218 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.218 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.218 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.219 INFO:tasks.workunit.client.0.smithi116.stdout:174/600: open fd = 3 2024-03-04T07:45:05.219 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.219 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.219 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.219 INFO:tasks.workunit.client.0.smithi116.stdout:175/600: open fd = 3 2024-03-04T07:45:05.219 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.219 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.220 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.220 INFO:tasks.workunit.client.0.smithi116.stdout:176/600: open fd = 3 2024-03-04T07:45:05.220 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.220 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.220 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.220 INFO:tasks.workunit.client.0.smithi116.stdout:177/600: open fd = 3 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:178/600: open fd = 3 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:179/600: open fd = 3 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:180/600: open fd = 3 2024-03-04T07:45:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:181/600: open fd = 3 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:182/600: open fd = 3 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:183/600: open fd = 3 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.222 INFO:tasks.workunit.client.0.smithi116.stdout:184/600: open fd = 3 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:185/600: open fd = 3 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:186/600: open fd = 3 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:187/600: open fd = 3 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.223 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:45:05.224 INFO:tasks.workunit.client.0.smithi116.stdout:188/600: open fd = 3 2024-03-04T07:45:05.224 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:05.224 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:45:07.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:06 smithi116 ceph-mon[32150]: pgmap v531: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:45:07.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:06 smithi152 ceph-mon[37154]: pgmap v531: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:45:07.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:06 smithi133 ceph-mon[36505]: pgmap v531: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:45:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 -- 172.21.15.116:0/390606036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81031b0 msgr2=0x7ff0d8109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 --2- 172.21.15.116:0/390606036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81031b0 0x7ff0d8109a40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4009ca0 tx=0x7ff0d402c0f0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 -- 172.21.15.116:0/390606036 shutdown_connections 2024-03-04T07:45:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 --2- 172.21.15.116:0/390606036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81031b0 0x7ff0d8109a40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 --2- 172.21.15.116:0/390606036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d81027f0 0x7ff0d8102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 --2- 172.21.15.116:0/390606036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8101600 0x7ff0d8101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 -- 172.21.15.116:0/390606036 >> 172.21.15.116:0/390606036 conn(0x7ff0d80fcb90 msgr2=0x7ff0d80fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 -- 172.21.15.116:0/390606036 shutdown_connections 2024-03-04T07:45:08.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.923+0000 7ff0e0800640 1 -- 172.21.15.116:0/390606036 wait complete. 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.924+0000 7ff0e0800640 1 Processor -- start 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.924+0000 7ff0e0800640 1 -- start start 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0e0800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d8101600 0x7ff0d8199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0e0800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81027f0 0x7ff0d8199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0e0800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d81031b0 0x7ff0d81a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0e0800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d810d280 con 0x7ff0d81031b0 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0e0800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d810d100 con 0x7ff0d81027f0 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0e0800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d810d400 con 0x7ff0d8101600 2024-03-04T07:45:08.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0de800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d81031b0 0x7ff0d81a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81027f0 0x7ff0d8199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81027f0 0x7ff0d8199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59402/0 (socket says 172.21.15.116:59402) 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0dde00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d8101600 0x7ff0d8199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0dd400640 1 -- 172.21.15.116:0/3907257745 learned_addr learned my addr 172.21.15.116:0/3907257745 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0de800640 1 -- 172.21.15.116:0/3907257745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d8101600 msgr2=0x7ff0d8199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0de800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d8101600 0x7ff0d8199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0de800640 1 -- 172.21.15.116:0/3907257745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81027f0 msgr2=0x7ff0d8199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0de800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81027f0 0x7ff0d8199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0de800640 1 -- 172.21.15.116:0/3907257745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d81a1640 con 0x7ff0d81031b0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.925+0000 7ff0dd400640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81027f0 0x7ff0d8199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.926+0000 7ff0de800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d81031b0 0x7ff0d81a0fd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4037040 tx=0x7ff0d4032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.926+0000 7ff0d2a00640 1 -- 172.21.15.116:0/3907257745 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4041020 con 0x7ff0d81031b0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.926+0000 7ff0d2a00640 1 -- 172.21.15.116:0/3907257745 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0d4034cd0 con 0x7ff0d81031b0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.926+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d81a18d0 con 0x7ff0d81031b0 2024-03-04T07:45:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.926+0000 7ff0d2a00640 1 -- 172.21.15.116:0/3907257745 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4039040 con 0x7ff0d81031b0 2024-03-04T07:45:08.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.926+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d81a1d90 con 0x7ff0d81031b0 2024-03-04T07:45:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.929+0000 7ff0e35aa740 1 Processor -- start 2024-03-04T07:45:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.929+0000 7ff0d2a00640 1 -- 172.21.15.116:0/3907257745 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff0d4044020 con 0x7ff0d81031b0 2024-03-04T07:45:08.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0e35aa740 1 -- start start 2024-03-04T07:45:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae41d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae44840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40ada2370 con 0x55d40ae3d240 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40ad4b550 con 0x55d40ae3c850 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40ad9e410 con 0x55d40ae423b0 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0dde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae41d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0dd400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59412/0 (socket says 172.21.15.116:59412) 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0dd400640 1 -- 172.21.15.116:0/542319205 learned_addr learned my addr 172.21.15.116:0/542319205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.930+0000 7ff0dd400640 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae44840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0dd400640 1 -- 172.21.15.116:0/542319205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 msgr2=0x55d40ae44840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0dd400640 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae44840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0dd400640 1 -- 172.21.15.116:0/542319205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 msgr2=0x55d40ae41d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0dd400640 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae41d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0dd400640 1 -- 172.21.15.116:0/542319205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d40ad5f3c0 con 0x55d40ae3c850 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0dd400640 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3cc70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009810 tx=0x7ff0c8009c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22b2984aeb8ecbec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0c7000640 1 -- 172.21.15.116:0/542319205 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800fc50 con 0x55d40ae3c850 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0c7000640 1 -- 172.21.15.116:0/542319205 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c800fdf0 con 0x55d40ae3c850 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0c7000640 1 -- 172.21.15.116:0/542319205 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800e730 con 0x55d40ae3c850 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0d2a00640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0c00a1e20 0x7ff0c00a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0d2a00640 1 -- 172.21.15.116:0/3907257745 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff0d40efb60 con 0x7ff0d81031b0 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.931+0000 7ff0dde00640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0c00a1e20 0x7ff0c00a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 -- 172.21.15.116:0/542319205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 msgr2=0x55d40ae3cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3cc70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009810 tx=0x7ff0c8009c80 comp rx=0 tx=0).stop 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 -- 172.21.15.116:0/542319205 shutdown_connections 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae44840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae41d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/542319205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3cc70 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 -- 172.21.15.116:0/542319205 >> 172.21.15.116:0/542319205 conn(0x55d40ae37dc0 msgr2=0x55d40ae381c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0dde00640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0c00a1e20 0x7ff0c00a42e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55d40ae3d640 tx=0x7ff0cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 -- 172.21.15.116:0/542319205 shutdown_connections 2024-03-04T07:45:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.932+0000 7ff0e35aa740 1 -- 172.21.15.116:0/542319205 wait complete. 2024-03-04T07:45:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 -- start start 2024-03-04T07:45:08.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 Processor -- start 2024-03-04T07:45:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3c850 0x55d40ae46be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae47120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae423b0 0x55d40ae4e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40ae45360 con 0x55d40ae3d240 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40ad5e9b0 con 0x55d40ae423b0 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40ae454e0 con 0x55d40ae3c850 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.933+0000 7ff0e35aa740 10 client.0 _ll_register_callbacks cb 0x55d40ad9a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae423b0 0x55d40ae4e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae423b0 0x55d40ae4e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59418/0 (socket says 172.21.15.116:59418) 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 -- 172.21.15.116:0/724436483 learned_addr learned my addr 172.21.15.116:0/724436483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0dd400640 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae47120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0dde00640 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3c850 0x55d40ae46be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 -- 172.21.15.116:0/724436483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3c850 msgr2=0x55d40ae46be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3c850 0x55d40ae46be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 -- 172.21.15.116:0/724436483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 msgr2=0x55d40ae47120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae47120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 -- 172.21.15.116:0/724436483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d40aab7210 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0de800640 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae423b0 0x55d40ae4e160 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40b92b0 tx=0x7ff0d40bc420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e91cee96b077bc41 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0bb400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0bb400640 1 -- 172.21.15.116:0/724436483 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d40098b0 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0bb400640 1 -- 172.21.15.116:0/724436483 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff0d40a70b0 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.934+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0c800fdf0 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0bb400640 1 -- 172.21.15.116:0/724436483 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0d40bc8e0 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 10 client.26375 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d40ab83620 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0bb400640 1 -- 172.21.15.116:0/724436483 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c800fdf0 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0bb400640 1 -- 172.21.15.116:0/724436483 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff0d403c030 con 0x55d40ae423b0 2024-03-04T07:45:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 10 client.26375 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 10 client.26375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 20 client.26375 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40ae52920 0x55d40ae54d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 4 client.26375 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.935+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d40adaf8b0 con 0x55d40ae52920 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.936+0000 7ff0dde00640 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40ae52920 0x55d40ae54d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.936+0000 7ff0dde00640 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40ae52920 0x55d40ae54d30 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1654624a22b09ff2 server_cookie=79eda28758de16fb in_seq=0 out_seq=0 2024-03-04T07:45:08.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.936+0000 7ff0bb400640 10 client.26375 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0bb400640 1 -- 172.21.15.116:0/724436483 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d40adaf8b0 con 0x55d40ae52920 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0bb400640 10 client.26375 handle_command_reply: tid=0 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 client.26375 shutdown 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae423b0 msgr2=0x55d40ae4e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae423b0 0x55d40ae4e160 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff0d40b92b0 tx=0x7ff0d40bc420 comp rx=0 tx=0).stop 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 shutdown_connections 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40ae52920 0x55d40ae54d30 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae423b0 0x55d40ae4e160 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0bb400640 0 client.26375 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae47120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/724436483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3c850 0x55d40ae46be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 >> 172.21.15.116:0/724436483 conn(0x55d40ae37dc0 msgr2=0x55d40ae36fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 shutdown_connections 2024-03-04T07:45:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 1 -- 172.21.15.116:0/724436483 wait complete. 2024-03-04T07:45:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.937+0000 7ff0e35aa740 20 client.26375 trim_cache size 0 max 16384 2024-03-04T07:45:08.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.952+0000 7ff0e35aa740 1 Processor -- start 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.952+0000 7ff0e35aa740 1 -- start start 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae4e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae3b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81a8b80 con 0x55d40ae3d240 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81ab160 con 0x55d40ae3c850 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81a9180 con 0x55d40ae423b0 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59434/0 (socket says 172.21.15.116:59434) 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 -- 172.21.15.116:0/696176680 learned_addr learned my addr 172.21.15.116:0/696176680 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0de800640 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae3b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dd400640 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae4e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 -- 172.21.15.116:0/696176680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 msgr2=0x55d40ae3b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae3b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 -- 172.21.15.116:0/696176680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 msgr2=0x55d40ae4e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae4e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dde00640 1 -- 172.21.15.116:0/696176680 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d40ae553b0 con 0x55d40ae3c850 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.953+0000 7ff0dd400640 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae4e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.954+0000 7ff0dde00640 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3b020 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc018000 tx=0x7ff0cc022040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61fb4b286a1ade4f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.954+0000 7ff0c6600640 1 -- 172.21.15.116:0/696176680 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc003c40 con 0x55d40ae3c850 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.954+0000 7ff0c6600640 1 -- 172.21.15.116:0/696176680 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0cc030070 con 0x55d40ae3c850 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.954+0000 7ff0c6600640 1 -- 172.21.15.116:0/696176680 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc01f750 con 0x55d40ae3c850 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 -- 172.21.15.116:0/696176680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 msgr2=0x55d40ae3b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3b020 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc018000 tx=0x7ff0cc022040 comp rx=0 tx=0).stop 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 -- 172.21.15.116:0/696176680 shutdown_connections 2024-03-04T07:45:08.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae423b0 0x55d40ae3b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae3d240 0x55d40ae4e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/696176680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae3b020 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 -- 172.21.15.116:0/696176680 >> 172.21.15.116:0/696176680 conn(0x55d40ae37dc0 msgr2=0x55d40ae3e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 -- 172.21.15.116:0/696176680 shutdown_connections 2024-03-04T07:45:08.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 -- 172.21.15.116:0/696176680 wait complete. 2024-03-04T07:45:08.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 -- start start 2024-03-04T07:45:08.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.955+0000 7ff0e35aa740 1 Processor -- start 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae4d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3d240 0x55d40ae4de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0e35aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae423b0 0x55d40ae4bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81a97e0 con 0x55d40ae423b0 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81aab00 con 0x55d40ae3c850 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0e35aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81a9e40 con 0x55d40ae3d240 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0e35aa740 10 client.0 _ll_register_callbacks cb 0x7ff0d4003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae4d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0de800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae423b0 0x55d40ae4bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae4d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59436/0 (socket says 172.21.15.116:59436) 2024-03-04T07:45:08.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 -- 172.21.15.116:0/2352857040 learned_addr learned my addr 172.21.15.116:0/2352857040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dd400640 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3d240 0x55d40ae4de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 -- 172.21.15.116:0/2352857040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3d240 msgr2=0x55d40ae4de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3d240 0x55d40ae4de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 -- 172.21.15.116:0/2352857040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae423b0 msgr2=0x55d40ae4bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae423b0 0x55d40ae4bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.956+0000 7ff0dde00640 1 -- 172.21.15.116:0/2352857040 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d40ae4c590 con 0x55d40ae3c850 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0dde00640 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae4d910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc019040 tx=0x7ff0cc003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1381bc6606e8fd25 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0bb400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0bb400640 1 -- 172.21.15.116:0/2352857040 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc02b880 con 0x55d40ae3c850 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0bb400640 1 -- 172.21.15.116:0/2352857040 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff0cc02ba20 con 0x55d40ae3c850 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0cc030070 con 0x55d40ae3c850 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0bb400640 1 -- 172.21.15.116:0/2352857040 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0cc046050 con 0x55d40ae3c850 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0e35aa740 10 client.26381 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:08.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d40aeaeb50 con 0x55d40ae3c850 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.957+0000 7ff0bb400640 1 -- 172.21.15.116:0/2352857040 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0cc030070 con 0x55d40ae3c850 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0bb400640 1 -- 172.21.15.116:0/2352857040 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff0cc03f450 con 0x55d40ae3c850 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0e35aa740 10 client.26381 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0e35aa740 10 client.26381 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0e35aa740 20 client.26381 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40aeaf480 0x55d40aeb18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0e35aa740 4 client.26381 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d40ada9c00 con 0x55d40aeaf480 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0dd400640 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40aeaf480 0x55d40aeb18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0dd400640 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40aeaf480 0x55d40aeb18a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f0226cfb0a6ed5 server_cookie=d06aaf3fd687cec6 in_seq=0 out_seq=0 2024-03-04T07:45:08.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.958+0000 7ff0bb400640 10 client.26381 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0bb400640 1 -- 172.21.15.116:0/2352857040 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d40ada9c00 con 0x55d40aeaf480 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0bb400640 10 client.26381 handle_command_reply: tid=0 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 client.26381 shutdown 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 msgr2=0x55d40ae4d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae4d910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc019040 tx=0x7ff0cc003cd0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 shutdown_connections 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d40aeaf480 0x55d40aeb18a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d40ae423b0 0x55d40ae4bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0bb400640 0 client.26381 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d40ae3d240 0x55d40ae4de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 --2- 172.21.15.116:0/2352857040 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d40ae3c850 0x55d40ae4d910 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.973+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 >> 172.21.15.116:0/2352857040 conn(0x55d40ae37dc0 msgr2=0x55d40ae3b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.974+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 shutdown_connections 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.974+0000 7ff0e35aa740 1 -- 172.21.15.116:0/2352857040 wait complete. 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.974+0000 7ff0e35aa740 20 client.26381 trim_cache size 0 max 16384 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:45:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:45:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0c00a1e20 msgr2=0x7ff0c00a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0c00a1e20 0x7ff0c00a42e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55d40ae3d640 tx=0x7ff0cc005f90 comp rx=0 tx=0).stop 2024-03-04T07:45:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d81031b0 msgr2=0x7ff0d81a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d81031b0 0x7ff0d81a0fd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4037040 tx=0x7ff0d4032ea0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 shutdown_connections 2024-03-04T07:45:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d81031b0 0x7ff0d81a0fd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d81027f0 0x7ff0d8199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0c00a1e20 0x7ff0c00a42e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 --2- 172.21.15.116:0/3907257745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d8101600 0x7ff0d8199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 >> 172.21.15.116:0/3907257745 conn(0x7ff0d80fcb90 msgr2=0x7ff0d8073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 shutdown_connections 2024-03-04T07:45:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:08.975+0000 7ff0e0800640 1 -- 172.21.15.116:0/3907257745 wait complete. 2024-03-04T07:45:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:08 smithi116 ceph-mon[32150]: pgmap v532: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:45:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:08 smithi152 ceph-mon[37154]: pgmap v532: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:45:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:08 smithi133 ceph-mon[36505]: pgmap v532: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:45:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:10 smithi116 ceph-mon[32150]: pgmap v533: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:10 smithi152 ceph-mon[37154]: pgmap v533: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:10 smithi133 ceph-mon[36505]: pgmap v533: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:12 smithi116 ceph-mon[32150]: pgmap v534: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:45:13.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:12 smithi152 ceph-mon[37154]: pgmap v534: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:45:13.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:12 smithi133 ceph-mon[36505]: pgmap v534: 97 pgs: 97 active+clean; 3.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:45:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:14 smithi116 ceph-mon[32150]: pgmap v535: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:14 smithi152 ceph-mon[37154]: pgmap v535: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:15.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:14 smithi133 ceph-mon[36505]: pgmap v535: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:17.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:16 smithi116 ceph-mon[32150]: pgmap v536: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:17.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:16 smithi152 ceph-mon[37154]: pgmap v536: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:17.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:16 smithi133 ceph-mon[36505]: pgmap v536: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:18 smithi152 ceph-mon[37154]: pgmap v537: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:45:19.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:18 smithi133 ceph-mon[36505]: pgmap v537: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:45:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:18 smithi116 ceph-mon[32150]: pgmap v537: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:45:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:20 smithi152 ceph-mon[37154]: pgmap v538: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:45:21.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:20 smithi133 ceph-mon[36505]: pgmap v538: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:45:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:20 smithi116 ceph-mon[32150]: pgmap v538: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:45:22.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.116+0000 7fc14be00640 1 -- 172.21.15.116:0/3808565811 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c103000 msgr2=0x7fc14c109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.116+0000 7fc14be00640 1 --2- 172.21.15.116:0/3808565811 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c103000 0x7fc14c109890 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc144009ca0 tx=0x7fc14402c100 comp rx=0 tx=0).stop 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.117+0000 7fc14be00640 1 -- 172.21.15.116:0/3808565811 shutdown_connections 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.117+0000 7fc14be00640 1 --2- 172.21.15.116:0/3808565811 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c103000 0x7fc14c109890 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.117+0000 7fc14be00640 1 --2- 172.21.15.116:0/3808565811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c102640 0x7fc14c102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.117+0000 7fc14be00640 1 --2- 172.21.15.116:0/3808565811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc14c101450 0x7fc14c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.117+0000 7fc14be00640 1 -- 172.21.15.116:0/3808565811 >> 172.21.15.116:0/3808565811 conn(0x7fc14c0fc9e0 msgr2=0x7fc14c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.117+0000 7fc14be00640 1 -- 172.21.15.116:0/3808565811 shutdown_connections 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.117+0000 7fc14be00640 1 -- 172.21.15.116:0/3808565811 wait complete. 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 Processor -- start 2024-03-04T07:45:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 -- start start 2024-03-04T07:45:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c101450 0x7fc14c199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c102640 0x7fc14c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc14c103000 0x7fc14c1a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c10d0e0 con 0x7fc14c102640 2024-03-04T07:45:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c10cf60 con 0x7fc14c101450 2024-03-04T07:45:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14be00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c10d260 con 0x7fc14c103000 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c101450 0x7fc14c199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c101450 0x7fc14c199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40774/0 (socket says 172.21.15.116:40774) 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc14aa00640 1 -- 172.21.15.116:0/928584321 learned_addr learned my addr 172.21.15.116:0/928584321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.118+0000 7fc143e00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c102640 0x7fc14c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14b400640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc14c103000 0x7fc14c1a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14aa00640 1 -- 172.21.15.116:0/928584321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc14c103000 msgr2=0x7fc14c1a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14aa00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc14c103000 0x7fc14c1a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14aa00640 1 -- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c102640 msgr2=0x7fc14c199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14aa00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c102640 0x7fc14c199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14aa00640 1 -- 172.21.15.116:0/928584321 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c1a1520 con 0x7fc14c101450 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14aa00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c101450 0x7fc14c199900 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00e9c0 tx=0x7fc13c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc150a00640 1 -- 172.21.15.116:0/928584321 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c00cd80 con 0x7fc14c101450 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc150a00640 1 -- 172.21.15.116:0/928584321 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc13c0035f0 con 0x7fc14c101450 2024-03-04T07:45:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc150a00640 1 -- 172.21.15.116:0/928584321 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c010690 con 0x7fc14c101450 2024-03-04T07:45:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.119+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c1a1810 con 0x7fc14c101450 2024-03-04T07:45:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.120+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14c1a1db0 con 0x7fc14c101450 2024-03-04T07:45:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc150a00640 1 -- 172.21.15.116:0/928584321 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fc13c0037a0 con 0x7fc14c101450 2024-03-04T07:45:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 Processor -- start 2024-03-04T07:45:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 -- start start 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f159e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f15ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f161a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f12f0bf370 con 0x55f12f159a10 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f12f068550 con 0x55f12f15a400 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc154a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f12f0bb410 con 0x55f12f15f570 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.123+0000 7fc14b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f15ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f159e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42010/0 (socket says 172.21.15.116:42010) 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 -- 172.21.15.116:0/2793919616 learned_addr learned my addr 172.21.15.116:0/2793919616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc143e00640 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f161a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 -- 172.21.15.116:0/2793919616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 msgr2=0x55f12f161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f161a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 -- 172.21.15.116:0/2793919616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 msgr2=0x55f12f15ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f15ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 -- 172.21.15.116:0/2793919616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f12f07c3c0 con 0x55f12f159a10 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc143e00640 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f161a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14aa00640 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f15ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc150a00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc1300a1df0 0x7fc1300a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14b400640 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f159e30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc144005e30 tx=0x7fc1440323d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdaa1abddeddc4a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc150a00640 1 -- 172.21.15.116:0/928584321 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc13c0ce160 con 0x7fc14c101450 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.124+0000 7fc14aa00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc1300a1df0 0x7fc1300a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.125+0000 7fc13be00640 1 -- 172.21.15.116:0/2793919616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144043020 con 0x55f12f159a10 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.125+0000 7fc13be00640 1 -- 172.21.15.116:0/2793919616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc144032c90 con 0x55f12f159a10 2024-03-04T07:45:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.125+0000 7fc13be00640 1 -- 172.21.15.116:0/2793919616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144039040 con 0x55f12f159a10 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.125+0000 7fc154a20740 1 -- 172.21.15.116:0/2793919616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 msgr2=0x55f12f159e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.125+0000 7fc154a20740 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f159e30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc144005e30 tx=0x7fc1440323d0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.125+0000 7fc14aa00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc1300a1df0 0x7fc1300a42b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc13c09ae20 tx=0x7fc13c0d1be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 -- 172.21.15.116:0/2793919616 shutdown_connections 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f161a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f15ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 --2- 172.21.15.116:0/2793919616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f159e30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 -- 172.21.15.116:0/2793919616 >> 172.21.15.116:0/2793919616 conn(0x55f12f154f80 msgr2=0x55f12f155380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:22.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 -- 172.21.15.116:0/2793919616 shutdown_connections 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 -- 172.21.15.116:0/2793919616 wait complete. 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 -- start start 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.126+0000 7fc154a20740 1 Processor -- start 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc154a20740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f163d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc154a20740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15a400 0x55f12f164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc154a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f16b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc154a20740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f12f1624c0 con 0x55f12f15a400 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc154a20740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f12f07b9b0 con 0x55f12f159a10 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc154a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f12f162640 con 0x55f12f15f570 2024-03-04T07:45:22.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc154a20740 10 client.0 _ll_register_callbacks cb 0x55f12f0b7340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc14b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f163d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc143e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15a400 0x55f12f164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc14b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f163d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40796/0 (socket says 172.21.15.116:40796) 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc143e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15a400 0x55f12f164260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42014/0 (socket says 172.21.15.116:42014) 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc14b400640 1 -- 172.21.15.116:0/120750861 learned_addr learned my addr 172.21.15.116:0/120750861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.127+0000 7fc14b400640 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f16b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc143e00640 1 -- 172.21.15.116:0/120750861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 msgr2=0x55f12f16b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc143e00640 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f16b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc143e00640 1 -- 172.21.15.116:0/120750861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 msgr2=0x55f12f163d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc143e00640 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f163d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc143e00640 1 -- 172.21.15.116:0/120750861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f12edd4210 con 0x55f12f15a400 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc143e00640 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15a400 0x55f12f164260 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc14c19ae10 tx=0x7fc134012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5efa9234ad40ca58 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc138c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:45:22.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc138c00640 1 -- 172.21.15.116:0/120750861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134004050 con 0x55f12f15a400 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc138c00640 1 -- 172.21.15.116:0/120750861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc1340041f0 con 0x55f12f15a400 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.128+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc144032c90 con 0x55f12f15a400 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc138c00640 1 -- 172.21.15.116:0/120750861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1340043f0 con 0x55f12f15a400 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 10 client.16266 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f12eea0620 con 0x55f12f15a400 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc138c00640 1 -- 172.21.15.116:0/120750861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc144032c90 con 0x55f12f15a400 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc138c00640 1 -- 172.21.15.116:0/120750861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc1340047a0 con 0x55f12f15a400 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 10 client.16266 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 10 client.16266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 20 client.16266 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f16fb70 0x55f12f171f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 4 client.16266 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f12f0cc8b0 con 0x55f12f16fb70 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.129+0000 7fc14b400640 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f16fb70 0x55f12f171f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.130+0000 7fc14b400640 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f16fb70 0x55f12f171f80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b2b25c8cecfd928 server_cookie=5fff02e63e343332 in_seq=0 out_seq=0 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.130+0000 7fc138c00640 10 client.16266 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.130+0000 7fc138c00640 1 -- 172.21.15.116:0/120750861 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f12f0cc8b0 con 0x55f12f16fb70 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.130+0000 7fc138c00640 10 client.16266 handle_command_reply: tid=0 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 client.16266 shutdown 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15a400 msgr2=0x55f12f164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15a400 0x55f12f164260 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc14c19ae10 tx=0x7fc134012b60 comp rx=0 tx=0).stop 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 shutdown_connections 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f16fb70 0x55f12f171f80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f16b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15a400 0x55f12f164260 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 --2- 172.21.15.116:0/120750861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f163d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc138c00640 0 client.16266 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:22.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 >> 172.21.15.116:0/120750861 conn(0x55f12f154f80 msgr2=0x55f12f154200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:22.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 shutdown_connections 2024-03-04T07:45:22.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 1 -- 172.21.15.116:0/120750861 wait complete. 2024-03-04T07:45:22.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.131+0000 7fc154a20740 20 client.16266 trim_cache size 0 max 16384 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 Processor -- start 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 -- start start 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f171f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f16b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f15b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c1a8ba0 con 0x55f12f159a10 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c1ab180 con 0x55f12f15a400 2024-03-04T07:45:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.147+0000 7fc154a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c1a91a0 con 0x55f12f15f570 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f16b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f16b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40798/0 (socket says 172.21.15.116:40798) 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 -- 172.21.15.116:0/3693719179 learned_addr learned my addr 172.21.15.116:0/3693719179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc14b400640 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f171f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc14b400640 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f15b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 -- 172.21.15.116:0/3693719179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 msgr2=0x55f12f15b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f15b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 -- 172.21.15.116:0/3693719179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 msgr2=0x55f12f171f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f171f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 -- 172.21.15.116:0/3693719179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f12f159df0 con 0x55f12f15a400 2024-03-04T07:45:22.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.148+0000 7fc143e00640 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f16b260 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc134000c00 tx=0x7fc134004820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5af7052ab5a1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.149+0000 7fc13b400640 1 -- 172.21.15.116:0/3693719179 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13400f790 con 0x55f12f15a400 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.149+0000 7fc13b400640 1 -- 172.21.15.116:0/3693719179 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc13400fe10 con 0x55f12f15a400 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.149+0000 7fc13b400640 1 -- 172.21.15.116:0/3693719179 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134027460 con 0x55f12f15a400 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.149+0000 7fc154a20740 1 -- 172.21.15.116:0/3693719179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 msgr2=0x55f12f16b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.149+0000 7fc154a20740 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f16b260 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc134000c00 tx=0x7fc134004820 comp rx=0 tx=0).stop 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 -- 172.21.15.116:0/3693719179 shutdown_connections 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15f570 0x55f12f15b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f15a400 0x55f12f16b260 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 --2- 172.21.15.116:0/3693719179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f159a10 0x55f12f171f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 -- 172.21.15.116:0/3693719179 >> 172.21.15.116:0/3693719179 conn(0x55f12f154f80 msgr2=0x55f12f155360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 -- 172.21.15.116:0/3693719179 shutdown_connections 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 -- 172.21.15.116:0/3693719179 wait complete. 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 -- start start 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.150+0000 7fc154a20740 1 Processor -- start 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc154a20740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f168a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc154a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15a400 0x55f12f168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc154a20740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15f570 0x55f12f169fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc154a20740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c1a9800 con 0x55f12f15f570 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc154a20740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c1aab20 con 0x55f12f159a10 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc154a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c1a9e60 con 0x55f12f15a400 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc154a20740 10 client.0 _ll_register_callbacks cb 0x7fc134009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc14b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f168a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc143e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15a400 0x55f12f168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc14b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15f570 0x55f12f169fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc143e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15a400 0x55f12f168fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:52858/0 (socket says 172.21.15.116:52858) 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc143e00640 1 -- 172.21.15.116:0/3664984302 learned_addr learned my addr 172.21.15.116:0/3664984302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc143e00640 1 -- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 msgr2=0x55f12f168a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.151+0000 7fc143e00640 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f168a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc143e00640 1 -- 172.21.15.116:0/3664984302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15f570 msgr2=0x55f12f169fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc143e00640 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15f570 0x55f12f169fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc143e00640 1 -- 172.21.15.116:0/3664984302 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f12f16a5a0 con 0x55f12f15a400 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc14b400640 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f168a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc143e00640 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15a400 0x55f12f168fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc134012420 tx=0x7fc134009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f570d3222eab7d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc138c00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc138c00640 1 -- 172.21.15.116:0/3664984302 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc134004e20 con 0x55f12f15a400 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc138c00640 1 -- 172.21.15.116:0/3664984302 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc13400f9a0 con 0x55f12f15a400 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc13400fe10 con 0x55f12f15a400 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc138c00640 1 -- 172.21.15.116:0/3664984302 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc13400fba0 con 0x55f12f15a400 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc154a20740 10 client.26323 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.152+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f12f1cb4b0 con 0x55f12f15a400 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc138c00640 1 -- 172.21.15.116:0/3664984302 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc13400fe10 con 0x55f12f15a400 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc138c00640 1 -- 172.21.15.116:0/3664984302 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc134041020 con 0x55f12f15a400 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc154a20740 10 client.26323 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc154a20740 10 client.26323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc154a20740 20 client.26323 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc154a20740 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f1cbd20 0x55f12f1ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc154a20740 4 client.26323 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:45:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f12f0c6c00 con 0x55f12f1cbd20 2024-03-04T07:45:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.153+0000 7fc14b400640 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f1cbd20 0x55f12f1ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.154+0000 7fc14b400640 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f1cbd20 0x55f12f1ce140 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc5036ee7b4ebacb server_cookie=c1da2f74af5dac82 in_seq=0 out_seq=0 2024-03-04T07:45:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.154+0000 7fc138c00640 10 client.26323 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.170+0000 7fc138c00640 1 -- 172.21.15.116:0/3664984302 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f12f0c6c00 con 0x55f12f1cbd20 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.170+0000 7fc138c00640 10 client.26323 handle_command_reply: tid=0 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 client.26323 shutdown 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15a400 msgr2=0x55f12f168fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15a400 0x55f12f168fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc134012420 tx=0x7fc134009640 comp rx=0 tx=0).stop 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 shutdown_connections 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f12f1cbd20 0x55f12f1ce140 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12f15f570 0x55f12f169fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f12f15a400 0x55f12f168fa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc138c00640 0 client.26323 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 --2- 172.21.15.116:0/3664984302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f12f159a10 0x55f12f168a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 >> 172.21.15.116:0/3664984302 conn(0x55f12f154f80 msgr2=0x55f12f163ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 shutdown_connections 2024-03-04T07:45:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 1 -- 172.21.15.116:0/3664984302 wait complete. 2024-03-04T07:45:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.171+0000 7fc154a20740 20 client.26323 trim_cache size 0 max 16384 2024-03-04T07:45:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:45:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:45:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:45:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:45:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc1300a1df0 msgr2=0x7fc1300a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc1300a1df0 0x7fc1300a42b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc13c09ae20 tx=0x7fc13c0d1be0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c101450 msgr2=0x7fc14c199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c101450 0x7fc14c199900 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00e9c0 tx=0x7fc13c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:45:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 shutdown_connections 2024-03-04T07:45:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc1300a1df0 0x7fc1300a42b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc14c103000 0x7fc14c1a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc14c102640 0x7fc14c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 --2- 172.21.15.116:0/928584321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc14c101450 0x7fc14c199900 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:22.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 >> 172.21.15.116:0/928584321 conn(0x7fc14c0fc9e0 msgr2=0x7fc14c073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:22.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 shutdown_connections 2024-03-04T07:45:22.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:22.172+0000 7fc14be00640 1 -- 172.21.15.116:0/928584321 wait complete. 2024-03-04T07:45:23.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:22 smithi152 ceph-mon[37154]: pgmap v539: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:23.086 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:22 smithi133 ceph-mon[36505]: pgmap v539: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:22 smithi116 ceph-mon[32150]: pgmap v539: 97 pgs: 97 active+clean; 3.2 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:24 smithi133 ceph-mon[36505]: pgmap v540: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:45:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:24 smithi116 ceph-mon[32150]: pgmap v540: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:45:25.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:24 smithi152 ceph-mon[37154]: pgmap v540: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:45:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:26 smithi133 ceph-mon[36505]: pgmap v541: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:45:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:26 smithi116 ceph-mon[32150]: pgmap v541: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:45:27.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:26 smithi152 ceph-mon[37154]: pgmap v541: 97 pgs: 97 active+clean; 3.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:45:27.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:45:27.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:45:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 -- 172.21.15.116:0/673660828 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec108de0 msgr2=0x7f7cec109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 --2- 172.21.15.116:0/673660828 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec108de0 0x7f7cec109260 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009900 tx=0x7f7cd802bc30 comp rx=0 tx=0).stop 2024-03-04T07:45:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 -- 172.21.15.116:0/673660828 shutdown_connections 2024-03-04T07:45:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 --2- 172.21.15.116:0/673660828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cec1097a0 0x7f7cec110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 --2- 172.21.15.116:0/673660828 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec108de0 0x7f7cec109260 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 --2- 172.21.15.116:0/673660828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cec107bf0 0x7f7cec108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 -- 172.21.15.116:0/673660828 >> 172.21.15.116:0/673660828 conn(0x7f7cec076b90 msgr2=0x7f7cec078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 -- 172.21.15.116:0/673660828 shutdown_connections 2024-03-04T07:45:28.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.000+0000 7f7cf4600640 1 -- 172.21.15.116:0/673660828 wait complete. 2024-03-04T07:45:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.001+0000 7f7cf4600640 1 Processor -- start 2024-03-04T07:45:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.001+0000 7f7cf4600640 1 -- start start 2024-03-04T07:45:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf4600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec107bf0 0x7f7cec19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf4600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cec108de0 0x7f7cec19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf4600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cec1097a0 0x7f7cec1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf4600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec113870 con 0x7f7cec108de0 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf4600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1136f0 con 0x7f7cec107bf0 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf4600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1139f0 con 0x7f7cec1097a0 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cec108de0 0x7f7cec19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf1c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec107bf0 0x7f7cec19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cec108de0 0x7f7cec19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41072/0 (socket says 172.21.15.116:41072) 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf1200640 1 -- 172.21.15.116:0/3129916463 learned_addr learned my addr 172.21.15.116:0/3129916463 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.002+0000 7f7cf2600640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cec1097a0 0x7f7cec1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf1c00640 1 -- 172.21.15.116:0/3129916463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cec1097a0 msgr2=0x7f7cec1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf1c00640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cec1097a0 0x7f7cec1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf1c00640 1 -- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cec108de0 msgr2=0x7f7cec19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf1c00640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cec108de0 0x7f7cec19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf1c00640 1 -- 172.21.15.116:0/3129916463 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec1a5b00 con 0x7f7cec107bf0 2024-03-04T07:45:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf1c00640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec107bf0 0x7f7cec19dee0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9c0 tx=0x7f7ce000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cea000640 1 -- 172.21.15.116:0/3129916463 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000ccc0 con 0x7f7cec107bf0 2024-03-04T07:45:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cea000640 1 -- 172.21.15.116:0/3129916463 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce00035f0 con 0x7f7cec107bf0 2024-03-04T07:45:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cea000640 1 -- 172.21.15.116:0/3129916463 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000ce60 con 0x7f7cec107bf0 2024-03-04T07:45:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cec1a5df0 con 0x7f7cec107bf0 2024-03-04T07:45:28.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.003+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec1a6200 con 0x7f7cec107bf0 2024-03-04T07:45:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.006+0000 7f7cea000640 1 -- 172.21.15.116:0/3129916463 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f7ce00037a0 con 0x7f7cec107bf0 2024-03-04T07:45:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 Processor -- start 2024-03-04T07:45:28.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 -- start start 2024-03-04T07:45:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e05292fb50 0x55e05292ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e052930540 0x55e052935040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0529356b0 0x55e052937b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05285d090 con 0x55e0529356b0 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e052894d30 con 0x55e05292fb50 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.007+0000 7f7cf7262740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05283da30 con 0x55e052930540 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e05292fb50 0x55e05292ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e05292fb50 0x55e05292ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52286/0 (socket says 172.21.15.116:52286) 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 -- 172.21.15.116:0/1973429419 learned_addr learned my addr 172.21.15.116:0/1973429419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf1c00640 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e052930540 0x55e052935040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf1200640 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0529356b0 0x55e052937b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 -- 172.21.15.116:0/1973429419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e052930540 msgr2=0x55e052935040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e052930540 0x55e052935040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 -- 172.21.15.116:0/1973429419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0529356b0 msgr2=0x55e052937b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0529356b0 0x55e052937b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 -- 172.21.15.116:0/1973429419 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0526737d0 con 0x55e05292fb50 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf1c00640 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e052930540 0x55e052935040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:28.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cea000640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7cd40a1e00 0x7f7cd40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e05292fb50 0x55e05292ff70 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fd0 tx=0x7f7ce40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7158cb1533982f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cea000640 1 -- 172.21.15.116:0/3129916463 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7ce00ce050 con 0x7f7cec107bf0 2024-03-04T07:45:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.008+0000 7f7cf1c00640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7cd40a1e00 0x7f7cd40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.009+0000 7f7cdea00640 1 -- 172.21.15.116:0/1973429419 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4004d20 con 0x55e05292fb50 2024-03-04T07:45:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.009+0000 7f7cdea00640 1 -- 172.21.15.116:0/1973429419 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce4004050 con 0x55e05292fb50 2024-03-04T07:45:28.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.009+0000 7f7cdea00640 1 -- 172.21.15.116:0/1973429419 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4005260 con 0x55e05292fb50 2024-03-04T07:45:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.009+0000 7f7cf7262740 1 -- 172.21.15.116:0/1973429419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e05292fb50 msgr2=0x55e05292ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.009+0000 7f7cf1c00640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7cd40a1e00 0x7f7cd40a42c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00cbb50 tx=0x7f7ce00d1ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.009+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e05292fb50 0x55e05292ff70 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fd0 tx=0x7f7ce40118d0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.009+0000 7f7cf7262740 1 -- 172.21.15.116:0/1973429419 shutdown_connections 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0529356b0 0x55e052937b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e052930540 0x55e052935040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1973429419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e05292fb50 0x55e05292ff70 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 -- 172.21.15.116:0/1973429419 >> 172.21.15.116:0/1973429419 conn(0x55e05292b0c0 msgr2=0x55e05292b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 -- 172.21.15.116:0/1973429419 shutdown_connections 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 -- 172.21.15.116:0/1973429419 wait complete. 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 -- start start 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.010+0000 7f7cf7262740 1 Processor -- start 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e05293a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052941480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf7262740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0528330c0 con 0x55e052930540 2024-03-04T07:45:28.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf7262740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e052890dd0 con 0x55e0529356b0 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf7262740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e052938660 con 0x55e05292fb50 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf7262740 10 client.0 _ll_register_callbacks cb 0x55e052850860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052941480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e05293a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052941480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52290/0 (socket says 172.21.15.116:52290) 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf2600640 1 -- 172.21.15.116:0/3859078818 learned_addr learned my addr 172.21.15.116:0/3859078818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf2600640 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf1200640 1 -- 172.21.15.116:0/3859078818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 msgr2=0x55e052939f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf1200640 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf1200640 1 -- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 msgr2=0x55e052941480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.011+0000 7f7cf1200640 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052941480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cf1200640 1 -- 172.21.15.116:0/3859078818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e0525a93f0 con 0x55e052930540 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cf2600640 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052941480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:28.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cf1200640 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e05293a4b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cd802c7b0 tx=0x7f7cd8004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f4bc54a3200e28 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cd3400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:45:28.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cd3400640 1 -- 172.21.15.116:0/3859078818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8005d50 con 0x55e052930540 2024-03-04T07:45:28.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cd3400640 1 -- 172.21.15.116:0/3859078818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7cd80046c0 con 0x55e052930540 2024-03-04T07:45:28.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ce4004050 con 0x55e052930540 2024-03-04T07:45:28.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cd3400640 1 -- 172.21.15.116:0/3859078818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cd80048c0 con 0x55e052930540 2024-03-04T07:45:28.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cf7262740 10 client.16284 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:28.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.012+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e0525690f0 con 0x55e052930540 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cd3400640 1 -- 172.21.15.116:0/3859078818 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce4004050 con 0x55e052930540 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cd3400640 1 -- 172.21.15.116:0/3859078818 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7cd80035f0 con 0x55e052930540 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cf7262740 10 client.16284 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cf7262740 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cf7262740 20 client.16284 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cf7262740 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e052945a00 0x55e052947e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cf7262740 4 client.16284 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0528a2270 con 0x55e052945a00 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.013+0000 7f7cf2600640 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e052945a00 0x55e052947e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.014+0000 7f7cf2600640 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e052945a00 0x55e052947e20 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e44b8a0f257f51a5 server_cookie=7c2c2f4c00fb5795 in_seq=0 out_seq=0 2024-03-04T07:45:28.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.014+0000 7f7cd3400640 10 client.16284 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.014+0000 7f7cd3400640 1 -- 172.21.15.116:0/3859078818 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e0528a2270 con 0x55e052945a00 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.014+0000 7f7cd3400640 10 client.16284 handle_command_reply: tid=0 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 client.16284 shutdown 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 msgr2=0x55e05293a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e05293a4b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cd802c7b0 tx=0x7f7cd8004250 comp rx=0 tx=0).stop 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 shutdown_connections 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e052945a00 0x55e052947e20 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052941480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e05293a4b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 --2- 172.21.15.116:0/3859078818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cd3400640 0 client.16284 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 >> 172.21.15.116:0/3859078818 conn(0x55e05292b0c0 msgr2=0x55e05292a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 shutdown_connections 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 1 -- 172.21.15.116:0/3859078818 wait complete. 2024-03-04T07:45:28.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.015+0000 7f7cf7262740 20 client.16284 trim_cache size 0 max 16384 2024-03-04T07:45:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.032+0000 7f7cf7262740 1 Processor -- start 2024-03-04T07:45:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf7262740 1 -- start start 2024-03-04T07:45:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e052941440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052929ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf7262740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1ad080 con 0x55e052930540 2024-03-04T07:45:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf7262740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1af660 con 0x55e0529356b0 2024-03-04T07:45:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf7262740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1ad680 con 0x55e05292fb50 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052929ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e052941440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052929ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52296/0 (socket says 172.21.15.116:52296) 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf2600640 1 -- 172.21.15.116:0/1301537042 learned_addr learned my addr 172.21.15.116:0/1301537042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.033+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cf1200640 1 -- 172.21.15.116:0/1301537042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 msgr2=0x55e052939590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cf1200640 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cf1200640 1 -- 172.21.15.116:0/1301537042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 msgr2=0x55e052929ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cf1200640 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052929ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cf1200640 1 -- 172.21.15.116:0/1301537042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e05292bb30 con 0x55e052930540 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cf1200640 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e052941440 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8003990 tx=0x7f7cd8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1163612ee1757813 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cde000640 1 -- 172.21.15.116:0/1301537042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8057020 con 0x55e052930540 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cde000640 1 -- 172.21.15.116:0/1301537042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cd8009520 con 0x55e052930540 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.034+0000 7f7cde000640 1 -- 172.21.15.116:0/1301537042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8000e60 con 0x55e052930540 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 -- 172.21.15.116:0/1301537042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 msgr2=0x55e052941440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e052941440 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8003990 tx=0x7f7cd8005fd0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 -- 172.21.15.116:0/1301537042 shutdown_connections 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e0529356b0 0x55e052929ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e052930540 0x55e052941440 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1301537042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e05292fb50 0x55e052939590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 -- 172.21.15.116:0/1301537042 >> 172.21.15.116:0/1301537042 conn(0x55e05292b0c0 msgr2=0x55e052938a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 -- 172.21.15.116:0/1301537042 shutdown_connections 2024-03-04T07:45:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 -- 172.21.15.116:0/1301537042 wait complete. 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.035+0000 7f7cf7262740 1 -- start start 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 1 Processor -- start 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e05292fb50 0x55e05293f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e052930540 0x55e05293f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e0529356b0 0x55e0529408a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1adce0 con 0x55e05292fb50 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1af000 con 0x55e052930540 2024-03-04T07:45:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1ae340 con 0x55e0529356b0 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf7262740 10 client.0 _ll_register_callbacks cb 0x7f7cd802c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e05292fb50 0x55e05293f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf1200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e052930540 0x55e05293f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf1200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e052930540 0x55e05293f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52310/0 (socket says 172.21.15.116:52310) 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf1200640 1 -- 172.21.15.116:0/1553015565 learned_addr learned my addr 172.21.15.116:0/1553015565 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.036+0000 7f7cf2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e05292fb50 0x55e05293f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41106/0 (socket says 172.21.15.116:41106) 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e0529356b0 0x55e0529408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf2600640 1 -- 172.21.15.116:0/1553015565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e0529356b0 msgr2=0x55e0529408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e0529356b0 0x55e0529408a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf2600640 1 -- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e052930540 msgr2=0x55e05293f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e052930540 0x55e05293f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf2600640 1 -- 172.21.15.116:0/1553015565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e052940e90 con 0x55e05292fb50 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf1200640 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e052930540 0x55e05293f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf2600640 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e05292fb50 0x55e05293f350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40096c0 tx=0x7f7cd802c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f42ec98e8749e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cd3400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cd3400640 1 -- 172.21.15.116:0/1553015565 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4004600 con 0x55e05292fb50 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cd3400640 1 -- 172.21.15.116:0/1553015565 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7ce40047a0 con 0x55e05292fb50 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.037+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cd8009520 con 0x55e05292fb50 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cd3400640 1 -- 172.21.15.116:0/1553015565 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ce400f4e0 con 0x55e05292fb50 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 10 client.16290 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e0529a1af0 con 0x55e05292fb50 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cd3400640 1 -- 172.21.15.116:0/1553015565 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cd8009520 con 0x55e05292fb50 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cd3400640 1 -- 172.21.15.116:0/1553015565 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f7ce4025040 con 0x55e05292fb50 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 10 client.16290 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 10 client.16290 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 20 client.16290 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e0529a2300 0x55e0529a4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 4 client.16290 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:45:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e052939d60 con 0x55e0529a2300 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.038+0000 7f7cf1200640 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e0529a2300 0x55e0529a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf1200640 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e0529a2300 0x55e0529a4720 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3520bb9d1ba56e5 server_cookie=d02da9d90b2089c1 in_seq=0 out_seq=0 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cd3400640 10 client.16290 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cd3400640 1 -- 172.21.15.116:0/1553015565 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e052939d60 con 0x55e0529a2300 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cd3400640 10 client.16290 handle_command_reply: tid=0 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 client.16290 shutdown 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e05292fb50 msgr2=0x55e05293f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e05292fb50 0x55e05293f350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40096c0 tx=0x7f7cd802c880 comp rx=0 tx=0).stop 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 shutdown_connections 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e0529a2300 0x55e0529a4720 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e0529356b0 0x55e0529408a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e052930540 0x55e05293f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 --2- 172.21.15.116:0/1553015565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e05292fb50 0x55e05293f350 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cd3400640 0 client.16290 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 >> 172.21.15.116:0/1553015565 conn(0x55e05292b0c0 msgr2=0x55e05292ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 shutdown_connections 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.039+0000 7f7cf7262740 1 -- 172.21.15.116:0/1553015565 wait complete. 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf7262740 20 client.16290 trim_cache size 0 max 16384 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:45:28.045 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:45:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7cd40a1e00 msgr2=0x7f7cd40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7cd40a1e00 0x7f7cd40a42c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00cbb50 tx=0x7f7ce00d1ad0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec107bf0 msgr2=0x7f7cec19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec107bf0 0x7f7cec19dee0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9c0 tx=0x7f7ce000ee80 comp rx=0 tx=0).stop 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 shutdown_connections 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f7cd40a1e00 0x7f7cd40a42c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cec1097a0 0x7f7cec1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cec108de0 0x7f7cec19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 --2- 172.21.15.116:0/3129916463 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7cec107bf0 0x7f7cec19dee0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 >> 172.21.15.116:0/3129916463 conn(0x7f7cec076b90 msgr2=0x7f7cec10c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 shutdown_connections 2024-03-04T07:45:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.040+0000 7f7cf4600640 1 -- 172.21.15.116:0/3129916463 wait complete. 2024-03-04T07:45:28.058 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:45:28.058 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:45:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 -- 172.21.15.116:0/894695066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c107c10 msgr2=0x7f3a4c108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 --2- 172.21.15.116:0/894695066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c107c10 0x7f3a4c108030 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009900 tx=0x7f3a4002bc30 comp rx=0 tx=0).stop 2024-03-04T07:45:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 -- 172.21.15.116:0/894695066 shutdown_connections 2024-03-04T07:45:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 --2- 172.21.15.116:0/894695066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a4c1097c0 0x7f3a4c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 --2- 172.21.15.116:0/894695066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a4c108e00 0x7f3a4c109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 --2- 172.21.15.116:0/894695066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c107c10 0x7f3a4c108030 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 -- 172.21.15.116:0/894695066 >> 172.21.15.116:0/894695066 conn(0x7f3a4c076be0 msgr2=0x7f3a4c079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 -- 172.21.15.116:0/894695066 shutdown_connections 2024-03-04T07:45:28.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.251+0000 7f3a54200640 1 -- 172.21.15.116:0/894695066 wait complete. 2024-03-04T07:45:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 Processor -- start 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 -- start start 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a4c107c10 0x7f3a4c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a4c108e00 0x7f3a4c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c1097c0 0x7f3a4c1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c113890 con 0x7f3a4c107c10 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c113710 con 0x7f3a4c1097c0 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a54200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c113a10 con 0x7f3a4c108e00 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.252+0000 7f3a51800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a4c107c10 0x7f3a4c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a52200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c1097c0 0x7f3a4c1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a51800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a4c107c10 0x7f3a4c19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41122/0 (socket says 172.21.15.116:41122) 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a51800640 1 -- 172.21.15.116:0/2314138231 learned_addr learned my addr 172.21.15.116:0/2314138231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a50e00640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a4c108e00 0x7f3a4c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a52200640 1 -- 172.21.15.116:0/2314138231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a4c108e00 msgr2=0x7f3a4c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a52200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a4c108e00 0x7f3a4c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a52200640 1 -- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a4c107c10 msgr2=0x7f3a4c19dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a52200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a4c107c10 0x7f3a4c19dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a52200640 1 -- 172.21.15.116:0/2314138231 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c1a5b00 con 0x7f3a4c1097c0 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a52200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c1097c0 0x7f3a4c1a5490 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009fd0 tx=0x7f3a4400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a4a000640 1 -- 172.21.15.116:0/2314138231 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4400ce20 con 0x7f3a4c1097c0 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a4a000640 1 -- 172.21.15.116:0/2314138231 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a44005020 con 0x7f3a4c1097c0 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.253+0000 7f3a4a000640 1 -- 172.21.15.116:0/2314138231 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44010720 con 0x7f3a4c1097c0 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.254+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c1a5df0 con 0x7f3a4c1097c0 2024-03-04T07:45:28.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.254+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c1a62b0 con 0x7f3a4c1097c0 2024-03-04T07:45:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.255+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c10ef30 con 0x7f3a4c1097c0 2024-03-04T07:45:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.256+0000 7f3a4a000640 1 -- 172.21.15.116:0/2314138231 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3a44005540 con 0x7f3a4c1097c0 2024-03-04T07:45:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.257+0000 7f3a4a000640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a340a1d80 0x7f3a340a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.258+0000 7f3a4a000640 1 -- 172.21.15.116:0/2314138231 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3a440ce440 con 0x7f3a4c1097c0 2024-03-04T07:45:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.258+0000 7f3a51800640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a340a1d80 0x7f3a340a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.258+0000 7f3a51800640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a340a1d80 0x7f3a340a4240 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3a4002c8e0 tx=0x7f3a40005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.262+0000 7f3a4a000640 1 -- 172.21.15.116:0/2314138231 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3a4409b830 con 0x7f3a4c1097c0 2024-03-04T07:45:28.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.499+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a4c0621d0 con 0x7f3a4c1097c0 2024-03-04T07:45:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.500+0000 7f3a4a000640 1 -- 172.21.15.116:0/2314138231 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f3a4409b130 con 0x7f3a4c1097c0 2024-03-04T07:45:28.502 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:45:28.502 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:45:28.502 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a340a1d80 msgr2=0x7f3a340a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a340a1d80 0x7f3a340a4240 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3a4002c8e0 tx=0x7f3a40005b50 comp rx=0 tx=0).stop 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c1097c0 msgr2=0x7f3a4c1a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c1097c0 0x7f3a4c1a5490 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009fd0 tx=0x7f3a4400ef10 comp rx=0 tx=0).stop 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 shutdown_connections 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a4c1097c0 0x7f3a4c1a5490 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a340a1d80 0x7f3a340a4240 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a4c108e00 0x7f3a4c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 --2- 172.21.15.116:0/2314138231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a4c107c10 0x7f3a4c19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 >> 172.21.15.116:0/2314138231 conn(0x7f3a4c076be0 msgr2=0x7f3a4c10c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 shutdown_connections 2024-03-04T07:45:28.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.503+0000 7f3a54200640 1 -- 172.21.15.116:0/2314138231 wait complete. 2024-03-04T07:45:28.524 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:45:28.524 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:45:28.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 -- 172.21.15.116:0/455290786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd194103080 msgr2=0x7fd194109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 --2- 172.21.15.116:0/455290786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd194103080 0x7fd194109910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009ca0 tx=0x7fd18c02c0f0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 -- 172.21.15.116:0/455290786 shutdown_connections 2024-03-04T07:45:28.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 --2- 172.21.15.116:0/455290786 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd194103080 0x7fd194109910 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 --2- 172.21.15.116:0/455290786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1941026c0 0x7fd194102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 --2- 172.21.15.116:0/455290786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1941014d0 0x7fd1941018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 -- 172.21.15.116:0/455290786 >> 172.21.15.116:0/455290786 conn(0x7fd1940fca40 msgr2=0x7fd1940fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 -- 172.21.15.116:0/455290786 shutdown_connections 2024-03-04T07:45:28.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.739+0000 7fd193e00640 1 -- 172.21.15.116:0/455290786 wait complete. 2024-03-04T07:45:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.740+0000 7fd193e00640 1 Processor -- start 2024-03-04T07:45:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.740+0000 7fd193e00640 1 -- start start 2024-03-04T07:45:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1941014d0 0x7fd194075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1941026c0 0x7fd1940739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd194103080 0x7fd194073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19410d160 con 0x7fd194103080 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19410cfe0 con 0x7fd1941026c0 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19410d2e0 con 0x7fd1941014d0 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd194103080 0x7fd194073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd18a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1941026c0 0x7fd1940739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd18a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1941026c0 0x7fd1940739c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52348/0 (socket says 172.21.15.116:52348) 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd18a000640 1 -- 172.21.15.116:0/2775310374 learned_addr learned my addr 172.21.15.116:0/2775310374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd194103080 0x7fd194073f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41132/0 (socket says 172.21.15.116:41132) 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd192a00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1941014d0 0x7fd194075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193400640 1 -- 172.21.15.116:0/2775310374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1941014d0 msgr2=0x7fd194075380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193400640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1941014d0 0x7fd194075380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193400640 1 -- 172.21.15.116:0/2775310374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1941026c0 msgr2=0x7fd1940739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193400640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1941026c0 0x7fd1940739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd193400640 1 -- 172.21.15.116:0/2775310374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd194074790 con 0x7fd194103080 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd192a00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1941014d0 0x7fd194075380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.741+0000 7fd18a000640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1941026c0 0x7fd1940739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.742+0000 7fd193400640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd194103080 0x7fd194073f00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd18c02c630 tx=0x7fd18c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.742+0000 7fd18be00640 1 -- 172.21.15.116:0/2775310374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c041020 con 0x7fd194103080 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.742+0000 7fd18be00640 1 -- 172.21.15.116:0/2775310374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd18c034cd0 con 0x7fd194103080 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.742+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1941a17e0 con 0x7fd194103080 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.742+0000 7fd18be00640 1 -- 172.21.15.116:0/2775310374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c039040 con 0x7fd194103080 2024-03-04T07:45:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.742+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1941a1d00 con 0x7fd194103080 2024-03-04T07:45:28.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.745+0000 7fd18be00640 1 -- 172.21.15.116:0/2775310374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd18c044020 con 0x7fd194103080 2024-03-04T07:45:28.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 Processor -- start 2024-03-04T07:45:28.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 -- start start 2024-03-04T07:45:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab3be70 0x5560dab3c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab41360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab419d0 0x5560dab43e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560daaa0d60 con 0x5560dab3be70 2024-03-04T07:45:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560daa49cf0 con 0x5560dab3c860 2024-03-04T07:45:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560daa9ce00 con 0x5560dab419d0 2024-03-04T07:45:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd192a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab41360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd18a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab3be70 0x5560dab3c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd192a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab41360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52352/0 (socket says 172.21.15.116:52352) 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.746+0000 7fd192a00640 1 -- 172.21.15.116:0/3551359880 learned_addr learned my addr 172.21.15.116:0/3551359880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd18a000640 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab419d0 0x5560dab43e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd192a00640 1 -- 172.21.15.116:0/3551359880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab419d0 msgr2=0x5560dab43e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd192a00640 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab419d0 0x5560dab43e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd192a00640 1 -- 172.21.15.116:0/3551359880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab3be70 msgr2=0x5560dab3c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd192a00640 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab3be70 0x5560dab3c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd192a00640 1 -- 172.21.15.116:0/3551359880 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560da7b53f0 con 0x5560dab3c860 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd18a000640 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab3be70 0x5560dab3c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd18be00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1780a1d80 0x7fd1780a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.747+0000 7fd192a00640 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab41360 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd184009cf0 tx=0x7fd1840117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97ccd7473362386c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.748+0000 7fd18be00640 1 -- 172.21.15.116:0/2775310374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd18c0efc20 con 0x7fd194103080 2024-03-04T07:45:28.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.748+0000 7fd18a000640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1780a1d80 0x7fd1780a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.748+0000 7fd183e00640 1 -- 172.21.15.116:0/3551359880 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184012450 con 0x5560dab3c860 2024-03-04T07:45:28.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.748+0000 7fd183e00640 1 -- 172.21.15.116:0/3551359880 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd184012ad0 con 0x5560dab3c860 2024-03-04T07:45:28.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.748+0000 7fd183e00640 1 -- 172.21.15.116:0/3551359880 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18400fad0 con 0x5560dab3c860 2024-03-04T07:45:28.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3551359880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 msgr2=0x5560dab41360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd18a000640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1780a1d80 0x7fd1780a4240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5560dab3d0c0 tx=0x7fd17c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab41360 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd184009cf0 tx=0x7fd1840117b0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3551359880 shutdown_connections 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab419d0 0x5560dab43e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab41360 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3551359880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab3be70 0x5560dab3c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3551359880 >> 172.21.15.116:0/3551359880 conn(0x5560dab373e0 msgr2=0x5560dab377e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3551359880 shutdown_connections 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3551359880 wait complete. 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.749+0000 7fd19c5cb740 1 -- start start 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 1 Processor -- start 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab46930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560dab44980 con 0x5560dab419d0 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560daa3f220 con 0x5560dab3c860 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560dab44b00 con 0x5560dab3be70 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd19c5cb740 10 client.0 _ll_register_callbacks cb 0x5560da7b4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd18a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab46930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd18a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab46930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52364/0 (socket says 172.21.15.116:52364) 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd18a000640 1 -- 172.21.15.116:0/3917617066 learned_addr learned my addr 172.21.15.116:0/3917617066 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd193400640 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.750+0000 7fd192a00640 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd18a000640 1 -- 172.21.15.116:0/3917617066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 msgr2=0x5560dab463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd18a000640 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd18a000640 1 -- 172.21.15.116:0/3917617066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 msgr2=0x5560dab4d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd18a000640 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd18a000640 1 -- 172.21.15.116:0/3917617066 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560da87f7d0 con 0x5560dab3c860 2024-03-04T07:45:28.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd18a000640 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab46930 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd17c010b90 tx=0x7fd17c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4890babf096aa06 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd180c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd180c00640 1 -- 172.21.15.116:0/3917617066 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c022910 con 0x5560dab3c860 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd180c00640 1 -- 172.21.15.116:0/3917617066 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd17c01c030 con 0x5560dab3c860 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.751+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd184012ad0 con 0x5560dab3c860 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd180c00640 1 -- 172.21.15.116:0/3917617066 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd17c022ab0 con 0x5560dab3c860 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 10 client.26435 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5560dab50d50 con 0x5560dab3c860 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd180c00640 1 -- 172.21.15.116:0/3917617066 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd184012ad0 con 0x5560dab3c860 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd180c00640 1 -- 172.21.15.116:0/3917617066 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd17c020c50 con 0x5560dab3c860 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 10 client.26435 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 10 client.26435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:28.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 20 client.26435 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dab523a0 0x5560dab54840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 4 client.26435 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560daaae2a0 con 0x5560dab523a0 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.752+0000 7fd192a00640 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dab523a0 0x5560dab54840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.753+0000 7fd192a00640 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dab523a0 0x5560dab54840 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fff3805c171648eb server_cookie=6fd51ef962765b94 in_seq=0 out_seq=0 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.753+0000 7fd180c00640 10 client.26435 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd180c00640 1 -- 172.21.15.116:0/3917617066 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5560daaae2a0 con 0x5560dab523a0 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd180c00640 10 client.26435 handle_command_reply: tid=0 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 client.26435 shutdown 2024-03-04T07:45:28.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 msgr2=0x5560dab46930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab46930 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd17c010b90 tx=0x7fd17c004730 comp rx=0 tx=0).stop 2024-03-04T07:45:28.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 shutdown_connections 2024-03-04T07:45:28.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dab523a0 0x5560dab54840 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab46930 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/3917617066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 >> 172.21.15.116:0/3917617066 conn(0x5560dab373e0 msgr2=0x5560dab36700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd180c00640 0 client.26435 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 shutdown_connections 2024-03-04T07:45:28.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 1 -- 172.21.15.116:0/3917617066 wait complete. 2024-03-04T07:45:28.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.754+0000 7fd19c5cb740 20 client.26435 trim_cache size 0 max 16384 2024-03-04T07:45:28.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.767+0000 7fd19c5cb740 1 Processor -- start 2024-03-04T07:45:28.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd19c5cb740 1 -- start start 2024-03-04T07:45:28.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab3ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab59100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941a89f0 con 0x5560dab419d0 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941aafd0 con 0x5560dab3c860 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941a8ff0 con 0x5560dab3be70 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd18a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd18a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52374/0 (socket says 172.21.15.116:52374) 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd18a000640 1 -- 172.21.15.116:0/2921692025 learned_addr learned my addr 172.21.15.116:0/2921692025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd192a00640 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab3ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd193400640 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab59100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd18a000640 1 -- 172.21.15.116:0/2921692025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 msgr2=0x5560dab3ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd18a000640 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab3ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.768+0000 7fd18a000640 1 -- 172.21.15.116:0/2921692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 msgr2=0x5560dab59100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.769+0000 7fd18a000640 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab59100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.769+0000 7fd18a000640 1 -- 172.21.15.116:0/2921692025 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560dab38150 con 0x5560dab3c860 2024-03-04T07:45:28.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.769+0000 7fd18a000640 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4d930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd17c02a490 tx=0x7fd17c010f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=640b3c7bab345d10 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.769+0000 7fd183400640 1 -- 172.21.15.116:0/2921692025 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c039020 con 0x5560dab3c860 2024-03-04T07:45:28.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.769+0000 7fd183400640 1 -- 172.21.15.116:0/2921692025 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd17c013230 con 0x5560dab3c860 2024-03-04T07:45:28.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.769+0000 7fd183400640 1 -- 172.21.15.116:0/2921692025 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c0207f0 con 0x5560dab3c860 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 -- 172.21.15.116:0/2921692025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 msgr2=0x5560dab4d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4d930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd17c02a490 tx=0x7fd17c010f90 comp rx=0 tx=0).stop 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 -- 172.21.15.116:0/2921692025 shutdown_connections 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab59100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4d930 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/2921692025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab3ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 -- 172.21.15.116:0/2921692025 >> 172.21.15.116:0/2921692025 conn(0x5560dab373e0 msgr2=0x5560dab3a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 -- 172.21.15.116:0/2921692025 shutdown_connections 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 -- 172.21.15.116:0/2921692025 wait complete. 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 -- start start 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.770+0000 7fd19c5cb740 1 Processor -- start 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab4b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd19c5cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941a9650 con 0x5560dab419d0 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941aa970 con 0x5560dab3c860 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd19c5cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941a9cb0 con 0x5560dab3be70 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd19c5cb740 10 client.0 _ll_register_callbacks cb 0x7fd17c010810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd192a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab4b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd18a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd192a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab4b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43420/0 (socket says 172.21.15.116:43420) 2024-03-04T07:45:28.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd192a00640 1 -- 172.21.15.116:0/4011449715 learned_addr learned my addr 172.21.15.116:0/4011449715 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd193400640 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd18a000640 1 -- 172.21.15.116:0/4011449715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 msgr2=0x5560dab4b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd18a000640 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab4b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd18a000640 1 -- 172.21.15.116:0/4011449715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 msgr2=0x5560dab4ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd18a000640 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.771+0000 7fd18a000640 1 -- 172.21.15.116:0/4011449715 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560dab4d080 con 0x5560dab3c860 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd18a000640 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4bad0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd17c02a510 tx=0x7fd17c0207a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce70391919b1d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd180c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd180c00640 1 -- 172.21.15.116:0/4011449715 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c039070 con 0x5560dab3c860 2024-03-04T07:45:28.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd17c013230 con 0x5560dab3c860 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd180c00640 1 -- 172.21.15.116:0/4011449715 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd17c020ca0 con 0x5560dab3c860 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd180c00640 1 -- 172.21.15.116:0/4011449715 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd17c004020 con 0x5560dab3c860 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd19c5cb740 10 client.26441 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5560dabae1e0 con 0x5560dab3c860 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd180c00640 1 -- 172.21.15.116:0/4011449715 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd17c013230 con 0x5560dab3c860 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.772+0000 7fd180c00640 1 -- 172.21.15.116:0/4011449715 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd17c003ad0 con 0x5560dab3c860 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd19c5cb740 10 client.26441 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd19c5cb740 10 client.26441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd19c5cb740 20 client.26441 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dabae9f0 0x5560dabb0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd19c5cb740 4 client.26441 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:45:28.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5560daaa85f0 con 0x5560dabae9f0 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd192a00640 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dabae9f0 0x5560dabb0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd192a00640 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dabae9f0 0x5560dabb0e10 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3962d04fa729e54a server_cookie=371ce48e374fb886 in_seq=0 out_seq=0 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd180c00640 10 client.26441 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd180c00640 1 -- 172.21.15.116:0/4011449715 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5560daaa85f0 con 0x5560dabae9f0 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd180c00640 10 client.26441 handle_command_reply: tid=0 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.773+0000 7fd19c5cb740 1 client.26441 shutdown 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 msgr2=0x5560dab4bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4bad0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd17c02a510 tx=0x7fd17c0207a0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 shutdown_connections 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5560dabae9f0 0x5560dabb0e10 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560dab419d0 0x5560dab4ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5560dab3c860 0x5560dab4bad0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 --2- 172.21.15.116:0/4011449715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560dab3be70 0x5560dab4b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd180c00640 0 client.26441 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:28.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 >> 172.21.15.116:0/4011449715 conn(0x5560dab373e0 msgr2=0x5560dab3aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 shutdown_connections 2024-03-04T07:45:28.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 1 -- 172.21.15.116:0/4011449715 wait complete. 2024-03-04T07:45:28.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd19c5cb740 20 client.26441 trim_cache size 0 max 16384 2024-03-04T07:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:45:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1780a1d80 msgr2=0x7fd1780a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd193e00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1780a1d80 0x7fd1780a4240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5560dab3d0c0 tx=0x7fd17c005f90 comp rx=0 tx=0).stop 2024-03-04T07:45:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd194103080 msgr2=0x7fd194073f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:28.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.774+0000 7fd193e00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd194103080 0x7fd194073f00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd18c02c630 tx=0x7fd18c032f20 comp rx=0 tx=0).stop 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 shutdown_connections 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd194103080 0x7fd194073f00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1941026c0 0x7fd1940739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd1780a1d80 0x7fd1780a4240 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 --2- 172.21.15.116:0/2775310374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd1941014d0 0x7fd194075380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 >> 172.21.15.116:0/2775310374 conn(0x7fd1940fca40 msgr2=0x7fd194105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 shutdown_connections 2024-03-04T07:45:28.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:28.775+0000 7fd193e00640 1 -- 172.21.15.116:0/2775310374 wait complete. 2024-03-04T07:45:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:28 smithi133 ceph-mon[36505]: pgmap v542: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:45:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:28 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2314138231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:45:29.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:28 smithi116 ceph-mon[32150]: pgmap v542: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:45:29.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:28 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2314138231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:45:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:28 smithi152 ceph-mon[37154]: pgmap v542: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:45:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:28 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2314138231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:45:29.793 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:45:29.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:45:29.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 -- 172.21.15.116:0/714856075 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff860107bf0 msgr2=0x7ff860108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 --2- 172.21.15.116:0/714856075 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff860107bf0 0x7ff860108010 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff850009860 tx=0x7ff85002bb70 comp rx=0 tx=0).stop 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 -- 172.21.15.116:0/714856075 shutdown_connections 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 --2- 172.21.15.116:0/714856075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8601097a0 0x7ff860110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 --2- 172.21.15.116:0/714856075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff860108de0 0x7ff860109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 --2- 172.21.15.116:0/714856075 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff860107bf0 0x7ff860108010 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 -- 172.21.15.116:0/714856075 >> 172.21.15.116:0/714856075 conn(0x7ff860076b90 msgr2=0x7ff860078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 -- 172.21.15.116:0/714856075 shutdown_connections 2024-03-04T07:45:30.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.019+0000 7ff866e00640 1 -- 172.21.15.116:0/714856075 wait complete. 2024-03-04T07:45:30.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.020+0000 7ff866e00640 1 Processor -- start 2024-03-04T07:45:30.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.020+0000 7ff866e00640 1 -- start start 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff866e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff860107bf0 0x7ff86019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff866e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff860108de0 0x7ff86019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff866e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8601097a0 0x7ff8601a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff866e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff860113870 con 0x7ff860108de0 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff866e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601136f0 con 0x7ff8601097a0 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff866e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601139f0 con 0x7ff860107bf0 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff860108de0 0x7ff86019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff860107bf0 0x7ff86019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff864e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8601097a0 0x7ff8601a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff860108de0 0x7ff86019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41206/0 (socket says 172.21.15.116:41206) 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85f400640 1 -- 172.21.15.116:0/633458998 learned_addr learned my addr 172.21.15.116:0/633458998 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85fe00640 1 -- 172.21.15.116:0/633458998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8601097a0 msgr2=0x7ff8601a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85fe00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8601097a0 0x7ff8601a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85fe00640 1 -- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff860108de0 msgr2=0x7ff86019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85fe00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff860108de0 0x7ff86019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff85fe00640 1 -- 172.21.15.116:0/633458998 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8601a5ad0 con 0x7ff860107bf0 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.021+0000 7ff864e00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8601097a0 0x7ff8601a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.022+0000 7ff85fe00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff860107bf0 0x7ff86019deb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff850009780 tx=0x7ff850032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.022+0000 7ff85cc00640 1 -- 172.21.15.116:0/633458998 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850041020 con 0x7ff860107bf0 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.022+0000 7ff85cc00640 1 -- 172.21.15.116:0/633458998 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff85002c9f0 con 0x7ff860107bf0 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.022+0000 7ff85cc00640 1 -- 172.21.15.116:0/633458998 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850039040 con 0x7ff860107bf0 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.022+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8601a5d60 con 0x7ff860107bf0 2024-03-04T07:45:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.022+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8601a61d0 con 0x7ff860107bf0 2024-03-04T07:45:30.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.025+0000 7ff85cc00640 1 -- 172.21.15.116:0/633458998 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff850044020 con 0x7ff860107bf0 2024-03-04T07:45:30.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.025+0000 7ff869a38740 1 Processor -- start 2024-03-04T07:45:30.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff869a38740 1 -- start start 2024-03-04T07:45:30.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff869a38740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2b0d0 0x556b8ec2b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff869a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec305c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff869a38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec30c30 0x556b8ec330c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff869a38740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8eb3ee20 con 0x556b8ec30c30 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff869a38740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8eb90a20 con 0x556b8ec2b0d0 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff869a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8eb39b00 con 0x556b8ec2bac0 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff864e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2b0d0 0x556b8ec2b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff864e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2b0d0 0x556b8ec2b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52402/0 (socket says 172.21.15.116:52402) 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff85f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec30c30 0x556b8ec330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff864e00640 1 -- 172.21.15.116:0/2317123274 learned_addr learned my addr 172.21.15.116:0/2317123274 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.026+0000 7ff85fe00640 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec305c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff864e00640 1 -- 172.21.15.116:0/2317123274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 msgr2=0x556b8ec305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff864e00640 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec305c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff864e00640 1 -- 172.21.15.116:0/2317123274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec30c30 msgr2=0x556b8ec330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff864e00640 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec30c30 0x556b8ec330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff864e00640 1 -- 172.21.15.116:0/2317123274 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b8eb4ee10 con 0x556b8ec2b0d0 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff85cc00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff8440a1e00 0x7ff8440a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff864e00640 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2b0d0 0x556b8ec2b4f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff8580117e0 tx=0x7ff858011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b4f19543ed4a220 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff855600640 1 -- 172.21.15.116:0/2317123274 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858011310 con 0x556b8ec2b0d0 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff85cc00640 1 -- 172.21.15.116:0/633458998 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff8500ef4f0 con 0x7ff860107bf0 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff855600640 1 -- 172.21.15.116:0/2317123274 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff858003e50 con 0x556b8ec2b0d0 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff855600640 1 -- 172.21.15.116:0/2317123274 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858004600 con 0x556b8ec2b0d0 2024-03-04T07:45:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.027+0000 7ff85fe00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff8440a1e00 0x7ff8440a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 -- 172.21.15.116:0/2317123274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2b0d0 msgr2=0x556b8ec2b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2b0d0 0x556b8ec2b4f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff8580117e0 tx=0x7ff858011ca0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff85fe00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff8440a1e00 0x7ff8440a42c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff85002c180 tx=0x7ff8500f2f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 -- 172.21.15.116:0/2317123274 shutdown_connections 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec30c30 0x556b8ec330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec305c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 --2- 172.21.15.116:0/2317123274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2b0d0 0x556b8ec2b4f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 -- 172.21.15.116:0/2317123274 >> 172.21.15.116:0/2317123274 conn(0x556b8ec26640 msgr2=0x556b8ec26a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 -- 172.21.15.116:0/2317123274 shutdown_connections 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.028+0000 7ff869a38740 1 -- 172.21.15.116:0/2317123274 wait complete. 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 -- start start 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 Processor -- start 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec35a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec30c30 0x556b8ec3ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8eb4cf40 con 0x556b8ec2b0d0 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8eb8cac0 con 0x556b8ec30c30 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b8ec33be0 con 0x556b8ec2bac0 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.029+0000 7ff869a38740 10 client.0 _ll_register_callbacks cb 0x556b8eb8baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec35a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff864e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec35a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43454/0 (socket says 172.21.15.116:43454) 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 -- 172.21.15.116:0/1036694143 learned_addr learned my addr 172.21.15.116:0/1036694143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff864e00640 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec30c30 0x556b8ec3ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 -- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec30c30 msgr2=0x556b8ec3ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec30c30 0x556b8ec3ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 -- 172.21.15.116:0/1036694143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 msgr2=0x556b8ec354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec354f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 -- 172.21.15.116:0/1036694143 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b8e96d7e0 con 0x556b8ec2bac0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff864e00640 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff85f400640 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec35a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff848011830 tx=0x7ff848011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=806d5cd2161e99fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.030+0000 7ff84e000640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.031+0000 7ff84e000640 1 -- 172.21.15.116:0/1036694143 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848004240 con 0x556b8ec2bac0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.031+0000 7ff84e000640 1 -- 172.21.15.116:0/1036694143 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff8480043e0 con 0x556b8ec2bac0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.031+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff858003e50 con 0x556b8ec2bac0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.031+0000 7ff84e000640 1 -- 172.21.15.116:0/1036694143 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff84800f8c0 con 0x556b8ec2bac0 2024-03-04T07:45:30.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.031+0000 7ff869a38740 10 client.26377 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.031+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556b8e9cdf70 con 0x556b8ec2bac0 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.031+0000 7ff84e000640 1 -- 172.21.15.116:0/1036694143 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff858003e50 con 0x556b8ec2bac0 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff84e000640 1 -- 172.21.15.116:0/1036694143 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff848019030 con 0x556b8ec2bac0 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff869a38740 10 client.26377 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff869a38740 10 client.26377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff869a38740 20 client.26377 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff869a38740 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec41500 0x556b8ec43920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff869a38740 4 client.26377 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b8eb9df60 con 0x556b8ec41500 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff864e00640 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec41500 0x556b8ec43920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff864e00640 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec41500 0x556b8ec43920 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81e05d245a86a92b server_cookie=9ee2e6d909ba75e5 in_seq=0 out_seq=0 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.032+0000 7ff84e000640 10 client.26377 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.033+0000 7ff84e000640 1 -- 172.21.15.116:0/1036694143 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556b8eb9df60 con 0x556b8ec41500 2024-03-04T07:45:30.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.033+0000 7ff84e000640 10 client.26377 handle_command_reply: tid=0 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.033+0000 7ff869a38740 1 client.26377 shutdown 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 msgr2=0x556b8ec35a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec35a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff848011830 tx=0x7ff848011cf0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 shutdown_connections 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec41500 0x556b8ec43920 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec30c30 0x556b8ec3ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec2bac0 0x556b8ec35a30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 --2- 172.21.15.116:0/1036694143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff84e000640 0 client.26377 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 >> 172.21.15.116:0/1036694143 conn(0x556b8ec26640 msgr2=0x556b8ec27890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 shutdown_connections 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 1 -- 172.21.15.116:0/1036694143 wait complete. 2024-03-04T07:45:30.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.034+0000 7ff869a38740 20 client.26377 trim_cache size 0 max 16384 2024-03-04T07:45:30.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.052+0000 7ff869a38740 1 Processor -- start 2024-03-04T07:45:30.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.052+0000 7ff869a38740 1 -- start start 2024-03-04T07:45:30.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff869a38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec25a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff869a38740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff869a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec29fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff869a38740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601acea0 con 0x556b8ec2b0d0 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff869a38740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601af480 con 0x556b8ec2bac0 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff869a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601ad4a0 con 0x556b8ec30c30 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52426/0 (socket says 172.21.15.116:52426) 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 -- 172.21.15.116:0/822841923 learned_addr learned my addr 172.21.15.116:0/822841923 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff864e00640 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec25a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 -- 172.21.15.116:0/822841923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 msgr2=0x556b8ec29fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec29fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 -- 172.21.15.116:0/822841923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 msgr2=0x556b8ec25a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec25a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff85f400640 1 -- 172.21.15.116:0/822841923 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b8e865170 con 0x556b8ec2bac0 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.053+0000 7ff864e00640 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec25a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:45:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff85f400640 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3ca30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff848011800 tx=0x7ff848004e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=999e22b9537d5f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff854c00640 1 -- 172.21.15.116:0/822841923 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848032070 con 0x556b8ec2bac0 2024-03-04T07:45:30.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff854c00640 1 -- 172.21.15.116:0/822841923 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff848011040 con 0x556b8ec2bac0 2024-03-04T07:45:30.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff854c00640 1 -- 172.21.15.116:0/822841923 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848029040 con 0x556b8ec2bac0 2024-03-04T07:45:30.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff869a38740 1 -- 172.21.15.116:0/822841923 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 msgr2=0x556b8ec3ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff869a38740 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3ca30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff848011800 tx=0x7ff848004e50 comp rx=0 tx=0).stop 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff869a38740 1 -- 172.21.15.116:0/822841923 shutdown_connections 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff869a38740 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec29fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff869a38740 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3ca30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff869a38740 1 --2- 172.21.15.116:0/822841923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec25a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.054+0000 7ff869a38740 1 -- 172.21.15.116:0/822841923 >> 172.21.15.116:0/822841923 conn(0x556b8ec26640 msgr2=0x556b8ec298a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.055+0000 7ff869a38740 1 -- 172.21.15.116:0/822841923 shutdown_connections 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.055+0000 7ff869a38740 1 -- 172.21.15.116:0/822841923 wait complete. 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.055+0000 7ff869a38740 1 -- start start 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.055+0000 7ff869a38740 1 Processor -- start 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.055+0000 7ff869a38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec3abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.055+0000 7ff869a38740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff869a38740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec3bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff869a38740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601adb00 con 0x556b8ec2b0d0 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff869a38740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601aee20 con 0x556b8ec2bac0 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff869a38740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601ae160 con 0x556b8ec30c30 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff869a38740 10 client.0 _ll_register_callbacks cb 0x7ff848009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff864e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec3abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff864e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec3abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41240/0 (socket says 172.21.15.116:41240) 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff85f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff864e00640 1 -- 172.21.15.116:0/1788392342 learned_addr learned my addr 172.21.15.116:0/1788392342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff864e00640 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec3bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff85f400640 1 -- 172.21.15.116:0/1788392342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 msgr2=0x556b8ec3bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff85f400640 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec3bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff85f400640 1 -- 172.21.15.116:0/1788392342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 msgr2=0x556b8ec3abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff85f400640 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec3abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff85f400640 1 -- 172.21.15.116:0/1788392342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b8ec3c660 con 0x556b8ec2bac0 2024-03-04T07:45:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff864e00640 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec3bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff85f400640 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3b120 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff848009eb0 tx=0x7ff848004ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c542e71a43f405 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.056+0000 7ff84e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff84e000640 1 -- 172.21.15.116:0/1788392342 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848032040 con 0x556b8ec2bac0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff84e000640 1 -- 172.21.15.116:0/1788392342 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff848011330 con 0x556b8ec2bac0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff848011040 con 0x556b8ec2bac0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff84e000640 1 -- 172.21.15.116:0/1788392342 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff848029040 con 0x556b8ec2bac0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 10 client.26459 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556b8ec9d770 con 0x556b8ec2bac0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff84e000640 1 -- 172.21.15.116:0/1788392342 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff848011040 con 0x556b8ec2bac0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff84e000640 1 -- 172.21.15.116:0/1788392342 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff848035070 con 0x556b8ec2bac0 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 10 client.26459 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 10 client.26459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 20 client.26459 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec9e010 0x556b8eca0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 4 client.26459 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:45:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.057+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556b8eb982b0 con 0x556b8ec9e010 2024-03-04T07:45:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.058+0000 7ff864e00640 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec9e010 0x556b8eca0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.058+0000 7ff864e00640 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec9e010 0x556b8eca0430 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3f972b70bd289e1 server_cookie=817b596b8c6c2bef in_seq=0 out_seq=0 2024-03-04T07:45:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.058+0000 7ff84e000640 10 client.26459 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.059+0000 7ff84e000640 1 -- 172.21.15.116:0/1788392342 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556b8eb982b0 con 0x556b8ec9e010 2024-03-04T07:45:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.059+0000 7ff84e000640 10 client.26459 handle_command_reply: tid=0 2024-03-04T07:45:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.059+0000 7ff869a38740 1 client.26459 shutdown 2024-03-04T07:45:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.059+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 msgr2=0x556b8ec3b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.059+0000 7ff869a38740 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3b120 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff848009eb0 tx=0x7ff848004ca0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 shutdown_connections 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556b8ec9e010 0x556b8eca0430 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b8ec30c30 0x556b8ec3bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556b8ec2bac0 0x556b8ec3b120 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff84e000640 0 client.26459 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 --2- 172.21.15.116:0/1788392342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b8ec2b0d0 0x556b8ec3abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 >> 172.21.15.116:0/1788392342 conn(0x556b8ec26640 msgr2=0x556b8ec2a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 shutdown_connections 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 1 -- 172.21.15.116:0/1788392342 wait complete. 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.060+0000 7ff869a38740 20 client.26459 trim_cache size 0 max 16384 2024-03-04T07:45:30.066 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:45:30.067 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:45:30.067 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "782faa2f-6ae3-4219-bcae-bbd88166d01e", 2024-03-04T07:45:30.067 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:45:30.067 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:45:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff8440a1e00 msgr2=0x7ff8440a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff8440a1e00 0x7ff8440a42c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff85002c180 tx=0x7ff8500f2f30 comp rx=0 tx=0).stop 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff860107bf0 msgr2=0x7ff86019deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff860107bf0 0x7ff86019deb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff850009780 tx=0x7ff850032430 comp rx=0 tx=0).stop 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 shutdown_connections 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff8601097a0 0x7ff8601a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff860108de0 0x7ff86019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff8440a1e00 0x7ff8440a42c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 --2- 172.21.15.116:0/633458998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff860107bf0 0x7ff86019deb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 >> 172.21.15.116:0/633458998 conn(0x7ff860076b90 msgr2=0x7ff86010c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 shutdown_connections 2024-03-04T07:45:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.061+0000 7ff866e00640 1 -- 172.21.15.116:0/633458998 wait complete. 2024-03-04T07:45:30.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:45:30.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:45:30.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 -- 172.21.15.116:0/3057900865 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac1053c0 msgr2=0x7f3dac10bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 --2- 172.21.15.116:0/3057900865 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac1053c0 0x7f3dac10bc50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009ca0 tx=0x7f3da402c100 comp rx=0 tx=0).stop 2024-03-04T07:45:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 -- 172.21.15.116:0/3057900865 shutdown_connections 2024-03-04T07:45:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 --2- 172.21.15.116:0/3057900865 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac1053c0 0x7f3dac10bc50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 --2- 172.21.15.116:0/3057900865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dac104a00 0x7f3dac104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 --2- 172.21.15.116:0/3057900865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac103810 0x7f3dac103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 -- 172.21.15.116:0/3057900865 >> 172.21.15.116:0/3057900865 conn(0x7f3dac0fed80 msgr2=0x7f3dac1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 -- 172.21.15.116:0/3057900865 shutdown_connections 2024-03-04T07:45:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.278+0000 7f3db1400640 1 -- 172.21.15.116:0/3057900865 wait complete. 2024-03-04T07:45:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.279+0000 7f3db1400640 1 Processor -- start 2024-03-04T07:45:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.279+0000 7f3db1400640 1 -- start start 2024-03-04T07:45:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3db1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac103810 0x7f3dac199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3db1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac104a00 0x7f3dac199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3db1400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dac1053c0 0x7f3dac1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3db1400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac10f490 con 0x7f3dac104a00 2024-03-04T07:45:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3db1400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac10f310 con 0x7f3dac103810 2024-03-04T07:45:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3db1400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac10f610 con 0x7f3dac1053c0 2024-03-04T07:45:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac103810 0x7f3dac199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac103810 0x7f3dac199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52456/0 (socket says 172.21.15.116:52456) 2024-03-04T07:45:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 -- 172.21.15.116:0/2302059173 learned_addr learned my addr 172.21.15.116:0/2302059173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3da9e00640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac104a00 0x7f3dac199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3dab200640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dac1053c0 0x7f3dac1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 -- 172.21.15.116:0/2302059173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dac1053c0 msgr2=0x7f3dac1a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dac1053c0 0x7f3dac1a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 -- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac104a00 msgr2=0x7f3dac199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac104a00 0x7f3dac199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3daa800640 1 -- 172.21.15.116:0/2302059173 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac1a16c0 con 0x7f3dac103810 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.280+0000 7f3da9e00640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac104a00 0x7f3dac199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.281+0000 7f3daa800640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac103810 0x7f3dac199aa0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00e9c0 tx=0x7f3d9c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.281+0000 7f3da3400640 1 -- 172.21.15.116:0/2302059173 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00cd80 con 0x7f3dac103810 2024-03-04T07:45:30.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.281+0000 7f3da3400640 1 -- 172.21.15.116:0/2302059173 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d9c0035f0 con 0x7f3dac103810 2024-03-04T07:45:30.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.281+0000 7f3da3400640 1 -- 172.21.15.116:0/2302059173 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c010690 con 0x7f3dac103810 2024-03-04T07:45:30.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.281+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dac1a19b0 con 0x7f3dac103810 2024-03-04T07:45:30.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.281+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dac1a1eb0 con 0x7f3dac103810 2024-03-04T07:45:30.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.285+0000 7f3db3ff3740 1 Processor -- start 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3db3ff3740 1 -- start start 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3da3400640 1 -- 172.21.15.116:0/2302059173 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3d9c0037a0 con 0x7f3dac103810 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f96ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f974040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f976b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56054f89c090 con 0x56054f9746b0 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56054f8d3d30 con 0x56054f96f540 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56054f87ca30 con 0x56054f96eb50 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3dab200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f96ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3daa800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f974040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3dab200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f96ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43512/0 (socket says 172.21.15.116:43512) 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.286+0000 7f3dab200640 1 -- 172.21.15.116:0/2057067656 learned_addr learned my addr 172.21.15.116:0/2057067656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3da9e00640 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f976b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3dab200640 1 -- 172.21.15.116:0/2057067656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 msgr2=0x56054f974040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3dab200640 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f974040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3dab200640 1 -- 172.21.15.116:0/2057067656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 msgr2=0x56054f976b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3dab200640 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f976b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3dab200640 1 -- 172.21.15.116:0/2057067656 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56054f6b27d0 con 0x56054f96eb50 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3dab200640 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f96ef70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3da402c710 tx=0x7f3da40323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69dfc90d9e93bb3f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.287+0000 7f3d97400640 1 -- 172.21.15.116:0/2057067656 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4043070 con 0x56054f96eb50 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.288+0000 7f3d97400640 1 -- 172.21.15.116:0/2057067656 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3da4032c90 con 0x56054f96eb50 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.288+0000 7f3d97400640 1 -- 172.21.15.116:0/2057067656 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4039040 con 0x56054f96eb50 2024-03-04T07:45:30.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.288+0000 7f3da3400640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3d900a1e00 0x7f3d900a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.288+0000 7f3da3400640 1 -- 172.21.15.116:0/2302059173 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3d9c0ce1a0 con 0x7f3dac103810 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.288+0000 7f3daa800640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3d900a1e00 0x7f3d900a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.288+0000 7f3db3ff3740 1 -- 172.21.15.116:0/2057067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 msgr2=0x56054f96ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.288+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f96ef70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3da402c710 tx=0x7f3da40323d0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 -- 172.21.15.116:0/2057067656 shutdown_connections 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f976b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f974040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/2057067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f96ef70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 -- 172.21.15.116:0/2057067656 >> 172.21.15.116:0/2057067656 conn(0x56054f96a0c0 msgr2=0x56054f96a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 -- 172.21.15.116:0/2057067656 shutdown_connections 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 -- 172.21.15.116:0/2057067656 wait complete. 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3daa800640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3d900a1e00 0x7f3d900a42c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c097860 tx=0x7f3d9c0d1be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.289+0000 7f3db3ff3740 1 -- start start 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 1 Processor -- start 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96eb50 0x56054f978ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f979420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f9803f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56054f8720c0 con 0x56054f9746b0 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56054f8cfdd0 con 0x56054f96eb50 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56054f977600 con 0x56054f96f540 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3db3ff3740 10 client.0 _ll_register_callbacks cb 0x56054f88f860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3dab200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96eb50 0x56054f978ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3dab200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f9803f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3da9e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f979420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3dab200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96eb50 0x56054f978ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52474/0 (socket says 172.21.15.116:52474) 2024-03-04T07:45:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.290+0000 7f3dab200640 1 -- 172.21.15.116:0/881299950 learned_addr learned my addr 172.21.15.116:0/881299950 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3da9e00640 1 -- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96eb50 msgr2=0x56054f978ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3da9e00640 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96eb50 0x56054f978ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3da9e00640 1 -- 172.21.15.116:0/881299950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 msgr2=0x56054f9803f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3da9e00640 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f9803f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3da9e00640 1 -- 172.21.15.116:0/881299950 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56054f5e83f0 con 0x56054f96f540 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3dab200640 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96eb50 0x56054f978ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3da9e00640 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f979420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3dac19afb0 tx=0x7f3d98013bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b75578b3bb977b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3d8be00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3d8be00640 1 -- 172.21.15.116:0/881299950 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98020060 con 0x56054f96f540 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3d8be00640 1 -- 172.21.15.116:0/881299950 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3d98011d70 con 0x56054f96f540 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.291+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3da4032c90 con 0x56054f96f540 2024-03-04T07:45:30.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3d8be00640 1 -- 172.21.15.116:0/881299950 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d9800f450 con 0x56054f96f540 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 10 client.26389 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56054f5a80f0 con 0x56054f96f540 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3d8be00640 1 -- 172.21.15.116:0/881299950 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3da4032c90 con 0x56054f96f540 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3d8be00640 1 -- 172.21.15.116:0/881299950 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3d98025070 con 0x56054f96f540 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 10 client.26389 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 10 client.26389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 20 client.26389 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f984a20 0x56054f986e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 4 client.26389 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.292+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56054f8e1270 con 0x56054f984a20 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.293+0000 7f3dab200640 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f984a20 0x56054f986e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.293+0000 7f3dab200640 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f984a20 0x56054f986e60 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd9710303af6a13f server_cookie=91d4b3597f450580 in_seq=0 out_seq=0 2024-03-04T07:45:30.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.293+0000 7f3d8be00640 10 client.26389 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3d8be00640 1 -- 172.21.15.116:0/881299950 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56054f8e1270 con 0x56054f984a20 2024-03-04T07:45:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3d8be00640 10 client.26389 handle_command_reply: tid=0 2024-03-04T07:45:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 client.26389 shutdown 2024-03-04T07:45:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 msgr2=0x56054f979420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f979420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3dac19afb0 tx=0x7f3d98013bf0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 shutdown_connections 2024-03-04T07:45:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f984a20 0x56054f986e60 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f9803f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f979420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/881299950 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96eb50 0x56054f978ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3d8be00640 0 client.26389 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 >> 172.21.15.116:0/881299950 conn(0x56054f96a0c0 msgr2=0x56054f969360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 shutdown_connections 2024-03-04T07:45:30.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.294+0000 7f3db3ff3740 1 -- 172.21.15.116:0/881299950 wait complete. 2024-03-04T07:45:30.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.295+0000 7f3db3ff3740 20 client.26389 trim_cache size 0 max 16384 2024-03-04T07:45:30.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 Processor -- start 2024-03-04T07:45:30.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 -- start start 2024-03-04T07:45:30.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f970aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f9803b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f970fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a8c40 con 0x56054f9746b0 2024-03-04T07:45:30.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1ab220 con 0x56054f96f540 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a9240 con 0x56054f96eb50 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.312+0000 7f3da9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f9803b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f9803b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52484/0 (socket says 172.21.15.116:52484) 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 -- 172.21.15.116:0/245164146 learned_addr learned my addr 172.21.15.116:0/245164146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3dab200640 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f970aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 -- 172.21.15.116:0/245164146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 msgr2=0x56054f970aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f970aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3dab200640 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f970fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 -- 172.21.15.116:0/245164146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 msgr2=0x56054f970fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f970fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 -- 172.21.15.116:0/245164146 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56054f979ed0 con 0x56054f96f540 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3dab200640 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f970fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3da9e00640 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f9803b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3d9801b850 tx=0x7f3d98005670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50979a995ff09623 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.313+0000 7f3d96a00640 1 -- 172.21.15.116:0/245164146 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9802b030 con 0x56054f96f540 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3d96a00640 1 -- 172.21.15.116:0/245164146 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d9802d070 con 0x56054f96f540 2024-03-04T07:45:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3d96a00640 1 -- 172.21.15.116:0/245164146 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800fa10 con 0x56054f96f540 2024-03-04T07:45:30.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 -- 172.21.15.116:0/245164146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 msgr2=0x56054f9803b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f9803b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3d9801b850 tx=0x7f3d98005670 comp rx=0 tx=0).stop 2024-03-04T07:45:30.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 -- 172.21.15.116:0/245164146 shutdown_connections 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f9746b0 0x56054f970fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f96f540 0x56054f9803b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/245164146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96eb50 0x56054f970aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 -- 172.21.15.116:0/245164146 >> 172.21.15.116:0/245164146 conn(0x56054f96a0c0 msgr2=0x56054f978ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 -- 172.21.15.116:0/245164146 shutdown_connections 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.314+0000 7f3db3ff3740 1 -- 172.21.15.116:0/245164146 wait complete. 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 -- start start 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 Processor -- start 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f96eb50 0x56054f97e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f97e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f9746b0 0x56054f97f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a98a0 con 0x56054f96eb50 2024-03-04T07:45:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1aabc0 con 0x56054f9746b0 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac1a9f00 con 0x56054f96f540 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.315+0000 7f3db3ff3740 10 client.0 _ll_register_callbacks cb 0x7f3d98009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f96eb50 0x56054f97e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f96eb50 0x56054f97e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41296/0 (socket says 172.21.15.116:41296) 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 -- 172.21.15.116:0/3808336439 learned_addr learned my addr 172.21.15.116:0/3808336439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3da9e00640 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f97e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f9746b0 0x56054f97f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 -- 172.21.15.116:0/3808336439 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 msgr2=0x56054f97e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f97e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 -- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f9746b0 msgr2=0x56054f97f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f9746b0 0x56054f97f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 -- 172.21.15.116:0/3808336439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56054f97fb30 con 0x56054f96eb50 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3dab200640 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f96eb50 0x56054f97e040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009ca0 tx=0x7f3da4032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd82df2a001b60b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.316+0000 7f3d8be00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3d8be00640 1 -- 172.21.15.116:0/3808336439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4039040 con 0x56054f96eb50 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3d8be00640 1 -- 172.21.15.116:0/3808336439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3da40574b0 con 0x56054f96eb50 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d9802d070 con 0x56054f96eb50 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3d8be00640 1 -- 172.21.15.116:0/3808336439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3da4043040 con 0x56054f96eb50 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 10 client.16347 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56054f9e07d0 con 0x56054f96eb50 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3d8be00640 1 -- 172.21.15.116:0/3808336439 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d9802d070 con 0x56054f96eb50 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3d8be00640 1 -- 172.21.15.116:0/3808336439 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3da4062050 con 0x56054f96eb50 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 10 client.16347 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 10 client.16347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 20 client.16347 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f9e0c70 0x56054f9e3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:30.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 4 client.16347 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.317+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56054f8db5c0 con 0x56054f9e0c70 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.318+0000 7f3da9e00640 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f9e0c70 0x56054f9e3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.318+0000 7f3d8be00640 10 client.16347 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.318+0000 7f3da9e00640 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f9e0c70 0x56054f9e3090 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baa6f2e96293c8f server_cookie=b7a4f6067f9a0d2 in_seq=0 out_seq=0 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.318+0000 7f3d8be00640 1 -- 172.21.15.116:0/3808336439 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56054f8db5c0 con 0x56054f9e0c70 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.318+0000 7f3d8be00640 10 client.16347 handle_command_reply: tid=0 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.318+0000 7f3db3ff3740 1 client.16347 shutdown 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f96eb50 msgr2=0x56054f97e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f96eb50 0x56054f97e040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009ca0 tx=0x7f3da4032590 comp rx=0 tx=0).stop 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 shutdown_connections 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56054f9e0c70 0x56054f9e3090 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56054f9746b0 0x56054f97f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56054f96f540 0x56054f97e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 --2- 172.21.15.116:0/3808336439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56054f96eb50 0x56054f97e040 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 >> 172.21.15.116:0/3808336439 conn(0x56054f96a0c0 msgr2=0x56054f96dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3d8be00640 0 client.16347 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:30.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 shutdown_connections 2024-03-04T07:45:30.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 1 -- 172.21.15.116:0/3808336439 wait complete. 2024-03-04T07:45:30.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.319+0000 7f3db3ff3740 20 client.16347 trim_cache size 0 max 16384 2024-03-04T07:45:30.326 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:45:30.326 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:45:30.326 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:45:30.327 INFO:teuthology.orchestra.run.smithi116.stdout: "782faa2f-6ae3-4219-bcae-bbd88166d01e": { 2024-03-04T07:45:30.327 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:45:30.327 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "782faa2f-6ae3-4219-bcae-bbd88166d01e", 2024-03-04T07:45:30.327 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:45:30.327 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:45:30.327 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:45:30.327 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3d900a1e00 msgr2=0x7f3d900a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3d900a1e00 0x7f3d900a42c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c097860 tx=0x7f3d9c0d1be0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac103810 msgr2=0x7f3dac199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac103810 0x7f3dac199aa0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00e9c0 tx=0x7f3d9c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 shutdown_connections 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3d900a1e00 0x7f3d900a42c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3dac1053c0 0x7f3dac1a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dac104a00 0x7f3dac199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 --2- 172.21.15.116:0/2302059173 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3dac103810 0x7f3dac199aa0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:30.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 >> 172.21.15.116:0/2302059173 conn(0x7f3dac0fed80 msgr2=0x7f3dac107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:30.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 shutdown_connections 2024-03-04T07:45:30.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:45:30.320+0000 7f3db1400640 1 -- 172.21.15.116:0/2302059173 wait complete. 2024-03-04T07:45:30.339 INFO:tasks.cephfs.filesystem:scrub status for tag:782faa2f-6ae3-4219-bcae-bbd88166d01e - {'path': '/', 'tag': '782faa2f-6ae3-4219-bcae-bbd88166d01e', 'options': 'recursive,force'} 2024-03-04T07:45:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:30 smithi133 ceph-mon[36505]: pgmap v543: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:45:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:30 smithi116 ceph-mon[32150]: pgmap v543: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:45:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:30 smithi152 ceph-mon[37154]: pgmap v543: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:45:32.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:31 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:45:32.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:31 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:45:32.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:31 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:45:32.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:31 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:45:32.160 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:31 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:45:32.161 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:31 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:45:32.161 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:31 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:45:32.161 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:31 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:45:32.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:31 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:45:32.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:31 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:45:32.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:31 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:45:32.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:31 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:45:33.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:32 smithi133 ceph-mon[36505]: pgmap v544: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:45:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:32 smithi116 ceph-mon[32150]: pgmap v544: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:45:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:32 smithi152 ceph-mon[37154]: pgmap v544: 97 pgs: 97 active+clean; 3.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:45:34.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:33 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:45:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:33 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:45:34.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:33 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:45:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:34 smithi133 ceph-mon[36505]: pgmap v545: 97 pgs: 97 active+clean; 3.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-04T07:45:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:34 smithi116 ceph-mon[32150]: pgmap v545: 97 pgs: 97 active+clean; 3.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-04T07:45:35.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:34 smithi152 ceph-mon[37154]: pgmap v545: 97 pgs: 97 active+clean; 3.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-04T07:45:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 -- 172.21.15.116:0/2245851367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb0103830 msgr2=0x7fdcb0103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 --2- 172.21.15.116:0/2245851367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb0103830 0x7fdcb0103c50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009900 tx=0x7fdca002bc30 comp rx=0 tx=0).stop 2024-03-04T07:45:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 -- 172.21.15.116:0/2245851367 shutdown_connections 2024-03-04T07:45:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 --2- 172.21.15.116:0/2245851367 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb01053e0 0x7fdcb010bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 --2- 172.21.15.116:0/2245851367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcb0104a20 0x7fdcb0104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 --2- 172.21.15.116:0/2245851367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb0103830 0x7fdcb0103c50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 -- 172.21.15.116:0/2245851367 >> 172.21.15.116:0/2245851367 conn(0x7fdcb00fedc0 msgr2=0x7fdcb01011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 -- 172.21.15.116:0/2245851367 shutdown_connections 2024-03-04T07:45:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.313+0000 7fdcb7800640 1 -- 172.21.15.116:0/2245851367 wait complete. 2024-03-04T07:45:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.314+0000 7fdcb7800640 1 Processor -- start 2024-03-04T07:45:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.314+0000 7fdcb7800640 1 -- start start 2024-03-04T07:45:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb7800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb0103830 0x7fdcb0199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb7800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcb0104a20 0x7fdcb0199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb7800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb01053e0 0x7fdcb01a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb7800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb010f4f0 con 0x7fdcb01053e0 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb7800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb010f370 con 0x7fdcb0103830 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb7800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb010f670 con 0x7fdcb0104a20 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb0103830 0x7fdcb0199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb01053e0 0x7fdcb01a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb0103830 0x7fdcb0199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52514/0 (socket says 172.21.15.116:52514) 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 -- 172.21.15.116:0/3167438658 learned_addr learned my addr 172.21.15.116:0/3167438658 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcafe00640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcb0104a20 0x7fdcb0199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 -- 172.21.15.116:0/3167438658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcb0104a20 msgr2=0x7fdcb0199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcb0104a20 0x7fdcb0199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 -- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb01053e0 msgr2=0x7fdcb01a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb01053e0 0x7fdcb01a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcb4e00640 1 -- 172.21.15.116:0/3167438658 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb01a16c0 con 0x7fdcb0103830 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.315+0000 7fdcafe00640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcb0104a20 0x7fdcb0199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.316+0000 7fdcb4e00640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb0103830 0x7fdcb0199aa0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fdca0000c00 tx=0x7fdca002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.316+0000 7fdcad600640 1 -- 172.21.15.116:0/3167438658 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0043020 con 0x7fdcb0103830 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.316+0000 7fdcad600640 1 -- 172.21.15.116:0/3167438658 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdca00054e0 con 0x7fdcb0103830 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.316+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb01a1950 con 0x7fdcb0103830 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.316+0000 7fdcad600640 1 -- 172.21.15.116:0/3167438658 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0003bd0 con 0x7fdcb0103830 2024-03-04T07:45:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.316+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb01a1ea0 con 0x7fdcb0103830 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.319+0000 7fdcad600640 1 -- 172.21.15.116:0/3167438658 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fdca0046020 con 0x7fdcb0103830 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.319+0000 7fdcba438740 1 Processor -- start 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcba438740 1 -- start start 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcba438740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b2fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcba438740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b34d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcba438740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b37840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcba438740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86a95370 con 0x563c86b30240 2024-03-04T07:45:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcba438740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86a3e550 con 0x563c86b2f850 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcba438740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86a91410 con 0x563c86b353b0 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b2fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b2fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52518/0 (socket says 172.21.15.116:52518) 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b34d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb5800640 1 -- 172.21.15.116:0/674067656 learned_addr learned my addr 172.21.15.116:0/674067656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcafe00640 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b37840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb5800640 1 -- 172.21.15.116:0/674067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 msgr2=0x563c86b37840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb5800640 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b37840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb5800640 1 -- 172.21.15.116:0/674067656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 msgr2=0x563c86b34d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.320+0000 7fdcb5800640 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b34d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdcb5800640 1 -- 172.21.15.116:0/674067656 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c86a523c0 con 0x563c86b2f850 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdcafe00640 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b37840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdcb5800640 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b2fc70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fdca8011440 tx=0x7fdca8011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2862608603d5a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdcad600640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdc940a1e20 0x7fdc940a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdca6000640 1 -- 172.21.15.116:0/674067656 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8003c30 con 0x563c86b2f850 2024-03-04T07:45:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdca6000640 1 -- 172.21.15.116:0/674067656 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdca8003dd0 con 0x563c86b2f850 2024-03-04T07:45:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdca6000640 1 -- 172.21.15.116:0/674067656 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca800fb50 con 0x563c86b2f850 2024-03-04T07:45:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdcad600640 1 -- 172.21.15.116:0/3167438658 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdca00efe70 con 0x7fdcb0103830 2024-03-04T07:45:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.321+0000 7fdcb4e00640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdc940a1e20 0x7fdc940a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 -- 172.21.15.116:0/674067656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 msgr2=0x563c86b2fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b2fc70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fdca8011440 tx=0x7fdca8011900 comp rx=0 tx=0).stop 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 -- 172.21.15.116:0/674067656 shutdown_connections 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b37840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b34d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 --2- 172.21.15.116:0/674067656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b2fc70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 -- 172.21.15.116:0/674067656 >> 172.21.15.116:0/674067656 conn(0x563c86b2adc0 msgr2=0x563c86b2b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcb4e00640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdc940a1e20 0x7fdc940a42e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdca00b9530 tx=0x7fdca00f38b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 -- 172.21.15.116:0/674067656 shutdown_connections 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 -- 172.21.15.116:0/674067656 wait complete. 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.322+0000 7fdcba438740 1 -- start start 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 1 Processor -- start 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b39c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b3a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b41130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86b38360 con 0x563c86b30240 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86a519b0 con 0x563c86b2f850 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c86b384e0 con 0x563c86b353b0 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcba438740 10 client.0 _ll_register_callbacks cb 0x563c86a8d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b39c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcafe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b3a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b39c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52532/0 (socket says 172.21.15.116:52532) 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcb5800640 1 -- 172.21.15.116:0/2970504994 learned_addr learned my addr 172.21.15.116:0/2970504994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.323+0000 7fdcb5800640 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b41130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcafe00640 1 -- 172.21.15.116:0/2970504994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 msgr2=0x563c86b41130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcafe00640 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b41130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcafe00640 1 -- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 msgr2=0x563c86b39c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcafe00640 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b39c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcafe00640 1 -- 172.21.15.116:0/2970504994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c867aa210 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcb5800640 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b39c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcafe00640 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b3a160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdcb019afb0 tx=0x7fdc98013bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8db4d74e80ab1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdc9ea00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdc9ea00640 1 -- 172.21.15.116:0/2970504994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98011d70 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdc9ea00640 1 -- 172.21.15.116:0/2970504994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdc98005020 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.324+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdca8003dd0 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdc9ea00640 1 -- 172.21.15.116:0/2970504994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc98005220 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 10 client.16359 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563c86876620 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdc9ea00640 1 -- 172.21.15.116:0/2970504994 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdca8003dd0 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdc9ea00640 1 -- 172.21.15.116:0/2970504994 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdc9801a030 con 0x563c86b30240 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 10 client.16359 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:35.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 10 client.16359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 20 client.16359 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86b45a70 0x563c86b47e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 4 client.16359 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.325+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c86aa28b0 con 0x563c86b45a70 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.326+0000 7fdcb5800640 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86b45a70 0x563c86b47e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.326+0000 7fdcb5800640 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86b45a70 0x563c86b47e80 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a838d2f534631390 server_cookie=cf05a918bd2dcf6c in_seq=0 out_seq=0 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.326+0000 7fdc9ea00640 10 client.16359 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.326+0000 7fdc9ea00640 1 -- 172.21.15.116:0/2970504994 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563c86aa28b0 con 0x563c86b45a70 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.326+0000 7fdc9ea00640 10 client.16359 handle_command_reply: tid=0 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 client.16359 shutdown 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 msgr2=0x563c86b3a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b3a160 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdcb019afb0 tx=0x7fdc98013bf0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 shutdown_connections 2024-03-04T07:45:35.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86b45a70 0x563c86b47e80 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b41130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b30240 0x563c86b3a160 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdc9ea00640 0 client.16359 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 --2- 172.21.15.116:0/2970504994 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b2f850 0x563c86b39c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 >> 172.21.15.116:0/2970504994 conn(0x563c86b2adc0 msgr2=0x563c86b29fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 shutdown_connections 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 1 -- 172.21.15.116:0/2970504994 wait complete. 2024-03-04T07:45:35.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.327+0000 7fdcba438740 20 client.16359 trim_cache size 0 max 16384 2024-03-04T07:45:35.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 Processor -- start 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 -- start start 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b2e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b410f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b4c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01a8c30 con 0x563c86b2f850 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01ab210 con 0x563c86b30240 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.342+0000 7fdcba438740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01a9230 con 0x563c86b353b0 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b2e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52540/0 (socket says 172.21.15.116:52540) 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 -- 172.21.15.116:0/1635666783 learned_addr learned my addr 172.21.15.116:0/1635666783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcb5800640 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b4c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 -- 172.21.15.116:0/1635666783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 msgr2=0x563c86b4c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b4c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 -- 172.21.15.116:0/1635666783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 msgr2=0x563c86b2e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b2e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 -- 172.21.15.116:0/1635666783 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c86b48500 con 0x563c86b30240 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcb5800640 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b4c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.343+0000 7fdcafe00640 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b410f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009c80 tx=0x7fdc98009d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43a6013080997582 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdca5600640 1 -- 172.21.15.116:0/1635666783 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9802a070 con 0x563c86b30240 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdca5600640 1 -- 172.21.15.116:0/1635666783 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc98011d70 con 0x563c86b30240 2024-03-04T07:45:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdca5600640 1 -- 172.21.15.116:0/1635666783 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc980049d0 con 0x563c86b30240 2024-03-04T07:45:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 -- 172.21.15.116:0/1635666783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 msgr2=0x563c86b410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b410f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009c80 tx=0x7fdc98009d20 comp rx=0 tx=0).stop 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 -- 172.21.15.116:0/1635666783 shutdown_connections 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b4c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b410f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 --2- 172.21.15.116:0/1635666783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b2e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 -- 172.21.15.116:0/1635666783 >> 172.21.15.116:0/1635666783 conn(0x563c86b2adc0 msgr2=0x563c86b2e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 -- 172.21.15.116:0/1635666783 shutdown_connections 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.344+0000 7fdcba438740 1 -- 172.21.15.116:0/1635666783 wait complete. 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 -- start start 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 Processor -- start 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b3ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b3f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b402f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01a9890 con 0x563c86b2f850 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01aabb0 con 0x563c86b30240 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01a9ef0 con 0x563c86b353b0 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcba438740 10 client.0 _ll_register_callbacks cb 0x7fdc98009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.345+0000 7fdcafe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b3f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b3ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b3f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:52550/0 (socket says 172.21.15.116:52550) 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 -- 172.21.15.116:0/131464352 learned_addr learned my addr 172.21.15.116:0/131464352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcb5800640 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 -- 172.21.15.116:0/131464352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 msgr2=0x563c86b402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b402f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 -- 172.21.15.116:0/131464352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 msgr2=0x563c86b3ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b3ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 -- 172.21.15.116:0/131464352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c86b2bb30 con 0x563c86b30240 2024-03-04T07:45:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcb5800640 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcafe00640 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b3f350 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdc980059f0 tx=0x7fdc9800ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abd90baa3db81a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdc9ea00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdc9ea00640 1 -- 172.21.15.116:0/131464352 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9802a040 con 0x563c86b30240 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdc9ea00640 1 -- 172.21.15.116:0/131464352 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdc980048e0 con 0x563c86b30240 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.346+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc98011d70 con 0x563c86b30240 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdc9ea00640 1 -- 172.21.15.116:0/131464352 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc980117b0 con 0x563c86b30240 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 10 client.26495 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563c86ba9e00 con 0x563c86b30240 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdc9ea00640 1 -- 172.21.15.116:0/131464352 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc98011d70 con 0x563c86b30240 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdc9ea00640 1 -- 172.21.15.116:0/131464352 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdc9801a040 con 0x563c86b30240 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 10 client.26495 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 10 client.26495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 20 client.26495 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86baa6c0 0x563c86bacae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 4 client.26495 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:45:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563c86a9cc00 con 0x563c86baa6c0 2024-03-04T07:45:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.347+0000 7fdcb5800640 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86baa6c0 0x563c86bacae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.348+0000 7fdcb5800640 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86baa6c0 0x563c86bacae0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ea8eea24efbbef server_cookie=8fb4e2461a15992f in_seq=0 out_seq=0 2024-03-04T07:45:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.348+0000 7fdc9ea00640 10 client.26495 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdc9ea00640 1 -- 172.21.15.116:0/131464352 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563c86a9cc00 con 0x563c86baa6c0 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdc9ea00640 10 client.26495 handle_command_reply: tid=0 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 client.26495 shutdown 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 msgr2=0x563c86b3f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b3f350 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdc980059f0 tx=0x7fdc9800ff70 comp rx=0 tx=0).stop 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 shutdown_connections 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563c86baa6c0 0x563c86bacae0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563c86b353b0 0x563c86b402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563c86b30240 0x563c86b3f350 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 --2- 172.21.15.116:0/131464352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c86b2f850 0x563c86b3ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdc9ea00640 0 client.26495 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.362+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 >> 172.21.15.116:0/131464352 conn(0x563c86b2adc0 msgr2=0x563c86b2e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.363+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 shutdown_connections 2024-03-04T07:45:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.363+0000 7fdcba438740 1 -- 172.21.15.116:0/131464352 wait complete. 2024-03-04T07:45:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.363+0000 7fdcba438740 20 client.26495 trim_cache size 0 max 16384 2024-03-04T07:45:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:45:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:45:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:45:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:45:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdc940a1e20 msgr2=0x7fdc940a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdc940a1e20 0x7fdc940a42e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdca00b9530 tx=0x7fdca00f38b0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb0103830 msgr2=0x7fdcb0199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb0103830 0x7fdcb0199aa0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fdca0000c00 tx=0x7fdca002c650 comp rx=0 tx=0).stop 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 shutdown_connections 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb01053e0 0x7fdcb01a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdc940a1e20 0x7fdc940a42e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdcb0104a20 0x7fdcb0199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 --2- 172.21.15.116:0/3167438658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdcb0103830 0x7fdcb0199aa0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 >> 172.21.15.116:0/3167438658 conn(0x7fdcb00fedc0 msgr2=0x7fdcb0107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 shutdown_connections 2024-03-04T07:45:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:35.364+0000 7fdcb7800640 1 -- 172.21.15.116:0/3167438658 wait complete. 2024-03-04T07:45:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:36 smithi133 ceph-mon[36505]: pgmap v546: 97 pgs: 97 active+clean; 3.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:45:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:36 smithi116 ceph-mon[32150]: pgmap v546: 97 pgs: 97 active+clean; 3.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:45:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:36 smithi152 ceph-mon[37154]: pgmap v546: 97 pgs: 97 active+clean; 3.5 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:45:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:38 smithi133 ceph-mon[36505]: pgmap v547: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:45:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:38 smithi116 ceph-mon[32150]: pgmap v547: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:45:39.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:38 smithi152 ceph-mon[37154]: pgmap v547: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:45:40.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:39 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:45:40.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:39 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:45:40.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:39 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:45:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:40 smithi133 ceph-mon[36505]: pgmap v548: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:45:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:45:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:40 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:45:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:40 smithi116 ceph-mon[32150]: pgmap v548: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:45:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:45:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:40 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:45:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:40 smithi152 ceph-mon[37154]: pgmap v548: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:45:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:45:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:40 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:45:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:42 smithi116 ceph-mon[32150]: pgmap v549: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:45:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:42 smithi152 ceph-mon[37154]: pgmap v549: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:45:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:42 smithi133 ceph-mon[36505]: pgmap v549: 97 pgs: 97 active+clean; 3.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:45:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:44 smithi116 ceph-mon[32150]: pgmap v550: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:44 smithi152 ceph-mon[37154]: pgmap v550: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:44 smithi133 ceph-mon[36505]: pgmap v550: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:46 smithi116 ceph-mon[32150]: pgmap v551: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:45:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:46 smithi152 ceph-mon[37154]: pgmap v551: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:45:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:46 smithi133 ceph-mon[36505]: pgmap v551: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:45:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 -- 172.21.15.116:0/3211791958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 msgr2=0x7fabd81092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3211791958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd81092a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fabc8009900 tx=0x7fabc802bc40 comp rx=0 tx=0).stop 2024-03-04T07:45:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 -- 172.21.15.116:0/3211791958 shutdown_connections 2024-03-04T07:45:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3211791958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd81097e0 0x7fabd8110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3211791958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd81092a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3211791958 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabd8107c30 0x7fabd8108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 -- 172.21.15.116:0/3211791958 >> 172.21.15.116:0/3211791958 conn(0x7fabd8076be0 msgr2=0x7fabd8079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 -- 172.21.15.116:0/3211791958 shutdown_connections 2024-03-04T07:45:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.506+0000 7fabdfc00640 1 -- 172.21.15.116:0/3211791958 wait complete. 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.507+0000 7fabdfc00640 1 Processor -- start 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.507+0000 7fabdfc00640 1 -- start start 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabdfc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd8107c30 0x7fabd819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabdfc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabdfc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabd81097e0 0x7fabd81a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabdfc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81138e0 con 0x7fabd8108e20 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabdfc00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd8113760 con 0x7fabd81097e0 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabdfc00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd8113a60 con 0x7fabd8107c30 2024-03-04T07:45:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabddc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabd81097e0 0x7fabd81a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd819e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49514/0 (socket says 172.21.15.116:49514) 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 -- 172.21.15.116:0/3585292932 learned_addr learned my addr 172.21.15.116:0/3585292932 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabdd200640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd8107c30 0x7fabd819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 -- 172.21.15.116:0/3585292932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd8107c30 msgr2=0x7fabd819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd8107c30 0x7fabd819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 -- 172.21.15.116:0/3585292932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabd81097e0 msgr2=0x7fabd81a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabd81097e0 0x7fabd81a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabcfe00640 1 -- 172.21.15.116:0/3585292932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd81a5a80 con 0x7fabd8108e20 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.508+0000 7fabddc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabd81097e0 0x7fabd81a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.509+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd819e3a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fabc802c8f0 tx=0x7fabc802c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.509+0000 7fabd6000640 1 -- 172.21.15.116:0/3585292932 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8041070 con 0x7fabd8108e20 2024-03-04T07:45:48.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.509+0000 7fabd6000640 1 -- 172.21.15.116:0/3585292932 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc8004450 con 0x7fabd8108e20 2024-03-04T07:45:48.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.509+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd81a5d10 con 0x7fabd8108e20 2024-03-04T07:45:48.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.509+0000 7fabd6000640 1 -- 172.21.15.116:0/3585292932 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8003df0 con 0x7fabd8108e20 2024-03-04T07:45:48.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.509+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd81a61b0 con 0x7fabd8108e20 2024-03-04T07:45:48.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.512+0000 7fabd6000640 1 -- 172.21.15.116:0/3585292932 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fabc8044070 con 0x7fabd8108e20 2024-03-04T07:45:48.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.512+0000 7fabe29c3740 1 Processor -- start 2024-03-04T07:45:48.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.512+0000 7fabe29c3740 1 -- start start 2024-03-04T07:45:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabe29c3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c96892e370 con 0x55c9689c9240 2024-03-04T07:45:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabe29c3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9688d7550 con 0x55c9689ce3b0 2024-03-04T07:45:48.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabe29c3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c96892a410 con 0x55c9689c8850 2024-03-04T07:45:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabdd200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabcfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabddc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabcfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55478/0 (socket says 172.21.15.116:55478) 2024-03-04T07:45:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabdd200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689cdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49530/0 (socket says 172.21.15.116:49530) 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabcfe00640 1 -- 172.21.15.116:0/3439319060 learned_addr learned my addr 172.21.15.116:0/3439319060 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabddc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:37962/0 (socket says 172.21.15.116:37962) 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabd6000640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fabb80a1e20 0x7fabb80a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabcfe00640 1 -- 172.21.15.116:0/3439319060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 msgr2=0x55c9689c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3439319060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabcfe00640 1 -- 172.21.15.116:0/3439319060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 msgr2=0x55c9689cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.513+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3439319060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.514+0000 7fabcfe00640 1 -- 172.21.15.116:0/3439319060 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9688eb3c0 con 0x55c9689ce3b0 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.514+0000 7fabddc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fabb80a1e20 0x7fabb80a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.514+0000 7fabd6000640 1 -- 172.21.15.116:0/3585292932 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fabc80ef910 con 0x7fabd8108e20 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.514+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3439319060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d0840 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fabc808f900 tx=0x7fabc8087750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be550fe58bbc1ae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.514+0000 7fabce000640 1 -- 172.21.15.116:0/3439319060 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc80b8040 con 0x55c9689ce3b0 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.514+0000 7fabce000640 1 -- 172.21.15.116:0/3439319060 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc80bcd00 con 0x55c9689ce3b0 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.514+0000 7fabce000640 1 -- 172.21.15.116:0/3439319060 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc80bc140 con 0x55c9689ce3b0 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 -- 172.21.15.116:0/3439319060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 msgr2=0x55c9689d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabddc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fabb80a1e20 0x7fabb80a42e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55c9689c9aa0 tx=0x7fabd0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3439319060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d0840 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fabc808f900 tx=0x7fabc8087750 comp rx=0 tx=0).stop 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 -- 172.21.15.116:0/3439319060 shutdown_connections 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3439319060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d0840 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3439319060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3439319060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 -- 172.21.15.116:0/3439319060 >> 172.21.15.116:0/3439319060 conn(0x55c9689c3dc0 msgr2=0x55c9689c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 -- 172.21.15.116:0/3439319060 shutdown_connections 2024-03-04T07:45:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.515+0000 7fabe29c3740 1 -- 172.21.15.116:0/3439319060 wait complete. 2024-03-04T07:45:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 -- start start 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 Processor -- start 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9688d7550 con 0x55c9689c8850 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9688ea9b0 con 0x55c9689ce3b0 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9689d1120 con 0x55c9689c9240 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.516+0000 7fabe29c3740 10 client.0 _ll_register_callbacks cb 0x55c968926340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55490/0 (socket says 172.21.15.116:55490) 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 -- 172.21.15.116:0/420912915 learned_addr learned my addr 172.21.15.116:0/420912915 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabdd200640 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabcfe00640 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 -- 172.21.15.116:0/420912915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 msgr2=0x55c9689d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 -- 172.21.15.116:0/420912915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 msgr2=0x55c9689d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 -- 172.21.15.116:0/420912915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c968643210 con 0x55c9689ce3b0 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabddc00640 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9bb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fabd001c6d0 tx=0x7fabd0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6387e62950113130 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.517+0000 7fabc6a00640 1 -- 172.21.15.116:0/420912915 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd002c060 con 0x55c9689ce3b0 2024-03-04T07:45:48.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabc6a00640 1 -- 172.21.15.116:0/420912915 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fabd000e880 con 0x55c9689ce3b0 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabc80bcd00 con 0x55c9689ce3b0 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabc6a00640 1 -- 172.21.15.116:0/420912915 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabd000ea80 con 0x55c9689ce3b0 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 10 client.26507 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c96870f620 con 0x55c9689ce3b0 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabc6a00640 1 -- 172.21.15.116:0/420912915 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc80bcd00 con 0x55c9689ce3b0 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabc6a00640 1 -- 172.21.15.116:0/420912915 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fabd0031070 con 0x55c9689ce3b0 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 10 client.26507 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 10 client.26507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 20 client.26507 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c9689de510 0x55c9689e0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 4 client.26507 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.518+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c96893b8b0 con 0x55c9689de510 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.519+0000 7fabdd200640 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c9689de510 0x55c9689e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.519+0000 7fabdd200640 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c9689de510 0x55c9689e0990 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d36793fe1caf2ac2 server_cookie=577d92ad0c5c4d59 in_seq=0 out_seq=0 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.519+0000 7fabc6a00640 10 client.26507 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.520+0000 7fabc6a00640 1 -- 172.21.15.116:0/420912915 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c96893b8b0 con 0x55c9689de510 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.520+0000 7fabc6a00640 10 client.26507 handle_command_reply: tid=0 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.520+0000 7fabe29c3740 1 client.26507 shutdown 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.520+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 msgr2=0x55c9689d9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.520+0000 7fabe29c3740 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9bb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fabd001c6d0 tx=0x7fabd0004ab0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 shutdown_connections 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c9689de510 0x55c9689e0990 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9bb0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 --2- 172.21.15.116:0/420912915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 >> 172.21.15.116:0/420912915 conn(0x55c9689c3dc0 msgr2=0x55c9689c2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabc6a00640 0 client.26507 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 shutdown_connections 2024-03-04T07:45:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 1 -- 172.21.15.116:0/420912915 wait complete. 2024-03-04T07:45:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.521+0000 7fabe29c3740 20 client.26507 trim_cache size 0 max 16384 2024-03-04T07:45:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 Processor -- start 2024-03-04T07:45:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 -- start start 2024-03-04T07:45:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689c7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81acf90 con 0x55c9689c9240 2024-03-04T07:45:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81af570 con 0x55c9689ce3b0 2024-03-04T07:45:48.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.536+0000 7fabe29c3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81ad590 con 0x55c9689c8850 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55502/0 (socket says 172.21.15.116:55502) 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 -- 172.21.15.116:0/3973099717 learned_addr learned my addr 172.21.15.116:0/3973099717 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabdd200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 -- 172.21.15.116:0/3973099717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 msgr2=0x55c9689c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 -- 172.21.15.116:0/3973099717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 msgr2=0x55c9689c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 -- 172.21.15.116:0/3973099717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9689c9f00 con 0x55c9689ce3b0 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabdd200640 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.537+0000 7fabddc00640 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9b70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fabd0005c40 tx=0x7fabd0011750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38fcebd1e5c95e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.538+0000 7fabcd600640 1 -- 172.21.15.116:0/3973099717 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0037070 con 0x55c9689ce3b0 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.538+0000 7fabcd600640 1 -- 172.21.15.116:0/3973099717 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd000e880 con 0x55c9689ce3b0 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.538+0000 7fabcd600640 1 -- 172.21.15.116:0/3973099717 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0026790 con 0x55c9689ce3b0 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.538+0000 7fabe29c3740 1 -- 172.21.15.116:0/3973099717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 msgr2=0x55c9689d9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.538+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9b70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fabd0005c40 tx=0x7fabd0011750 comp rx=0 tx=0).stop 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 -- 172.21.15.116:0/3973099717 shutdown_connections 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d9b70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c9240 0x55c9689c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3973099717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c8850 0x55c9689c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 -- 172.21.15.116:0/3973099717 >> 172.21.15.116:0/3973099717 conn(0x55c9689c3dc0 msgr2=0x55c9689ca420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 -- 172.21.15.116:0/3973099717 shutdown_connections 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 -- 172.21.15.116:0/3973099717 wait complete. 2024-03-04T07:45:48.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 -- start start 2024-03-04T07:45:48.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.539+0000 7fabe29c3740 1 Processor -- start 2024-03-04T07:45:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabe29c3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabe29c3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81adbf0 con 0x55c9689c8850 2024-03-04T07:45:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabe29c3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81aef10 con 0x55c9689ce3b0 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabe29c3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81ae250 con 0x55c9689c9240 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabe29c3740 10 client.0 _ll_register_callbacks cb 0x7fabd000fd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49560/0 (socket says 172.21.15.116:49560) 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 -- 172.21.15.116:0/3242623797 learned_addr learned my addr 172.21.15.116:0/3242623797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 -- 172.21.15.116:0/3242623797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 msgr2=0x55c9689d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 -- 172.21.15.116:0/3242623797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 msgr2=0x55c9689d8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabdd200640 1 -- 172.21.15.116:0/3242623797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9689d8c20 con 0x55c9689c8850 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.540+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabdd200640 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d7130 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009e50 tx=0x7fabc0010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b6a23e33d45995 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:45:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabc6a00640 1 -- 172.21.15.116:0/3242623797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0024070 con 0x55c9689c8850 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabc6a00640 1 -- 172.21.15.116:0/3242623797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fabc0004250 con 0x55c9689c8850 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabd000e880 con 0x55c9689c8850 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabc6a00640 1 -- 172.21.15.116:0/3242623797 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabc0030050 con 0x55c9689c8850 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabe29c3740 10 client.16383 fetch_fsmap learned FSMap version 19 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c968a39e20 con 0x55c9689c8850 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.541+0000 7fabc6a00640 1 -- 172.21.15.116:0/3242623797 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd000e880 con 0x55c9689c8850 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabc6a00640 1 -- 172.21.15.116:0/3242623797 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fabc00039c0 con 0x55c9689c8850 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabe29c3740 10 client.16383 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabe29c3740 10 client.16383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabe29c3740 20 client.16383 populate_metadata read hostname 'smithi116' 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c968a3a640 0x55c968a3cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabe29c3740 4 client.16383 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:45:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c968935c00 con 0x55c968a3a640 2024-03-04T07:45:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c968a3a640 0x55c968a3cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:45:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabcfe00640 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c968a3a640 0x55c968a3cae0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1365ca18b2134be server_cookie=c533a400eddd460f in_seq=0 out_seq=0 2024-03-04T07:45:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.542+0000 7fabc6a00640 10 client.16383 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabc6a00640 1 -- 172.21.15.116:0/3242623797 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c968935c00 con 0x55c968a3a640 2024-03-04T07:45:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabc6a00640 10 client.16383 handle_command_reply: tid=0 2024-03-04T07:45:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 client.16383 shutdown 2024-03-04T07:45:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 msgr2=0x55c9689d7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d7130 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009e50 tx=0x7fabc0010a40 comp rx=0 tx=0).stop 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 shutdown_connections 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c968a3a640 0x55c968a3cae0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c9689ce3b0 0x55c9689d8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9689c9240 0x55c9689d7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 --2- 172.21.15.116:0/3242623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c9689c8850 0x55c9689d7130 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabc6a00640 0 client.16383 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 >> 172.21.15.116:0/3242623797 conn(0x55c9689c3dc0 msgr2=0x55c9689d25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 shutdown_connections 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 1 -- 172.21.15.116:0/3242623797 wait complete. 2024-03-04T07:45:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabe29c3740 20 client.16383 trim_cache size 0 max 16384 2024-03-04T07:45:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:45:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:45:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:45:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:45:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fabb80a1e20 msgr2=0x7fabb80a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fabb80a1e20 0x7fabb80a42e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55c9689c9aa0 tx=0x7fabd0004a40 comp rx=0 tx=0).stop 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 msgr2=0x7fabd819e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.560+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd819e3a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fabc802c8f0 tx=0x7fabc802c660 comp rx=0 tx=0).stop 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 shutdown_connections 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fabd81097e0 0x7fabd81a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabd8108e20 0x7fabd819e3a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fabb80a1e20 0x7fabb80a42e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 --2- 172.21.15.116:0/3585292932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd8107c30 0x7fabd819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 >> 172.21.15.116:0/3585292932 conn(0x7fabd8076be0 msgr2=0x7fabd810c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 shutdown_connections 2024-03-04T07:45:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:45:48.561+0000 7fabdfc00640 1 -- 172.21.15.116:0/3585292932 wait complete. 2024-03-04T07:45:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:48 smithi116 ceph-mon[32150]: pgmap v552: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:45:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:48 smithi152 ceph-mon[37154]: pgmap v552: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:45:49.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:48 smithi133 ceph-mon[36505]: pgmap v552: 97 pgs: 97 active+clean; 3.6 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:45:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:50 smithi116 ceph-mon[32150]: pgmap v553: 97 pgs: 97 active+clean; 3.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:45:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:50 smithi152 ceph-mon[37154]: pgmap v553: 97 pgs: 97 active+clean; 3.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:45:51.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:50 smithi133 ceph-mon[36505]: pgmap v553: 97 pgs: 97 active+clean; 3.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:45:53.139 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:52 smithi133 ceph-mon[36505]: pgmap v554: 97 pgs: 97 active+clean; 3.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:52 smithi116 ceph-mon[32150]: pgmap v554: 97 pgs: 97 active+clean; 3.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:52 smithi152 ceph-mon[37154]: pgmap v554: 97 pgs: 97 active+clean; 3.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:45:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:54 smithi116 ceph-mon[32150]: pgmap v555: 97 pgs: 97 active+clean; 3.7 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:54 smithi152 ceph-mon[37154]: pgmap v555: 97 pgs: 97 active+clean; 3.7 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:55.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:54 smithi133 ceph-mon[36505]: pgmap v555: 97 pgs: 97 active+clean; 3.7 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:56 smithi116 ceph-mon[32150]: pgmap v556: 97 pgs: 97 active+clean; 3.7 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:56 smithi152 ceph-mon[37154]: pgmap v556: 97 pgs: 97 active+clean; 3.7 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:57.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:56 smithi133 ceph-mon[36505]: pgmap v556: 97 pgs: 97 active+clean; 3.7 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:45:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:45:58 smithi116 ceph-mon[32150]: pgmap v557: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:45:58 smithi152 ceph-mon[37154]: pgmap v557: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:45:59.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:45:58 smithi133 ceph-mon[36505]: pgmap v557: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:00.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:46:00.341 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 -- 172.21.15.116:0/3670707331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 msgr2=0x7f9d241018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 --2- 172.21.15.116:0/3670707331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 0x7f9d241018d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009900 tx=0x7f9d1402bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 -- 172.21.15.116:0/3670707331 shutdown_connections 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 --2- 172.21.15.116:0/3670707331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d24103060 0x7f9d241098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 --2- 172.21.15.116:0/3670707331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d241026a0 0x7f9d24102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 --2- 172.21.15.116:0/3670707331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 0x7f9d241018d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 -- 172.21.15.116:0/3670707331 >> 172.21.15.116:0/3670707331 conn(0x7f9d240fca40 msgr2=0x7f9d240fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.562+0000 7f9d23e00640 1 -- 172.21.15.116:0/3670707331 shutdown_connections 2024-03-04T07:46:00.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.563+0000 7f9d23e00640 1 -- 172.21.15.116:0/3670707331 wait complete. 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.563+0000 7f9d23e00640 1 Processor -- start 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23e00640 1 -- start start 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 0x7f9d24199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d241026a0 0x7f9d24199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d24103060 0x7f9d241a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2410d190 con 0x7f9d241014b0 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2410d010 con 0x7f9d241026a0 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2410d310 con 0x7f9d24103060 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d22a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 0x7f9d24199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d24103060 0x7f9d241a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d1a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d241026a0 0x7f9d24199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d1a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d241026a0 0x7f9d24199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59958/0 (socket says 172.21.15.116:59958) 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d1a000640 1 -- 172.21.15.116:0/2699622131 learned_addr learned my addr 172.21.15.116:0/2699622131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d23400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d24103060 0x7f9d241a0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:42844/0 (socket says 172.21.15.116:42844) 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d22a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 0x7f9d24199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35464/0 (socket says 172.21.15.116:35464) 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.564+0000 7f9d1a000640 1 -- 172.21.15.116:0/2699622131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d24103060 msgr2=0x7f9d241a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d24103060 0x7f9d241a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1a000640 1 -- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 msgr2=0x7f9d24199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 0x7f9d24199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1a000640 1 -- 172.21.15.116:0/2699622131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d241a1630 con 0x7f9d241026a0 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d241026a0 0x7f9d24199f50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00e9f0 tx=0x7f9d0c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1be00640 1 -- 172.21.15.116:0/2699622131 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00ccc0 con 0x7f9d241026a0 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1be00640 1 -- 172.21.15.116:0/2699622131 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c00ce60 con 0x7f9d241026a0 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d1be00640 1 -- 172.21.15.116:0/2699622131 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c010690 con 0x7f9d241026a0 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d241a1920 con 0x7f9d241026a0 2024-03-04T07:46:00.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.565+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d241a1e20 con 0x7f9d241026a0 2024-03-04T07:46:00.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.568+0000 7f9d1be00640 1 -- 172.21.15.116:0/2699622131 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9d0c0046c0 con 0x7f9d241026a0 2024-03-04T07:46:00.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.568+0000 7f9d2c7de740 1 Processor -- start 2024-03-04T07:46:00.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d2c7de740 1 -- start start 2024-03-04T07:46:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596d9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d9870 0x5615596dbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561559601090 con 0x5615596d4700 2024-03-04T07:46:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561559638d30 con 0x5615596d3d10 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615595e1a30 con 0x5615596d9870 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59972/0 (socket says 172.21.15.116:59972) 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d23400640 1 -- 172.21.15.116:0/2965781239 learned_addr learned my addr 172.21.15.116:0/2965781239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d9870 0x5615596dbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.569+0000 7f9d22a00640 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596d9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d23400640 1 -- 172.21.15.116:0/2965781239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d9870 msgr2=0x5615596dbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d23400640 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d9870 0x5615596dbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d23400640 1 -- 172.21.15.116:0/2965781239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 msgr2=0x5615596d9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d23400640 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596d9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d23400640 1 -- 172.21.15.116:0/2965781239 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615594177d0 con 0x5615596d3d10 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d9870 0x5615596dbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d23400640 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d4130 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c011440 tx=0x7f9d1c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf6dd4dc09af2eca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d1be00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d080a1d80 0x7f9d080a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d13e00640 1 -- 172.21.15.116:0/2965781239 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c003c30 con 0x5615596d3d10 2024-03-04T07:46:00.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d13e00640 1 -- 172.21.15.116:0/2965781239 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d1c003dd0 con 0x5615596d3d10 2024-03-04T07:46:00.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d13e00640 1 -- 172.21.15.116:0/2965781239 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c00fb50 con 0x5615596d3d10 2024-03-04T07:46:00.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d1be00640 1 -- 172.21.15.116:0/2699622131 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9d0c0ce1e0 con 0x7f9d241026a0 2024-03-04T07:46:00.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.570+0000 7f9d22a00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d080a1d80 0x7f9d080a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2965781239 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 msgr2=0x5615596d4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d4130 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c011440 tx=0x7f9d1c011900 comp rx=0 tx=0).stop 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2965781239 shutdown_connections 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d9870 0x5615596dbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596d9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2965781239 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d4130 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2965781239 >> 172.21.15.116:0/2965781239 conn(0x5615596cf280 msgr2=0x5615596cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d22a00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d080a1d80 0x7f9d080a4240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9d1402c170 tx=0x7f9d1402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2965781239 shutdown_connections 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.571+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2965781239 wait complete. 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.572+0000 7f9d2c7de740 1 -- start start 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.572+0000 7f9d2c7de740 1 Processor -- start 2024-03-04T07:46:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.572+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596de190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.572+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d4700 0x5615596de6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.572+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596e5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.572+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615595d70c0 con 0x5615596d9870 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561559634dd0 con 0x5615596d4700 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615596dc820 con 0x5615596d3d10 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d2c7de740 10 client.0 _ll_register_callbacks cb 0x5615595f4860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d4700 0x5615596de6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d23400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596e5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d4700 0x5615596de6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59986/0 (socket says 172.21.15.116:59986) 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 -- 172.21.15.116:0/2455366666 learned_addr learned my addr 172.21.15.116:0/2455366666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d22a00640 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596de190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 -- 172.21.15.116:0/2455366666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 msgr2=0x5615596de190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596de190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 -- 172.21.15.116:0/2455366666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 msgr2=0x5615596e5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596e5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 -- 172.21.15.116:0/2455366666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56155934d3f0 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d22a00640 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596de190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d1a000640 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d4700 0x5615596de6d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0cbca0 tx=0x7f9d0c09aad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc0c4275a1906631 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.573+0000 7f9d10c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d10c00640 1 -- 172.21.15.116:0/2455366666 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0a33e0 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d10c00640 1 -- 172.21.15.116:0/2455366666 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9d0c09a320 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d1c003dd0 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d10c00640 1 -- 172.21.15.116:0/2455366666 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d0c09f020 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 10 client.26522 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56155930d0f0 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d10c00640 1 -- 172.21.15.116:0/2455366666 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d1c003dd0 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d10c00640 1 -- 172.21.15.116:0/2455366666 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9d0c0d1af0 con 0x5615596d4700 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 10 client.26522 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:00.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 10 client.26522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 20 client.26522 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5615596e9b50 0x5615596ebf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 4 client.26522 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.574+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561559646270 con 0x5615596e9b50 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.575+0000 7f9d22a00640 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5615596e9b50 0x5615596ebf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.575+0000 7f9d22a00640 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5615596e9b50 0x5615596ebf60 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e02fae7a3414955c server_cookie=f045c634a89bf145 in_seq=0 out_seq=0 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.575+0000 7f9d10c00640 10 client.26522 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d10c00640 1 -- 172.21.15.116:0/2455366666 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561559646270 con 0x5615596e9b50 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d10c00640 10 client.26522 handle_command_reply: tid=0 2024-03-04T07:46:00.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 client.26522 shutdown 2024-03-04T07:46:00.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d4700 msgr2=0x5615596de6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d4700 0x5615596de6d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0cbca0 tx=0x7f9d0c09aad0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 shutdown_connections 2024-03-04T07:46:00.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5615596e9b50 0x5615596ebf60 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596e5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d4700 0x5615596de6d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/2455366666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596de190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d10c00640 0 client.26522 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.576+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 >> 172.21.15.116:0/2455366666 conn(0x5615596cf280 msgr2=0x5615596ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.577+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 shutdown_connections 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.577+0000 7f9d2c7de740 1 -- 172.21.15.116:0/2455366666 wait complete. 2024-03-04T07:46:00.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.577+0000 7f9d2c7de740 20 client.26522 trim_cache size 0 max 16384 2024-03-04T07:46:00.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 Processor -- start 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 -- start start 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d4700 0x5615596e56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596f0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241a8ad0 con 0x5615596d9870 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241ab0b0 con 0x5615596d3d10 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.594+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241a90d0 con 0x5615596d4700 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59994/0 (socket says 172.21.15.116:59994) 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 -- 172.21.15.116:0/3179644357 learned_addr learned my addr 172.21.15.116:0/3179644357 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:00.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d23400640 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d1a000640 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d4700 0x5615596e56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 -- 172.21.15.116:0/3179644357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d4700 msgr2=0x5615596e56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d4700 0x5615596e56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 -- 172.21.15.116:0/3179644357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 msgr2=0x5615596f0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596f0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 -- 172.21.15.116:0/3179644357 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615596c98d0 con 0x5615596d3d10 2024-03-04T07:46:00.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d1a000640 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d4700 0x5615596e56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d22a00640 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d2bf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9d1402c240 tx=0x7f9d14044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ec562c2e6d80dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.595+0000 7f9d13400640 1 -- 172.21.15.116:0/3179644357 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14044070 con 0x5615596d3d10 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d13400640 1 -- 172.21.15.116:0/3179644357 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d14058070 con 0x5615596d3d10 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d13400640 1 -- 172.21.15.116:0/3179644357 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d14003e20 con 0x5615596d3d10 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 -- 172.21.15.116:0/3179644357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 msgr2=0x5615596d2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d2bf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9d1402c240 tx=0x7f9d14044000 comp rx=0 tx=0).stop 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 -- 172.21.15.116:0/3179644357 shutdown_connections 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d9870 0x5615596f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d4700 0x5615596e56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/3179644357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d3d10 0x5615596d2bf0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 -- 172.21.15.116:0/3179644357 >> 172.21.15.116:0/3179644357 conn(0x5615596cf280 msgr2=0x5615596d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 -- 172.21.15.116:0/3179644357 shutdown_connections 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.596+0000 7f9d2c7de740 1 -- 172.21.15.116:0/3179644357 wait complete. 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 -- start start 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 Processor -- start 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596e3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d9870 0x5615596e4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241a9730 con 0x5615596d4700 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241aaa50 con 0x5615596d9870 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d241a9d90 con 0x5615596d3d10 2024-03-04T07:46:00.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.597+0000 7f9d2c7de740 10 client.0 _ll_register_callbacks cb 0x7f9d0c097170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d9870 0x5615596e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d1a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d9870 0x5615596e4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60006/0 (socket says 172.21.15.116:60006) 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 -- 172.21.15.116:0/1700295259 learned_addr learned my addr 172.21.15.116:0/1700295259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d22a00640 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 -- 172.21.15.116:0/1700295259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 msgr2=0x5615596e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596e33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 -- 172.21.15.116:0/1700295259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 msgr2=0x5615596e3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596e3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 -- 172.21.15.116:0/1700295259 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5615596e4f30 con 0x5615596d9870 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d1a000640 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596e3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d23400640 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d9870 0x5615596e4940 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c011440 tx=0x7f9d1c009af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8edf2bbf7fe801f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.598+0000 7f9d10c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d10c00640 1 -- 172.21.15.116:0/1700295259 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c00f750 con 0x5615596d9870 2024-03-04T07:46:00.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d10c00640 1 -- 172.21.15.116:0/1700295259 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9d1c00f8f0 con 0x5615596d9870 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d14058070 con 0x5615596d9870 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d10c00640 1 -- 172.21.15.116:0/1700295259 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d1c00faf0 con 0x5615596d9870 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 10 client.26528 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56155974e230 con 0x5615596d9870 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d10c00640 1 -- 172.21.15.116:0/1700295259 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d14058070 con 0x5615596d9870 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d10c00640 1 -- 172.21.15.116:0/1700295259 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9d1c019430 con 0x5615596d9870 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 10 client.26528 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 20 client.26528 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56155974ea90 0x561559750eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 4 client.26528 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:46:00.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.599+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5615596405c0 con 0x56155974ea90 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d22a00640 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56155974ea90 0x561559750eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d22a00640 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56155974ea90 0x561559750eb0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ef0024e42cc819 server_cookie=fd3e83e9b196b58b in_seq=0 out_seq=0 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d10c00640 10 client.26528 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d10c00640 1 -- 172.21.15.116:0/1700295259 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5615596405c0 con 0x56155974ea90 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d10c00640 10 client.26528 handle_command_reply: tid=0 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d2c7de740 1 client.26528 shutdown 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d9870 msgr2=0x5615596e4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d9870 0x5615596e4940 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c011440 tx=0x7f9d1c009af0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.600+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 shutdown_connections 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56155974ea90 0x561559750eb0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5615596d9870 0x5615596e4940 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615596d4700 0x5615596e3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 1 --2- 172.21.15.116:0/1700295259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5615596d3d10 0x5615596e33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 >> 172.21.15.116:0/1700295259 conn(0x5615596cf280 msgr2=0x5615596d2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d10c00640 0 client.26528 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 shutdown_connections 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 1 -- 172.21.15.116:0/1700295259 wait complete. 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d2c7de740 20 client.26528 trim_cache size 0 max 16384 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:46:00.606 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d080a1d80 msgr2=0x7f9d080a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d23e00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d080a1d80 0x7f9d080a4240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9d1402c170 tx=0x7f9d1402df90 comp rx=0 tx=0).stop 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d241026a0 msgr2=0x7f9d24199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.601+0000 7f9d23e00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d241026a0 0x7f9d24199f50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00e9f0 tx=0x7f9d0c00edf0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 shutdown_connections 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9d080a1d80 0x7f9d080a4240 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d24103060 0x7f9d241a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9d241026a0 0x7f9d24199f50 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 --2- 172.21.15.116:0/2699622131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d241014b0 0x7f9d24199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 >> 172.21.15.116:0/2699622131 conn(0x7f9d240fca40 msgr2=0x7f9d24073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 shutdown_connections 2024-03-04T07:46:00.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.602+0000 7f9d23e00640 1 -- 172.21.15.116:0/2699622131 wait complete. 2024-03-04T07:46:00.618 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:46:00.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 -- 172.21.15.116:0/4014050763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 msgr2=0x7ff1f0107e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 --2- 172.21.15.116:0/4014050763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 0x7ff1f0107e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009900 tx=0x7ff1e002bc50 comp rx=0 tx=0).stop 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 -- 172.21.15.116:0/4014050763 shutdown_connections 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 --2- 172.21.15.116:0/4014050763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f01095e0 0x7ff1f010fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 --2- 172.21.15.116:0/4014050763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f0108c20 0x7ff1f01090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 --2- 172.21.15.116:0/4014050763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 0x7ff1f0107e50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 -- 172.21.15.116:0/4014050763 >> 172.21.15.116:0/4014050763 conn(0x7ff1f00769e0 msgr2=0x7ff1f0078e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 -- 172.21.15.116:0/4014050763 shutdown_connections 2024-03-04T07:46:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.819+0000 7ff1ef200640 1 -- 172.21.15.116:0/4014050763 wait complete. 2024-03-04T07:46:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 Processor -- start 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 -- start start 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 0x7ff1f019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f0108c20 0x7ff1f019e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f01095e0 0x7ff1f01a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f0113700 con 0x7ff1f01095e0 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f0113580 con 0x7ff1f0108c20 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1ef200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f0113880 con 0x7ff1f0107a30 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.821+0000 7ff1e7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f0108c20 0x7ff1f019e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f0108c20 0x7ff1f019e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60022/0 (socket says 172.21.15.116:60022) 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 -- 172.21.15.116:0/3379241851 learned_addr learned my addr 172.21.15.116:0/3379241851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1ee800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f01095e0 0x7ff1f01a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1ede00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 0x7ff1f019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 -- 172.21.15.116:0/3379241851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 msgr2=0x7ff1f019dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 0x7ff1f019dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 -- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f01095e0 msgr2=0x7ff1f01a52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f01095e0 0x7ff1f01a52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 -- 172.21.15.116:0/3379241851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f01a5940 con 0x7ff1f0108c20 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1ede00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 0x7ff1f019dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e7e00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f0108c20 0x7ff1f019e270 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800e9f0 tx=0x7ff1d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.822+0000 7ff1e6a00640 1 -- 172.21.15.116:0/3379241851 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d800cd80 con 0x7ff1f0108c20 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.823+0000 7ff1e6a00640 1 -- 172.21.15.116:0/3379241851 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8010910 con 0x7ff1f0108c20 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.823+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f01a5c30 con 0x7ff1f0108c20 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.823+0000 7ff1e6a00640 1 -- 172.21.15.116:0/3379241851 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80188f0 con 0x7ff1f0108c20 2024-03-04T07:46:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.823+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f01a61a0 con 0x7ff1f0108c20 2024-03-04T07:46:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.824+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f010ed50 con 0x7ff1f0108c20 2024-03-04T07:46:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.826+0000 7ff1e6a00640 1 -- 172.21.15.116:0/3379241851 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff1d8010ac0 con 0x7ff1f0108c20 2024-03-04T07:46:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.827+0000 7ff1e6a00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff1d40a1e00 0x7ff1d40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.827+0000 7ff1e6a00640 1 -- 172.21.15.116:0/3379241851 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff1d80ce180 con 0x7ff1f0108c20 2024-03-04T07:46:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.827+0000 7ff1ede00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff1d40a1e00 0x7ff1d40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.828+0000 7ff1ede00640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff1d40a1e00 0x7ff1d40a42c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff1e002c190 tx=0x7ff1e0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:00.831+0000 7ff1e6a00640 1 -- 172.21.15.116:0/3379241851 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff1d809af20 con 0x7ff1f0108c20 2024-03-04T07:46:01.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:00 smithi116 ceph-mon[32150]: pgmap v558: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:01.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.061+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1f00621d0 con 0x7ff1f0108c20 2024-03-04T07:46:01.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.063+0000 7ff1e6a00640 1 -- 172.21.15.116:0/3379241851 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7ff1d809efc0 con 0x7ff1f0108c20 2024-03-04T07:46:01.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:46:01.063 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:46:01.063 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff1d40a1e00 msgr2=0x7ff1d40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff1d40a1e00 0x7ff1d40a42c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff1e002c190 tx=0x7ff1e0000bc0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f0108c20 msgr2=0x7ff1f019e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f0108c20 0x7ff1f019e270 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800e9f0 tx=0x7ff1d800eeb0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 shutdown_connections 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f01095e0 0x7ff1f01a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1f0108c20 0x7ff1f019e270 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff1d40a1e00 0x7ff1d40a42c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 --2- 172.21.15.116:0/3379241851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1f0107a30 0x7ff1f019dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 >> 172.21.15.116:0/3379241851 conn(0x7ff1f00769e0 msgr2=0x7ff1f010bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 shutdown_connections 2024-03-04T07:46:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.066+0000 7ff1ef200640 1 -- 172.21.15.116:0/3379241851 wait complete. 2024-03-04T07:46:01.085 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:46:01.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:46:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:00 smithi152 ceph-mon[37154]: pgmap v558: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:01.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 -- 172.21.15.116:0/2153261279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1049e0 msgr2=0x7f971c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 --2- 172.21.15.116:0/2153261279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1049e0 0x7f971c104e60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9708009900 tx=0x7f970802bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 -- 172.21.15.116:0/2153261279 shutdown_connections 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 --2- 172.21.15.116:0/2153261279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f971c1053a0 0x7f971c10bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 --2- 172.21.15.116:0/2153261279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1049e0 0x7f971c104e60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 --2- 172.21.15.116:0/2153261279 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 0x7f971c103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 -- 172.21.15.116:0/2153261279 >> 172.21.15.116:0/2153261279 conn(0x7f971c0fed80 msgr2=0x7f971c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 -- 172.21.15.116:0/2153261279 shutdown_connections 2024-03-04T07:46:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.290+0000 7f9723a00640 1 -- 172.21.15.116:0/2153261279 wait complete. 2024-03-04T07:46:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.291+0000 7f9723a00640 1 Processor -- start 2024-03-04T07:46:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.291+0000 7f9723a00640 1 -- start start 2024-03-04T07:46:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9723a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 0x7f971c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9723a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f971c1049e0 0x7f971c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9723a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1053a0 0x7f971c1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9723a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c10f4c0 con 0x7f971c1053a0 2024-03-04T07:46:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9723a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c10f340 con 0x7f971c1037f0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9723a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c10f640 con 0x7f971c1049e0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 0x7f971c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 0x7f971c199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60044/0 (socket says 172.21.15.116:60044) 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 -- 172.21.15.116:0/2624371851 learned_addr learned my addr 172.21.15.116:0/2624371851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1053a0 0x7f971c1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f971be00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f971c1049e0 0x7f971c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 -- 172.21.15.116:0/2624371851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f971c1049e0 msgr2=0x7f971c19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f971c1049e0 0x7f971c19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 -- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1053a0 msgr2=0x7f971c1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1053a0 0x7f971c1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.292+0000 7f9721000640 1 -- 172.21.15.116:0/2624371851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c1a16f0 con 0x7f971c1037f0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.293+0000 7f9721000640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 0x7f971c199ad0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f971000e9c0 tx=0x7f971000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.293+0000 7f9719600640 1 -- 172.21.15.116:0/2624371851 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971000cde0 con 0x7f971c1037f0 2024-03-04T07:46:01.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.293+0000 7f9719600640 1 -- 172.21.15.116:0/2624371851 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9710010910 con 0x7f971c1037f0 2024-03-04T07:46:01.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.293+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c1a19e0 con 0x7f971c1037f0 2024-03-04T07:46:01.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.293+0000 7f9719600640 1 -- 172.21.15.116:0/2624371851 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9710018a00 con 0x7f971c1037f0 2024-03-04T07:46:01.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.293+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c1a1f00 con 0x7f971c1037f0 2024-03-04T07:46:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.296+0000 7f9719600640 1 -- 172.21.15.116:0/2624371851 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f971001f020 con 0x7f971c1037f0 2024-03-04T07:46:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.296+0000 7f972660e740 1 Processor -- start 2024-03-04T07:46:01.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.296+0000 7f972660e740 1 -- start start 2024-03-04T07:46:01.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f972660e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc610c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f972660e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc61690 0x56038fc66190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f972660e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc66800 0x56038fc68c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f972660e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56038fbc5d60 con 0x56038fc66800 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f972660e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56038fb6ecf0 con 0x56038fc61690 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f972660e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56038fbc1e00 con 0x56038fc60ca0 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc61690 0x56038fc66190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:42928/0 (socket says 172.21.15.116:42928) 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f971be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc66800 0x56038fc68c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc61690 0x56038fc66190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60048/0 (socket says 172.21.15.116:60048) 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721a00640 1 -- 172.21.15.116:0/296426131 learned_addr learned my addr 172.21.15.116:0/296426131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721000640 1 -- 172.21.15.116:0/296426131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 msgr2=0x56038fc610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721000640 1 --2- 172.21.15.116:0/296426131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721000640 1 -- 172.21.15.116:0/296426131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc66800 msgr2=0x56038fc68c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721000640 1 --2- 172.21.15.116:0/296426131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc66800 0x56038fc68c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.297+0000 7f9721000640 1 -- 172.21.15.116:0/296426131 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56038f8da3f0 con 0x56038fc61690 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.298+0000 7f9721000640 1 --2- 172.21.15.116:0/296426131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc61690 0x56038fc66190 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f97100cbad0 tx=0x7f97100d6320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30cf6c1840467df3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.298+0000 7f9719600640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97040a1d80 0x7f97040a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.298+0000 7f970e000640 1 -- 172.21.15.116:0/296426131 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97100e1070 con 0x56038fc61690 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.298+0000 7f9721a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97040a1d80 0x7f97040a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.298+0000 7f970e000640 1 -- 172.21.15.116:0/296426131 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f971009b460 con 0x56038fc61690 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.298+0000 7f970e000640 1 -- 172.21.15.116:0/296426131 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971009ab30 con 0x56038fc61690 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.298+0000 7f9719600640 1 -- 172.21.15.116:0/2624371851 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f97100ce070 con 0x7f971c1037f0 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 -- 172.21.15.116:0/296426131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc61690 msgr2=0x56038fc66190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 --2- 172.21.15.116:0/296426131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc61690 0x56038fc66190 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f97100cbad0 tx=0x7f97100d6320 comp rx=0 tx=0).stop 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f9721a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97040a1d80 0x7f97040a4240 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x56038fc61ef0 tx=0x7f9714009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 -- 172.21.15.116:0/296426131 shutdown_connections 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 --2- 172.21.15.116:0/296426131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc66800 0x56038fc68c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 --2- 172.21.15.116:0/296426131 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc61690 0x56038fc66190 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 --2- 172.21.15.116:0/296426131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 -- 172.21.15.116:0/296426131 >> 172.21.15.116:0/296426131 conn(0x56038fc5c210 msgr2=0x56038fc5c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 -- 172.21.15.116:0/296426131 shutdown_connections 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 -- 172.21.15.116:0/296426131 wait complete. 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.299+0000 7f972660e740 1 -- start start 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 1 Processor -- start 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc6b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc66800 0x56038fc726f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56038fc697b0 con 0x56038fc61690 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56038fb64220 con 0x56038fc66800 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56038fc69930 con 0x56038fc60ca0 2024-03-04T07:46:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f972660e740 10 client.0 _ll_register_callbacks cb 0x56038f8d9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f9721000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc6b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f971be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f9721000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc6b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:42932/0 (socket says 172.21.15.116:42932) 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f9721000640 1 -- 172.21.15.116:0/3960203157 learned_addr learned my addr 172.21.15.116:0/3960203157 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.300+0000 7f9721a00640 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc66800 0x56038fc726f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9721000640 1 -- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc66800 msgr2=0x56038fc726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9721000640 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc66800 0x56038fc726f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9721000640 1 -- 172.21.15.116:0/3960203157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 msgr2=0x56038fc6b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9721000640 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9721000640 1 -- 172.21.15.116:0/3960203157 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56038f9a47d0 con 0x56038fc60ca0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9721000640 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc6b170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f971000eff0 tx=0x7f97100d1880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a16327b363fcc579 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9702a00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9702a00640 1 -- 172.21.15.116:0/3960203157 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971005ab90 con 0x56038fc60ca0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9702a00640 1 -- 172.21.15.116:0/3960203157 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f97100cbbb0 con 0x56038fc60ca0 2024-03-04T07:46:01.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f971009b460 con 0x56038fc60ca0 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f9702a00640 1 -- 172.21.15.116:0/3960203157 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97100582d0 con 0x56038fc60ca0 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f972660e740 10 client.26458 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.301+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56038fc697b0 con 0x56038fc60ca0 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f9702a00640 1 -- 172.21.15.116:0/3960203157 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f971009b460 con 0x56038fc60ca0 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f9702a00640 1 -- 172.21.15.116:0/3960203157 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9710085050 con 0x56038fc60ca0 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f972660e740 10 client.26458 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f972660e740 10 client.26458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f972660e740 20 client.26458 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f972660e740 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc76c80 0x56038fc79070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f972660e740 4 client.26458 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56038fbd32a0 con 0x56038fc76c80 2024-03-04T07:46:01.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.302+0000 7f971be00640 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc76c80 0x56038fc79070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.303+0000 7f971be00640 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc76c80 0x56038fc79070 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f0ed09a0f08c444 server_cookie=5685984e5f8f918a in_seq=0 out_seq=0 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.303+0000 7f9702a00640 10 client.26458 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.303+0000 7f9702a00640 1 -- 172.21.15.116:0/3960203157 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56038fbd32a0 con 0x56038fc76c80 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.303+0000 7f9702a00640 10 client.26458 handle_command_reply: tid=0 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.303+0000 7f972660e740 1 client.26458 shutdown 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 msgr2=0x56038fc6b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc6b170 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f971000eff0 tx=0x7f97100d1880 comp rx=0 tx=0).stop 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 shutdown_connections 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc76c80 0x56038fc79070 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc66800 0x56038fc726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f9702a00640 0 client.26458 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 --2- 172.21.15.116:0/3960203157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc60ca0 0x56038fc6b170 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 >> 172.21.15.116:0/3960203157 conn(0x56038fc5c210 msgr2=0x56038fc5b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 shutdown_connections 2024-03-04T07:46:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 1 -- 172.21.15.116:0/3960203157 wait complete. 2024-03-04T07:46:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.304+0000 7f972660e740 20 client.26458 trim_cache size 0 max 16384 2024-03-04T07:46:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.317+0000 7f972660e740 1 Processor -- start 2024-03-04T07:46:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.317+0000 7f972660e740 1 -- start start 2024-03-04T07:46:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f972660e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc5c210 0x56038fc6ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f972660e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc72680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f972660e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f972660e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1a8cd0 con 0x56038fc61690 2024-03-04T07:46:01.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f972660e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1ab2b0 con 0x56038fc60ca0 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f972660e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1a92d0 con 0x56038fc5c210 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc72680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f9721a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc72680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60060/0 (socket says 172.21.15.116:60060) 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 -- 172.21.15.116:0/2439448632 learned_addr learned my addr 172.21.15.116:0/2439448632 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f9721000640 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc5c210 0x56038fc6ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 -- 172.21.15.116:0/2439448632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc5c210 msgr2=0x56038fc6ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc5c210 0x56038fc6ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 -- 172.21.15.116:0/2439448632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 msgr2=0x56038fc6b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f971be00640 1 -- 172.21.15.116:0/2439448632 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56038fc6a270 con 0x56038fc60ca0 2024-03-04T07:46:01.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.318+0000 7f9721a00640 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f971be00640 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc72680 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f970802c7b0 tx=0x7f970802df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77b3940e7eadfab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f970d600640 1 -- 172.21.15.116:0/2439448632 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970804e020 con 0x56038fc60ca0 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f970d600640 1 -- 172.21.15.116:0/2439448632 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f970804a2f0 con 0x56038fc60ca0 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f970d600640 1 -- 172.21.15.116:0/2439448632 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9708048050 con 0x56038fc60ca0 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f972660e740 1 -- 172.21.15.116:0/2439448632 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 msgr2=0x56038fc72680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f972660e740 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc72680 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f970802c7b0 tx=0x7f970802df90 comp rx=0 tx=0).stop 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f972660e740 1 -- 172.21.15.116:0/2439448632 shutdown_connections 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.319+0000 7f972660e740 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc61690 0x56038fc6b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.320+0000 7f972660e740 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc72680 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.320+0000 7f972660e740 1 --2- 172.21.15.116:0/2439448632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc5c210 0x56038fc6ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.320+0000 7f972660e740 1 -- 172.21.15.116:0/2439448632 >> 172.21.15.116:0/2439448632 conn(0x56038fc62470 msgr2=0x56038fc5c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.320+0000 7f972660e740 1 -- 172.21.15.116:0/2439448632 shutdown_connections 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.320+0000 7f972660e740 1 -- 172.21.15.116:0/2439448632 wait complete. 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.320+0000 7f972660e740 1 -- start start 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.320+0000 7f972660e740 1 Processor -- start 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f972660e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc5c210 0x56038fc70330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f972660e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc70870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f972660e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc61690 0x56038fc71830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f972660e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1a9930 con 0x56038fc5c210 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f972660e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1aac50 con 0x56038fc60ca0 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f972660e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1a9f90 con 0x56038fc61690 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f972660e740 10 client.0 _ll_register_callbacks cb 0x7f97100df070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc5c210 0x56038fc70330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc5c210 0x56038fc70330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35580/0 (socket says 172.21.15.116:35580) 2024-03-04T07:46:01.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 -- 172.21.15.116:0/630881477 learned_addr learned my addr 172.21.15.116:0/630881477 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f971be00640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc70870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721a00640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc61690 0x56038fc71830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 -- 172.21.15.116:0/630881477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc61690 msgr2=0x56038fc71830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc61690 0x56038fc71830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 -- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 msgr2=0x56038fc70870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc70870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 -- 172.21.15.116:0/630881477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56038fc71e20 con 0x56038fc5c210 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f971be00640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc70870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721a00640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc61690 0x56038fc71830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9721000640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc5c210 0x56038fc70330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f97100d16d0 tx=0x7f97100973d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae932221c128feb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.321+0000 7f9702a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f9702a00640 1 -- 172.21.15.116:0/630881477 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97100584a0 con 0x56038fc5c210 2024-03-04T07:46:01.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f9702a00640 1 -- 172.21.15.116:0/630881477 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f97100cbbb0 con 0x56038fc5c210 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56038fbcd5f0 con 0x56038fc5c210 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f9702a00640 1 -- 172.21.15.116:0/630881477 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9710055d80 con 0x56038fc5c210 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 10 client.16425 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56038fcd2a20 con 0x56038fc5c210 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f9702a00640 1 -- 172.21.15.116:0/630881477 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56038fbcd5f0 con 0x56038fc5c210 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f9702a00640 1 -- 172.21.15.116:0/630881477 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9710085070 con 0x56038fc5c210 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 10 client.16425 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 20 client.16425 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc66800 0x56038fcd5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 4 client.16425 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.322+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56038fb4b500 con 0x56038fc66800 2024-03-04T07:46:01.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f971be00640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc66800 0x56038fcd5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f971be00640 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc66800 0x56038fcd5170 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1efdd8aedb429107 server_cookie=ac800c49e3502986 in_seq=0 out_seq=0 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f9702a00640 10 client.16425 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f9702a00640 1 -- 172.21.15.116:0/630881477 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56038fb4b500 con 0x56038fc66800 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f9702a00640 10 client.16425 handle_command_reply: tid=0 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 client.16425 shutdown 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc5c210 msgr2=0x56038fc70330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc5c210 0x56038fc70330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f97100d16d0 tx=0x7f97100973d0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 shutdown_connections 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56038fc66800 0x56038fcd5170 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56038fc61690 0x56038fc71830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56038fc60ca0 0x56038fc70870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f9702a00640 0 client.16425 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 --2- 172.21.15.116:0/630881477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56038fc5c210 0x56038fc70330 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.323+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 >> 172.21.15.116:0/630881477 conn(0x56038fc62470 msgr2=0x56038fc6ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 shutdown_connections 2024-03-04T07:46:01.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f972660e740 1 -- 172.21.15.116:0/630881477 wait complete. 2024-03-04T07:46:01.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f972660e740 20 client.16425 trim_cache size 0 max 16384 2024-03-04T07:46:01.333 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97040a1d80 msgr2=0x7f97040a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97040a1d80 0x7f97040a4240 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x56038fc61ef0 tx=0x7f9714009500 comp rx=0 tx=0).stop 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 msgr2=0x7f971c199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 0x7f971c199ad0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f971000e9c0 tx=0x7f971000ee80 comp rx=0 tx=0).stop 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 shutdown_connections 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f971c1053a0 0x7f971c1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f97040a1d80 0x7f97040a4240 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f971c1049e0 0x7f971c19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 --2- 172.21.15.116:0/2624371851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f971c1037f0 0x7f971c199ad0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.324+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 >> 172.21.15.116:0/2624371851 conn(0x7f971c0fed80 msgr2=0x7f971c107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.325+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 shutdown_connections 2024-03-04T07:46:01.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:01.325+0000 7f9723a00640 1 -- 172.21.15.116:0/2624371851 wait complete. 2024-03-04T07:46:01.336 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:00 smithi133 ceph-mon[36505]: pgmap v558: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 -- 172.21.15.116:0/1252929459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 msgr2=0x7f00881092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 --2- 172.21.15.116:0/1252929459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 0x7f00881092a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0070009900 tx=0x7f007002bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 -- 172.21.15.116:0/1252929459 shutdown_connections 2024-03-04T07:46:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 --2- 172.21.15.116:0/1252929459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00881097e0 0x7f0088110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 --2- 172.21.15.116:0/1252929459 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 0x7f00881092a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 --2- 172.21.15.116:0/1252929459 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0088107c30 0x7f0088108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 -- 172.21.15.116:0/1252929459 >> 172.21.15.116:0/1252929459 conn(0x7f0088076be0 msgr2=0x7f0088079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 -- 172.21.15.116:0/1252929459 shutdown_connections 2024-03-04T07:46:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.700+0000 7f008da00640 1 -- 172.21.15.116:0/1252929459 wait complete. 2024-03-04T07:46:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.701+0000 7f008da00640 1 Processor -- start 2024-03-04T07:46:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f008da00640 1 -- start start 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f008da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088107c30 0x7f008819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f008da00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 0x7f008819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f008da00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00881097e0 0x7f00881a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f008da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881138b0 con 0x7f0088107c30 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f008da00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0088113730 con 0x7f0088108e20 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f008da00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0088113a30 con 0x7f00881097e0 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f0086a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088107c30 0x7f008819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f0086a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088107c30 0x7f008819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35604/0 (socket says 172.21.15.116:35604) 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f007fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 0x7f008819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f0086a00640 1 -- 172.21.15.116:0/94774974 learned_addr learned my addr 172.21.15.116:0/94774974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.702+0000 7f0087400640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00881097e0 0x7f00881a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f0086a00640 1 -- 172.21.15.116:0/94774974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00881097e0 msgr2=0x7f00881a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f0086a00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00881097e0 0x7f00881a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f0086a00640 1 -- 172.21.15.116:0/94774974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 msgr2=0x7f008819e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f0086a00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 0x7f008819e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f0086a00640 1 -- 172.21.15.116:0/94774974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00881a5ab0 con 0x7f0088107c30 2024-03-04T07:46:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f007fe00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 0x7f008819e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f0086a00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088107c30 0x7f008819de90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f007800e9c0 tx=0x7f007800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f007f400640 1 -- 172.21.15.116:0/94774974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007800cd80 con 0x7f0088107c30 2024-03-04T07:46:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f007f400640 1 -- 172.21.15.116:0/94774974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00780035f0 con 0x7f0088107c30 2024-03-04T07:46:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f007f400640 1 -- 172.21.15.116:0/94774974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078010690 con 0x7f0088107c30 2024-03-04T07:46:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00881a5da0 con 0x7f0088107c30 2024-03-04T07:46:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.703+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00881a61b0 con 0x7f0088107c30 2024-03-04T07:46:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f007f400640 1 -- 172.21.15.116:0/94774974 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f00780037a0 con 0x7f0088107c30 2024-03-04T07:46:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 Processor -- start 2024-03-04T07:46:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 -- start start 2024-03-04T07:46:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae53850 0x55778ae53c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae54240 0x55778ae58d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae5b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55778adb9370 con 0x55778ae593b0 2024-03-04T07:46:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55778ad62550 con 0x55778ae53850 2024-03-04T07:46:01.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.707+0000 7f00907f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55778adb5410 con 0x55778ae54240 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae53850 0x55778ae53c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae53850 0x55778ae53c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60090/0 (socket says 172.21.15.116:60090) 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 -- 172.21.15.116:0/1530634879 learned_addr learned my addr 172.21.15.116:0/1530634879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f007fe00640 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae5b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0086a00640 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae54240 0x55778ae58d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 -- 172.21.15.116:0/1530634879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae54240 msgr2=0x55778ae58d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae54240 0x55778ae58d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 -- 172.21.15.116:0/1530634879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 msgr2=0x55778ae5b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae5b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 -- 172.21.15.116:0/1530634879 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55778ad763c0 con 0x55778ae53850 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f0087400640 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae53850 0x55778ae53c70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0080009fd0 tx=0x7f00800118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb07f7ef24387d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.708+0000 7f007f400640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f006c0a1ea0 0x7f006c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f0077e00640 1 -- 172.21.15.116:0/1530634879 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080004d20 con 0x55778ae53850 2024-03-04T07:46:01.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f0077e00640 1 -- 172.21.15.116:0/1530634879 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0080004050 con 0x55778ae53850 2024-03-04T07:46:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f0077e00640 1 -- 172.21.15.116:0/1530634879 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080005260 con 0x55778ae53850 2024-03-04T07:46:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f0086a00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f006c0a1ea0 0x7f006c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f007f400640 1 -- 172.21.15.116:0/94774974 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f00780ce080 con 0x7f0088107c30 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f00907f0740 1 -- 172.21.15.116:0/1530634879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae53850 msgr2=0x55778ae53c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f0086a00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f006c0a1ea0 0x7f006c0a4360 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f00780a3020 tx=0x7f00780045f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.709+0000 7f00907f0740 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae53850 0x55778ae53c70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0080009fd0 tx=0x7f00800118d0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 -- 172.21.15.116:0/1530634879 shutdown_connections 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae5b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae54240 0x55778ae58d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 --2- 172.21.15.116:0/1530634879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae53850 0x55778ae53c70 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 -- 172.21.15.116:0/1530634879 >> 172.21.15.116:0/1530634879 conn(0x55778ae4edc0 msgr2=0x55778ae4f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 -- 172.21.15.116:0/1530634879 shutdown_connections 2024-03-04T07:46:01.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 -- 172.21.15.116:0/1530634879 wait complete. 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 -- start start 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.710+0000 7f00907f0740 1 Processor -- start 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f00907f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f00907f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f00907f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae65100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f00907f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55778ae5c360 con 0x55778ae593b0 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f00907f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55778ad759b0 con 0x55778ae54240 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f00907f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55778ae5c4e0 con 0x55778ae53850 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f00907f0740 10 client.0 _ll_register_callbacks cb 0x55778adb1340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f0087400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae65100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60098/0 (socket says 172.21.15.116:60098) 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 -- 172.21.15.116:0/2858893845 learned_addr learned my addr 172.21.15.116:0/2858893845 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f0087400640 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 -- 172.21.15.116:0/2858893845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 msgr2=0x55778ae5db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 -- 172.21.15.116:0/2858893845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 msgr2=0x55778ae65100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae65100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f007fe00640 1 -- 172.21.15.116:0/2858893845 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55778aace210 con 0x55778ae54240 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.711+0000 7f0087400640 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae65100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f007fe00640 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0070036040 tx=0x7f0070005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2a7c436fde9a158 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f0074c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f0074c00640 1 -- 172.21.15.116:0/2858893845 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070005d50 con 0x55778ae54240 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f0074c00640 1 -- 172.21.15.116:0/2858893845 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f00700046c0 con 0x55778ae54240 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0080004050 con 0x55778ae54240 2024-03-04T07:46:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f0074c00640 1 -- 172.21.15.116:0/2858893845 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00700048c0 con 0x55778ae54240 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f00907f0740 10 client.26564 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55778ab9a620 con 0x55778ae54240 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.712+0000 7f0074c00640 1 -- 172.21.15.116:0/2858893845 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0080004050 con 0x55778ae54240 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f0074c00640 1 -- 172.21.15.116:0/2858893845 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0070044070 con 0x55778ae54240 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f00907f0740 10 client.26564 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f00907f0740 10 client.26564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f00907f0740 20 client.26564 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f00907f0740 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778ae699a0 0x55778ae6bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f00907f0740 4 client.26564 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55778adc68b0 con 0x55778ae699a0 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f0087400640 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778ae699a0 0x55778ae6bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f0087400640 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778ae699a0 0x55778ae6bd90 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38c2c3d61283db61 server_cookie=cee867553c11c24b in_seq=0 out_seq=0 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.713+0000 7f0074c00640 10 client.26564 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f0074c00640 1 -- 172.21.15.116:0/2858893845 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55778adc68b0 con 0x55778ae699a0 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f0074c00640 10 client.26564 handle_command_reply: tid=0 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 client.26564 shutdown 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 msgr2=0x55778ae5e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0070036040 tx=0x7f0070005250 comp rx=0 tx=0).stop 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 shutdown_connections 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778ae699a0 0x55778ae6bd90 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae593b0 0x55778ae65100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 --2- 172.21.15.116:0/2858893845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.714+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 >> 172.21.15.116:0/2858893845 conn(0x55778ae4edc0 msgr2=0x55778ae4dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.715+0000 7f0074c00640 0 client.26564 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.715+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 shutdown_connections 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.715+0000 7f00907f0740 1 -- 172.21.15.116:0/2858893845 wait complete. 2024-03-04T07:46:01.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.715+0000 7f00907f0740 20 client.26564 trim_cache size 0 max 16384 2024-03-04T07:46:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.729+0000 7f00907f0740 1 Processor -- start 2024-03-04T07:46:01.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.729+0000 7f00907f0740 1 -- start start 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f00907f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f00907f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f00907f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae593b0 0x55778ae650c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f00907f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881ad030 con 0x55778ae53850 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f00907f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881af610 con 0x55778ae54240 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f00907f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881ad630 con 0x55778ae593b0 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60102/0 (socket says 172.21.15.116:60102) 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 -- 172.21.15.116:0/3448443030 learned_addr learned my addr 172.21.15.116:0/3448443030 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f0087400640 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 -- 172.21.15.116:0/3448443030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae593b0 msgr2=0x55778ae650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae593b0 0x55778ae650c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 -- 172.21.15.116:0/3448443030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae53850 msgr2=0x55778ae5db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.730+0000 7f007fe00640 1 -- 172.21.15.116:0/3448443030 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55778ae54ad0 con 0x55778ae54240 2024-03-04T07:46:01.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f007fe00640 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0070006010 tx=0x7f0070004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24abd03e053a94d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f0077400640 1 -- 172.21.15.116:0/3448443030 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070005d50 con 0x55778ae54240 2024-03-04T07:46:01.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f0077400640 1 -- 172.21.15.116:0/3448443030 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0070055020 con 0x55778ae54240 2024-03-04T07:46:01.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f0077400640 1 -- 172.21.15.116:0/3448443030 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070004a60 con 0x55778ae54240 2024-03-04T07:46:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f00907f0740 1 -- 172.21.15.116:0/3448443030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 msgr2=0x55778ae5e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f00907f0740 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0070006010 tx=0x7f0070004500 comp rx=0 tx=0).stop 2024-03-04T07:46:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f00907f0740 1 -- 172.21.15.116:0/3448443030 shutdown_connections 2024-03-04T07:46:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f00907f0740 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae593b0 0x55778ae650c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f00907f0740 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae54240 0x55778ae5e0c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f00907f0740 1 --2- 172.21.15.116:0/3448443030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae53850 0x55778ae5db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.731+0000 7f00907f0740 1 -- 172.21.15.116:0/3448443030 >> 172.21.15.116:0/3448443030 conn(0x55778ae4edc0 msgr2=0x55778ae4f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 -- 172.21.15.116:0/3448443030 shutdown_connections 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 -- 172.21.15.116:0/3448443030 wait complete. 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 -- start start 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 Processor -- start 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae62c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae54240 0x55778ae631d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae593b0 0x55778ae641e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881adc90 con 0x55778ae54240 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881aefb0 con 0x55778ae593b0 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881ae2f0 con 0x55778ae53850 2024-03-04T07:46:01.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.732+0000 7f00907f0740 10 client.0 _ll_register_callbacks cb 0x7f0070009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae54240 0x55778ae631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae54240 0x55778ae631d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35634/0 (socket says 172.21.15.116:35634) 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 -- 172.21.15.116:0/950708390 learned_addr learned my addr 172.21.15.116:0/950708390 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f0087400640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae593b0 0x55778ae641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f0087400640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae62c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 -- 172.21.15.116:0/950708390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 msgr2=0x55778ae62c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae62c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 -- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae593b0 msgr2=0x55778ae641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae593b0 0x55778ae641e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 -- 172.21.15.116:0/950708390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55778ae647d0 con 0x55778ae54240 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f0087400640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae62c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f007fe00640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae54240 0x55778ae631d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0070005290 tx=0x7f0070005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380ae3aa45a6cb92 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.733+0000 7f0074c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:01.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f0074c00640 1 -- 172.21.15.116:0/950708390 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070005d50 con 0x55778ae54240 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f0074c00640 1 -- 172.21.15.116:0/950708390 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0070003800 con 0x55778ae54240 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0070055020 con 0x55778ae54240 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f0074c00640 1 -- 172.21.15.116:0/950708390 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0070003a00 con 0x55778ae54240 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 10 client.16443 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55778aec56d0 con 0x55778ae54240 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f0074c00640 1 -- 172.21.15.116:0/950708390 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0070055020 con 0x55778ae54240 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f0074c00640 1 -- 172.21.15.116:0/950708390 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0070044070 con 0x55778ae54240 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 10 client.16443 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 20 client.16443 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778aec5ff0 0x55778aec8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 4 client.16443 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:46:01.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.734+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55778adc0c00 con 0x55778aec5ff0 2024-03-04T07:46:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.735+0000 7f0087400640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778aec5ff0 0x55778aec8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.735+0000 7f0087400640 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778aec5ff0 0x55778aec8410 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=968dc77992113b98 server_cookie=461d4877408d25ea in_seq=0 out_seq=0 2024-03-04T07:46:01.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.735+0000 7f0074c00640 10 client.16443 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f0074c00640 1 -- 172.21.15.116:0/950708390 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55778adc0c00 con 0x55778aec5ff0 2024-03-04T07:46:01.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f0074c00640 10 client.16443 handle_command_reply: tid=0 2024-03-04T07:46:01.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 client.16443 shutdown 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae54240 msgr2=0x55778ae631d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae54240 0x55778ae631d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0070005290 tx=0x7f0070005780 comp rx=0 tx=0).stop 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 shutdown_connections 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55778aec5ff0 0x55778aec8410 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55778ae593b0 0x55778ae641e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55778ae54240 0x55778ae631d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 --2- 172.21.15.116:0/950708390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55778ae53850 0x55778ae62c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 >> 172.21.15.116:0/950708390 conn(0x55778ae4edc0 msgr2=0x55778ae52020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.750+0000 7f0074c00640 0 client.16443 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.751+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 shutdown_connections 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.751+0000 7f00907f0740 1 -- 172.21.15.116:0/950708390 wait complete. 2024-03-04T07:46:01.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.751+0000 7f00907f0740 20 client.16443 trim_cache size 0 max 16384 2024-03-04T07:46:01.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:46:01.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:46:01.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:46:01.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f006c0a1ea0 msgr2=0x7f006c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f006c0a1ea0 0x7f006c0a4360 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f00780a3020 tx=0x7f00780045f0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088107c30 msgr2=0x7f008819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088107c30 0x7f008819de90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f007800e9c0 tx=0x7f007800ee80 comp rx=0 tx=0).stop 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 shutdown_connections 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f006c0a1ea0 0x7f006c0a4360 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00881097e0 0x7f00881a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0088108e20 0x7f008819e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 --2- 172.21.15.116:0/94774974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0088107c30 0x7f008819de90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 >> 172.21.15.116:0/94774974 conn(0x7f0088076be0 msgr2=0x7f008810c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:01.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 shutdown_connections 2024-03-04T07:46:01.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:01.752+0000 7f008da00640 1 -- 172.21.15.116:0/94774974 wait complete. 2024-03-04T07:46:02.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:01 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3379241851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:02.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:01 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3379241851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:02.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:01 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3379241851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:02.341 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:46:02.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:46:02.341 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:46:02.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 -- 172.21.15.116:0/944639572 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541053a0 msgr2=0x7fca5410bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 --2- 172.21.15.116:0/944639572 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541053a0 0x7fca5410bc30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009900 tx=0x7fca4c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:46:02.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 -- 172.21.15.116:0/944639572 shutdown_connections 2024-03-04T07:46:02.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 --2- 172.21.15.116:0/944639572 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541053a0 0x7fca5410bc30 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 --2- 172.21.15.116:0/944639572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca541049e0 0x7fca54104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 --2- 172.21.15.116:0/944639572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca541037f0 0x7fca54103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 -- 172.21.15.116:0/944639572 >> 172.21.15.116:0/944639572 conn(0x7fca540fed80 msgr2=0x7fca541011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 -- 172.21.15.116:0/944639572 shutdown_connections 2024-03-04T07:46:02.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.564+0000 7fca5a400640 1 -- 172.21.15.116:0/944639572 wait complete. 2024-03-04T07:46:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 Processor -- start 2024-03-04T07:46:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 -- start start 2024-03-04T07:46:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541037f0 0x7fca54199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca541049e0 0x7fca54199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca541053a0 0x7fca541a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5410f4c0 con 0x7fca541053a0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5410f340 con 0x7fca541037f0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca5a400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5410f640 con 0x7fca541049e0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca53e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca541053a0 0x7fca541a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca53400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541037f0 0x7fca54199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.565+0000 7fca53400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541037f0 0x7fca54199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60134/0 (socket says 172.21.15.116:60134) 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca53400640 1 -- 172.21.15.116:0/1868312997 learned_addr learned my addr 172.21.15.116:0/1868312997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca4aa00640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca541049e0 0x7fca54199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca53400640 1 -- 172.21.15.116:0/1868312997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca541049e0 msgr2=0x7fca54199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca53400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca541049e0 0x7fca54199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca53400640 1 -- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca541053a0 msgr2=0x7fca541a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca53400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca541053a0 0x7fca541a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca53400640 1 -- 172.21.15.116:0/1868312997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca541a16d0 con 0x7fca541037f0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca53400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541037f0 0x7fca54199ab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fca4400eaa0 tx=0x7fca4400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.566+0000 7fca50c00640 1 -- 172.21.15.116:0/1868312997 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44016070 con 0x7fca541037f0 2024-03-04T07:46:02.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.567+0000 7fca50c00640 1 -- 172.21.15.116:0/1868312997 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca44003e50 con 0x7fca541037f0 2024-03-04T07:46:02.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.567+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca541a19c0 con 0x7fca541037f0 2024-03-04T07:46:02.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.567+0000 7fca50c00640 1 -- 172.21.15.116:0/1868312997 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca440116d0 con 0x7fca541037f0 2024-03-04T07:46:02.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.567+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca541a1ec0 con 0x7fca541037f0 2024-03-04T07:46:02.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 Processor -- start 2024-03-04T07:46:02.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca50c00640 1 -- 172.21.15.116:0/1868312997 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fca44019070 con 0x7fca541037f0 2024-03-04T07:46:02.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 -- start start 2024-03-04T07:46:02.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841bf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6840d2e20 con 0x55b6841bfac0 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b684124a20 con 0x55b6841bf0d0 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6840cdb00 con 0x55b6841c4c30 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca53400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.570+0000 7fca53e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841bf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35664/0 (socket says 172.21.15.116:35664) 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 -- 172.21.15.116:0/1363924974 learned_addr learned my addr 172.21.15.116:0/1363924974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca4aa00640 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 -- 172.21.15.116:0/1363924974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 msgr2=0x55b6841c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 -- 172.21.15.116:0/1363924974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 msgr2=0x55b6841bf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841bf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 -- 172.21.15.116:0/1363924974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6840e2e10 con 0x55b6841bfac0 2024-03-04T07:46:02.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca53400640 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c45c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fca440d1f00 tx=0x7fca440d4820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00090c08a40110d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca50c00640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca380a1d80 0x7fca380a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.571+0000 7fca48c00640 1 -- 172.21.15.116:0/1363924974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4409a040 con 0x55b6841bfac0 2024-03-04T07:46:02.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca48c00640 1 -- 172.21.15.116:0/1363924974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca44084180 con 0x55b6841bfac0 2024-03-04T07:46:02.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca53e00640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca380a1d80 0x7fca380a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca48c00640 1 -- 172.21.15.116:0/1363924974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4409a370 con 0x55b6841bfac0 2024-03-04T07:46:02.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca50c00640 1 -- 172.21.15.116:0/1868312997 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fca440cde50 con 0x7fca541037f0 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1363924974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 msgr2=0x55b6841c45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c45c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fca440d1f00 tx=0x7fca440d4820 comp rx=0 tx=0).stop 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca53e00640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca380a1d80 0x7fca380a4240 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fca4c036040 tx=0x7fca4c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1363924974 shutdown_connections 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c45c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1363924974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841bf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.572+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1363924974 >> 172.21.15.116:0/1363924974 conn(0x55b6841ba640 msgr2=0x55b6841baa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.573+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1363924974 shutdown_connections 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.573+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1363924974 wait complete. 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.573+0000 7fca5d1f2740 1 -- start start 2024-03-04T07:46:02.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.573+0000 7fca5d1f2740 1 Processor -- start 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6840e0f40 con 0x55b6841bfac0 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b684120ac0 con 0x55b6841bf0d0 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6841c7be0 con 0x55b6841c4c30 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca5d1f2740 10 client.0 _ll_register_callbacks cb 0x55b68411faa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca4aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60152/0 (socket says 172.21.15.116:60152) 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 -- 172.21.15.116:0/348550235 learned_addr learned my addr 172.21.15.116:0/348550235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53e00640 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 -- 172.21.15.116:0/348550235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 msgr2=0x55b6841d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 -- 172.21.15.116:0/348550235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 msgr2=0x55b6841c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.574+0000 7fca53400640 1 -- 172.21.15.116:0/348550235 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b683f017e0 con 0x55b6841bf0d0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca53400640 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841c93e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fca4409ab80 tx=0x7fca44005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f969d539c0b822bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca41600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca41600640 1 -- 172.21.15.116:0/348550235 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44020070 con 0x55b6841bf0d0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca41600640 1 -- 172.21.15.116:0/348550235 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fca440cba20 con 0x55b6841bf0d0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca44084180 con 0x55b6841bf0d0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca41600640 1 -- 172.21.15.116:0/348550235 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca4409a550 con 0x55b6841bf0d0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca5d1f2740 10 client.26582 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b683f61f70 con 0x55b6841bf0d0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca41600640 1 -- 172.21.15.116:0/348550235 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca44084180 con 0x55b6841bf0d0 2024-03-04T07:46:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.575+0000 7fca41600640 1 -- 172.21.15.116:0/348550235 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fca4400f040 con 0x55b6841bf0d0 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca5d1f2740 10 client.26582 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca5d1f2740 10 client.26582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca5d1f2740 20 client.26582 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6841d4e50 0x55b6841d7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca5d1f2740 4 client.26582 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b684131f60 con 0x55b6841d4e50 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca4aa00640 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6841d4e50 0x55b6841d7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca41600640 10 client.26582 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.576+0000 7fca4aa00640 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6841d4e50 0x55b6841d7240 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ab8eb18f9c26f59 server_cookie=7efd4d7453dccbae in_seq=0 out_seq=0 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.577+0000 7fca41600640 1 -- 172.21.15.116:0/348550235 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b684131f60 con 0x55b6841d4e50 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.577+0000 7fca41600640 10 client.26582 handle_command_reply: tid=0 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.577+0000 7fca5d1f2740 1 client.26582 shutdown 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 msgr2=0x55b6841c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841c93e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fca4409ab80 tx=0x7fca44005ac0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 shutdown_connections 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6841d4e50 0x55b6841d7240 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841c4c30 0x55b6841d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/348550235 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841bf0d0 0x55b6841c93e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca41600640 0 client.26582 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 >> 172.21.15.116:0/348550235 conn(0x55b6841ba640 msgr2=0x55b6841bb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 shutdown_connections 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 1 -- 172.21.15.116:0/348550235 wait complete. 2024-03-04T07:46:02.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.578+0000 7fca5d1f2740 20 client.26582 trim_cache size 0 max 16384 2024-03-04T07:46:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.595+0000 7fca5d1f2740 1 Processor -- start 2024-03-04T07:46:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca5d1f2740 1 -- start start 2024-03-04T07:46:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bf0d0 0x55b6841c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bfac0 0x55b6841d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541a8cf0 con 0x55b6841bf0d0 2024-03-04T07:46:02.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541ab2d0 con 0x55b6841c4c30 2024-03-04T07:46:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541a92f0 con 0x55b6841bfac0 2024-03-04T07:46:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca53400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bf0d0 0x55b6841c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca53e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca4aa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bfac0 0x55b6841d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca53e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841b98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60158/0 (socket says 172.21.15.116:60158) 2024-03-04T07:46:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca53e00640 1 -- 172.21.15.116:0/1852686176 learned_addr learned my addr 172.21.15.116:0/1852686176 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.596+0000 7fca53400640 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bf0d0 0x55b6841c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35682/0 (socket says 172.21.15.116:35682) 2024-03-04T07:46:02.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca4aa00640 1 -- 172.21.15.116:0/1852686176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 msgr2=0x55b6841b98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca4aa00640 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841b98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca4aa00640 1 -- 172.21.15.116:0/1852686176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bf0d0 msgr2=0x55b6841c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca4aa00640 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bf0d0 0x55b6841c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca4aa00640 1 -- 172.21.15.116:0/1852686176 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b683df9170 con 0x55b6841bfac0 2024-03-04T07:46:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca53e00640 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca4aa00640 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bfac0 0x55b6841d08f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x55b6841d79b0 tx=0x7fca3c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bca28d68b55c6a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca43e00640 1 -- 172.21.15.116:0/1852686176 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c003be0 con 0x55b6841bfac0 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca43e00640 1 -- 172.21.15.116:0/1852686176 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca3c003d80 con 0x55b6841bfac0 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.597+0000 7fca43e00640 1 -- 172.21.15.116:0/1852686176 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c010b90 con 0x55b6841bfac0 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1852686176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bfac0 msgr2=0x55b6841d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bfac0 0x55b6841d08f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x55b6841d79b0 tx=0x7fca3c005f90 comp rx=0 tx=0).stop 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1852686176 shutdown_connections 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bfac0 0x55b6841d08f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/1852686176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bf0d0 0x55b6841c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1852686176 >> 172.21.15.116:0/1852686176 conn(0x55b6841ba640 msgr2=0x55b6841b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1852686176 shutdown_connections 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 -- 172.21.15.116:0/1852686176 wait complete. 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.598+0000 7fca5d1f2740 1 -- start start 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 1 Processor -- start 2024-03-04T07:46:02.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bf0d0 0x55b6841ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541a9950 con 0x55b6841bfac0 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541aac70 con 0x55b6841c4c30 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541a9fb0 con 0x55b6841bf0d0 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca5d1f2740 10 client.0 _ll_register_callbacks cb 0x7fca4c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca4aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca4aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35688/0 (socket says 172.21.15.116:35688) 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca4aa00640 1 -- 172.21.15.116:0/3502772373 learned_addr learned my addr 172.21.15.116:0/3502772373 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca53e00640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.599+0000 7fca53400640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bf0d0 0x55b6841ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca53e00640 1 -- 172.21.15.116:0/3502772373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bf0d0 msgr2=0x55b6841ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca53e00640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bf0d0 0x55b6841ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca53e00640 1 -- 172.21.15.116:0/3502772373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 msgr2=0x55b6841cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca53e00640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca53e00640 1 -- 172.21.15.116:0/3502772373 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6841d0200 con 0x55b6841c4c30 2024-03-04T07:46:02.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca4aa00640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca53e00640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841cfc10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fca4c005ab0 tx=0x7fca4c02bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=892179e5c510050b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca41600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:02.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca41600640 1 -- 172.21.15.116:0/3502772373 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c057060 con 0x55b6841c4c30 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.600+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca3c003d80 con 0x55b6841c4c30 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca41600640 1 -- 172.21.15.116:0/3502772373 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fca4c047470 con 0x55b6841c4c30 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca41600640 1 -- 172.21.15.116:0/3502772373 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca4c003e10 con 0x55b6841c4c30 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 10 client.26588 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b684230e10 con 0x55b6841c4c30 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca41600640 1 -- 172.21.15.116:0/3502772373 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca3c003d80 con 0x55b6841c4c30 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca41600640 1 -- 172.21.15.116:0/3502772373 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fca4c045040 con 0x55b6841c4c30 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 10 client.26588 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 10 client.26588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 20 client.26588 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b684231600 0x55b684233a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 4 client.26588 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.601+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b68412c2b0 con 0x55b684231600 2024-03-04T07:46:02.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.602+0000 7fca53400640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b684231600 0x55b684233a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.602+0000 7fca53400640 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b684231600 0x55b684233a80 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de8cee61f2bc8319 server_cookie=f4c7d1920204176c in_seq=0 out_seq=0 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.602+0000 7fca41600640 10 client.26588 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca41600640 1 -- 172.21.15.116:0/3502772373 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b68412c2b0 con 0x55b684231600 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca41600640 10 client.26588 handle_command_reply: tid=0 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 client.26588 shutdown 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 msgr2=0x55b6841cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841cfc10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fca4c005ab0 tx=0x7fca4c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 shutdown_connections 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b684231600 0x55b684233a80 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6841c4c30 0x55b6841cfc10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6841bfac0 0x55b6841cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 --2- 172.21.15.116:0/3502772373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b6841bf0d0 0x55b6841ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca41600640 0 client.26588 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.603+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 >> 172.21.15.116:0/3502772373 conn(0x55b6841ba640 msgr2=0x55b6841c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 shutdown_connections 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5d1f2740 1 -- 172.21.15.116:0/3502772373 wait complete. 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5d1f2740 20 client.26588 trim_cache size 0 max 16384 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "0acc9642-6b7e-4326-bef0-a2d89e7647f1", 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:46:02.609 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca380a1d80 msgr2=0x7fca380a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca380a1d80 0x7fca380a4240 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fca4c036040 tx=0x7fca4c02cf90 comp rx=0 tx=0).stop 2024-03-04T07:46:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541037f0 msgr2=0x7fca54199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541037f0 0x7fca54199ab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fca4400eaa0 tx=0x7fca4400ef60 comp rx=0 tx=0).stop 2024-03-04T07:46:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 shutdown_connections 2024-03-04T07:46:02.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca541053a0 0x7fca541a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca380a1d80 0x7fca380a4240 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca541049e0 0x7fca54199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 --2- 172.21.15.116:0/1868312997 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca541037f0 0x7fca54199ab0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.604+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 >> 172.21.15.116:0/1868312997 conn(0x7fca540fed80 msgr2=0x7fca54107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.605+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 shutdown_connections 2024-03-04T07:46:02.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.605+0000 7fca5a400640 1 -- 172.21.15.116:0/1868312997 wait complete. 2024-03-04T07:46:02.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:46:02.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.838+0000 7f84a6400640 1 -- 172.21.15.116:0/1125358664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01053a0 msgr2=0x7f84a010bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.838+0000 7f84a6400640 1 --2- 172.21.15.116:0/1125358664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01053a0 0x7f84a010bc30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8498009900 tx=0x7f849802bc50 comp rx=0 tx=0).stop 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.839+0000 7f84a6400640 1 -- 172.21.15.116:0/1125358664 shutdown_connections 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.839+0000 7f84a6400640 1 --2- 172.21.15.116:0/1125358664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01053a0 0x7f84a010bc30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.839+0000 7f84a6400640 1 --2- 172.21.15.116:0/1125358664 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84a01049e0 0x7f84a0104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.839+0000 7f84a6400640 1 --2- 172.21.15.116:0/1125358664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 0x7f84a0103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.839+0000 7f84a6400640 1 -- 172.21.15.116:0/1125358664 >> 172.21.15.116:0/1125358664 conn(0x7f84a00fed80 msgr2=0x7f84a01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.839+0000 7f84a6400640 1 -- 172.21.15.116:0/1125358664 shutdown_connections 2024-03-04T07:46:02.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.839+0000 7f84a6400640 1 -- 172.21.15.116:0/1125358664 wait complete. 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.840+0000 7f84a6400640 1 Processor -- start 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.840+0000 7f84a6400640 1 -- start start 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f84a6400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 0x7f84a0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f84a6400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01049e0 0x7f84a0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f84a6400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84a01053a0 0x7f84a01a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f84a6400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a010f4a0 con 0x7f84a01037f0 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f84a6400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a010f320 con 0x7f84a01049e0 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f84a6400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a010f620 con 0x7f84a01053a0 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 0x7f84a0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01049e0 0x7f84a0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 0x7f84a0199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35700/0 (socket says 172.21.15.116:35700) 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01049e0 0x7f84a0199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60178/0 (socket says 172.21.15.116:60178) 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849f400640 1 -- 172.21.15.116:0/2354166585 learned_addr learned my addr 172.21.15.116:0/2354166585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849fe00640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84a01053a0 0x7f84a01a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849ea00640 1 -- 172.21.15.116:0/2354166585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84a01053a0 msgr2=0x7f84a01a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849ea00640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84a01053a0 0x7f84a01a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849ea00640 1 -- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 msgr2=0x7f84a0199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849ea00640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 0x7f84a0199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849ea00640 1 -- 172.21.15.116:0/2354166585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a01a16d0 con 0x7f84a01049e0 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.841+0000 7f849f400640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 0x7f84a0199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.842+0000 7f849ea00640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01049e0 0x7f84a0199ff0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8488005fd0 tx=0x7f848800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.842+0000 7f8497e00640 1 -- 172.21.15.116:0/2354166585 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488009c30 con 0x7f84a01049e0 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.842+0000 7f8497e00640 1 -- 172.21.15.116:0/2354166585 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8488009dd0 con 0x7f84a01049e0 2024-03-04T07:46:02.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.842+0000 7f8497e00640 1 -- 172.21.15.116:0/2354166585 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488010460 con 0x7f84a01049e0 2024-03-04T07:46:02.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.842+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a01a19c0 con 0x7f84a01049e0 2024-03-04T07:46:02.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.842+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a01a1ec0 con 0x7f84a01049e0 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.845+0000 7f84a917f740 1 Processor -- start 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.845+0000 7f8497e00640 1 -- 172.21.15.116:0/2354166585 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8488003f50 con 0x7f84a01049e0 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.845+0000 7f84a917f740 1 -- start start 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.845+0000 7f84a917f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bb7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f84a917f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bbd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f84a917f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bbd6b0 0x558521bbfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f84a917f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558521ae5090 con 0x558521bb7b50 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f84a917f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558521b1cd30 con 0x558521bb8540 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f84a917f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558521ac5a30 con 0x558521bbd6b0 2024-03-04T07:46:02.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bb7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bbd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bb7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35710/0 (socket says 172.21.15.116:35710) 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849fe00640 1 -- 172.21.15.116:0/2090651172 learned_addr learned my addr 172.21.15.116:0/2090651172 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849ea00640 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bbd6b0 0x558521bbfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849f400640 1 -- 172.21.15.116:0/2090651172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bbd6b0 msgr2=0x558521bbfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849f400640 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bbd6b0 0x558521bbfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849f400640 1 -- 172.21.15.116:0/2090651172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 msgr2=0x558521bb7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849f400640 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bb7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849f400640 1 -- 172.21.15.116:0/2090651172 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585218fb7d0 con 0x558521bb8540 2024-03-04T07:46:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.846+0000 7f849ea00640 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bbd6b0 0x558521bbfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:02.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f849f400640 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bbd040 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8490011830 tx=0x7f8490011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=383d1f22889a6475 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f8497e00640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f84840a1e00 0x7f84840a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f8494c00640 1 -- 172.21.15.116:0/2090651172 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490004d20 con 0x558521bb8540 2024-03-04T07:46:02.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f8494c00640 1 -- 172.21.15.116:0/2090651172 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8490003e50 con 0x558521bb8540 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f8494c00640 1 -- 172.21.15.116:0/2090651172 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490004480 con 0x558521bb8540 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f8497e00640 1 -- 172.21.15.116:0/2354166585 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f84880cdbf0 con 0x7f84a01049e0 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f849fe00640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f84840a1e00 0x7f84840a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f84a917f740 1 -- 172.21.15.116:0/2090651172 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 msgr2=0x558521bbd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.847+0000 7f84a917f740 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bbd040 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8490011830 tx=0x7f8490011cf0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 -- 172.21.15.116:0/2090651172 shutdown_connections 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bbd6b0 0x558521bbfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bbd040 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 --2- 172.21.15.116:0/2090651172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bb7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 -- 172.21.15.116:0/2090651172 >> 172.21.15.116:0/2090651172 conn(0x558521bb30c0 msgr2=0x558521bb34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 -- 172.21.15.116:0/2090651172 shutdown_connections 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 -- 172.21.15.116:0/2090651172 wait complete. 2024-03-04T07:46:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f849fe00640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f84840a1e00 0x7f84840a42c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f84980096c0 tx=0x7f849802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 -- start start 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.848+0000 7f84a917f740 1 Processor -- start 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f84a917f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bc1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f84a917f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb8540 0x558521bc2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f84a917f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f84a917f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558521abb0c0 con 0x558521bb8540 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f84a917f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558521b18dd0 con 0x558521bbd6b0 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f84a917f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558521bc0660 con 0x558521bb7b50 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f84a917f740 10 client.0 _ll_register_callbacks cb 0x558521ad8860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb8540 0x558521bc2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bc1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bc1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43086/0 (socket says 172.21.15.116:43086) 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb8540 0x558521bc2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35724/0 (socket says 172.21.15.116:35724) 2024-03-04T07:46:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 -- 172.21.15.116:0/321409301 learned_addr learned my addr 172.21.15.116:0/321409301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849fe00640 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 -- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 msgr2=0x558521bc9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 -- 172.21.15.116:0/321409301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb8540 msgr2=0x558521bc2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb8540 0x558521bc2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.849+0000 7f849f400640 1 -- 172.21.15.116:0/321409301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5585218313f0 con 0x558521bb7b50 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f849ea00640 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb8540 0x558521bc2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f849f400640 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bc1ef0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8490009a30 tx=0x7f849000fc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2569f08806b5c8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f848d600640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f848d600640 1 -- 172.21.15.116:0/321409301 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849001e040 con 0x558521bb7b50 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f848d600640 1 -- 172.21.15.116:0/321409301 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8490011240 con 0x558521bb7b50 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8490003e50 con 0x558521bb7b50 2024-03-04T07:46:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f848d600640 1 -- 172.21.15.116:0/321409301 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8490011440 con 0x558521bb7b50 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f84a917f740 10 client.26509 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.850+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5585217f10f0 con 0x558521bb7b50 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f848d600640 1 -- 172.21.15.116:0/321409301 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8490003e50 con 0x558521bb7b50 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f848d600640 1 -- 172.21.15.116:0/321409301 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8490018070 con 0x558521bb7b50 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f84a917f740 10 client.26509 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f84a917f740 10 client.26509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f84a917f740 20 client.26509 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f84a917f740 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521bcd720 0x558521bcfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f84a917f740 4 client.26509 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558521b2a270 con 0x558521bcd720 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.851+0000 7f849ea00640 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521bcd720 0x558521bcfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.852+0000 7f849ea00640 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521bcd720 0x558521bcfb70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ba6a38d83de767b server_cookie=1a7410d70a2b3fef in_seq=0 out_seq=0 2024-03-04T07:46:02.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.852+0000 7f848d600640 10 client.26509 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.852+0000 7f848d600640 1 -- 172.21.15.116:0/321409301 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558521b2a270 con 0x558521bcd720 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.852+0000 7f848d600640 10 client.26509 handle_command_reply: tid=0 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.852+0000 7f84a917f740 1 client.26509 shutdown 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 msgr2=0x558521bc1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bc1ef0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8490009a30 tx=0x7f849000fc40 comp rx=0 tx=0).stop 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 shutdown_connections 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521bcd720 0x558521bcfb70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb8540 0x558521bc2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f848d600640 0 client.26509 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 --2- 172.21.15.116:0/321409301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bc1ef0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 >> 172.21.15.116:0/321409301 conn(0x558521bb30c0 msgr2=0x558521bb2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 shutdown_connections 2024-03-04T07:46:02.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 1 -- 172.21.15.116:0/321409301 wait complete. 2024-03-04T07:46:02.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.853+0000 7f84a917f740 20 client.26509 trim_cache size 0 max 16384 2024-03-04T07:46:02.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.870+0000 7f84a917f740 1 Processor -- start 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.870+0000 7f84a917f740 1 -- start start 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f84a917f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bd4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f84a917f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bc9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f84a917f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bbd6b0 0x558521bd4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f84a917f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01a8c90 con 0x558521bbd6b0 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f84a917f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01ab270 con 0x558521bb8540 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f84a917f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01a9290 con 0x558521bb7b50 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bc9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bc9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60214/0 (socket says 172.21.15.116:60214) 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849ea00640 1 -- 172.21.15.116:0/1847074108 learned_addr learned my addr 172.21.15.116:0/1847074108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849f400640 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bd4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849fe00640 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bbd6b0 0x558521bd4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849ea00640 1 -- 172.21.15.116:0/1847074108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 msgr2=0x558521bd4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849ea00640 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bd4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849ea00640 1 -- 172.21.15.116:0/1847074108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bbd6b0 msgr2=0x558521bd4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.871+0000 7f849ea00640 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bbd6b0 0x558521bd4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.872+0000 7f849ea00640 1 -- 172.21.15.116:0/1847074108 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558521bbcc40 con 0x558521bb8540 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.872+0000 7f849ea00640 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bc9430 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8488097360 tx=0x7f84880cb580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed790889947ff05 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.872+0000 7f848fe00640 1 -- 172.21.15.116:0/1847074108 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488020040 con 0x558521bb8540 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.872+0000 7f848fe00640 1 -- 172.21.15.116:0/1847074108 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f848809ea30 con 0x558521bb8540 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.872+0000 7f848fe00640 1 -- 172.21.15.116:0/1847074108 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488097000 con 0x558521bb8540 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 -- 172.21.15.116:0/1847074108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 msgr2=0x558521bc9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bc9430 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8488097360 tx=0x7f84880cb580 comp rx=0 tx=0).stop 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 -- 172.21.15.116:0/1847074108 shutdown_connections 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bbd6b0 0x558521bd4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bb8540 0x558521bc9430 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 --2- 172.21.15.116:0/1847074108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb7b50 0x558521bd4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 -- 172.21.15.116:0/1847074108 >> 172.21.15.116:0/1847074108 conn(0x558521bb30c0 msgr2=0x558521bb6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 -- 172.21.15.116:0/1847074108 shutdown_connections 2024-03-04T07:46:02.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 -- 172.21.15.116:0/1847074108 wait complete. 2024-03-04T07:46:02.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 -- start start 2024-03-04T07:46:02.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.873+0000 7f84a917f740 1 Processor -- start 2024-03-04T07:46:02.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f84a917f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bc70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f84a917f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb8540 0x558521bc7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f84a917f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f84a917f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01a98f0 con 0x558521bb7b50 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f84a917f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01aac10 con 0x558521bbd6b0 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f84a917f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01a9f50 con 0x558521bb8540 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f84a917f740 10 client.0 _ll_register_callbacks cb 0x7f8490011680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bc70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bc70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35738/0 (socket says 172.21.15.116:35738) 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 -- 172.21.15.116:0/2061179862 learned_addr learned my addr 172.21.15.116:0/2061179862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849ea00640 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb8540 0x558521bc7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849fe00640 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 -- 172.21.15.116:0/2061179862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb8540 msgr2=0x558521bc7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb8540 0x558521bc7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 -- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 msgr2=0x558521bc8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.874+0000 7f849f400640 1 -- 172.21.15.116:0/2061179862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558521bc8c20 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f849f400640 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bc70e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8490005050 tx=0x7f84900052c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b51c7afff0e843 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f848d600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f848d600640 1 -- 172.21.15.116:0/2061179862 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84900059c0 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f848d600640 1 -- 172.21.15.116:0/2061179862 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f849002e3f0 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f848809ea30 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f848d600640 1 -- 172.21.15.116:0/2061179862 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f849002e5f0 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f84a917f740 10 client.16479 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558521c29580 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f848d600640 1 -- 172.21.15.116:0/2061179862 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f848809ea30 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.875+0000 7f848d600640 1 -- 172.21.15.116:0/2061179862 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8490040050 con 0x558521bb7b50 2024-03-04T07:46:02.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f84a917f740 10 client.16479 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f84a917f740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f84a917f740 20 client.16479 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f84a917f740 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521c29e40 0x558521c2c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f84a917f740 4 client.16479 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558521b245c0 con 0x558521c29e40 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f849ea00640 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521c29e40 0x558521c2c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f849ea00640 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521c29e40 0x558521c2c260 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d9d03666822dc58 server_cookie=15bc92f547b6736a in_seq=0 out_seq=0 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f848d600640 10 client.16479 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f848d600640 1 -- 172.21.15.116:0/2061179862 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558521b245c0 con 0x558521c29e40 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f848d600640 10 client.16479 handle_command_reply: tid=0 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.876+0000 7f84a917f740 1 client.16479 shutdown 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 msgr2=0x558521bc70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bc70e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8490005050 tx=0x7f84900052c0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 shutdown_connections 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558521c29e40 0x558521c2c260 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558521bbd6b0 0x558521bc8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558521bb8540 0x558521bc7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 --2- 172.21.15.116:0/2061179862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558521bb7b50 0x558521bc70e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f848d600640 0 client.16479 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 >> 172.21.15.116:0/2061179862 conn(0x558521bb30c0 msgr2=0x558521bb1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 shutdown_connections 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 1 -- 172.21.15.116:0/2061179862 wait complete. 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.877+0000 7f84a917f740 20 client.16479 trim_cache size 0 max 16384 2024-03-04T07:46:02.882 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:46:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:46:02.883 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:02.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f84840a1e00 msgr2=0x7f84840a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f84840a1e00 0x7f84840a42c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f84980096c0 tx=0x7f849802c8a0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01049e0 msgr2=0x7f84a0199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01049e0 0x7f84a0199ff0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8488005fd0 tx=0x7f848800dde0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 shutdown_connections 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f84840a1e00 0x7f84840a42c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84a01053a0 0x7f84a01a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f84a01049e0 0x7f84a0199ff0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 --2- 172.21.15.116:0/2354166585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a01037f0 0x7f84a0199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 >> 172.21.15.116:0/2354166585 conn(0x7f84a00fed80 msgr2=0x7f84a0107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 shutdown_connections 2024-03-04T07:46:02.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:02.878+0000 7f84a6400640 1 -- 172.21.15.116:0/2354166585 wait complete. 2024-03-04T07:46:02.897 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:46:02.897 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:46:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 -- 172.21.15.116:0/3896156759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa41095e0 msgr2=0x7fafa410fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 --2- 172.21.15.116:0/3896156759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa41095e0 0x7fafa410fe70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009ca0 tx=0x7faf9c02c100 comp rx=0 tx=0).stop 2024-03-04T07:46:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 -- 172.21.15.116:0/3896156759 shutdown_connections 2024-03-04T07:46:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 --2- 172.21.15.116:0/3896156759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa41095e0 0x7fafa410fe70 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 --2- 172.21.15.116:0/3896156759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafa4108c20 0x7fafa41090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 --2- 172.21.15.116:0/3896156759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa4107a30 0x7fafa4107e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 -- 172.21.15.116:0/3896156759 >> 172.21.15.116:0/3896156759 conn(0x7fafa40769e0 msgr2=0x7fafa4078e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 -- 172.21.15.116:0/3896156759 shutdown_connections 2024-03-04T07:46:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.103+0000 7fafa3200640 1 -- 172.21.15.116:0/3896156759 wait complete. 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.104+0000 7fafa3200640 1 Processor -- start 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.104+0000 7fafa3200640 1 -- start start 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.104+0000 7fafa3200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafa4107a30 0x7fafa419dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.104+0000 7fafa3200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa4108c20 0x7fafa419e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa41095e0 0x7fafa41a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa3200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4113690 con 0x7fafa41095e0 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa3200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4113510 con 0x7fafa4108c20 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa3200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4113810 con 0x7fafa4107a30 2024-03-04T07:46:03.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7faf99400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa4108c20 0x7fafa419e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa41095e0 0x7fafa41a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7faf99400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa4108c20 0x7fafa419e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60254/0 (socket says 172.21.15.116:60254) 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7faf99400640 1 -- 172.21.15.116:0/4117400840 learned_addr learned my addr 172.21.15.116:0/4117400840 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa1e00640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafa4107a30 0x7fafa419dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa2800640 1 -- 172.21.15.116:0/4117400840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafa4107a30 msgr2=0x7fafa419dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa2800640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafa4107a30 0x7fafa419dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa2800640 1 -- 172.21.15.116:0/4117400840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa4108c20 msgr2=0x7fafa419e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa2800640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa4108c20 0x7fafa419e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7fafa2800640 1 -- 172.21.15.116:0/4117400840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa41a58c0 con 0x7fafa41095e0 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.105+0000 7faf99400640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa4108c20 0x7fafa419e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.106+0000 7fafa2800640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa41095e0 0x7fafa41a5220 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faf9c037040 tx=0x7faf9c035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.106+0000 7faf9b400640 1 -- 172.21.15.116:0/4117400840 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c039040 con 0x7fafa41095e0 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.106+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa41a5b50 con 0x7fafa41095e0 2024-03-04T07:46:03.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.106+0000 7faf9b400640 1 -- 172.21.15.116:0/4117400840 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf9c044070 con 0x7fafa41095e0 2024-03-04T07:46:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.106+0000 7faf9b400640 1 -- 172.21.15.116:0/4117400840 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c004450 con 0x7fafa41095e0 2024-03-04T07:46:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.106+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa41a6060 con 0x7fafa41095e0 2024-03-04T07:46:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.107+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa410ecf0 con 0x7fafa41095e0 2024-03-04T07:46:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.109+0000 7faf9b400640 1 -- 172.21.15.116:0/4117400840 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7faf9c0045f0 con 0x7fafa41095e0 2024-03-04T07:46:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.110+0000 7faf9b400640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf880a1d80 0x7faf880a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.110+0000 7faf9b400640 1 -- 172.21.15.116:0/4117400840 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7faf9c0f03f0 con 0x7fafa41095e0 2024-03-04T07:46:03.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.111+0000 7fafa1e00640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf880a1d80 0x7faf880a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.111+0000 7fafa1e00640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf880a1d80 0x7faf880a4240 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fafa419eb50 tx=0x7faf94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.115+0000 7faf9b400640 1 -- 172.21.15.116:0/4117400840 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faf9c0bd7e0 con 0x7fafa41095e0 2024-03-04T07:46:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:02 smithi116 ceph-mon[32150]: pgmap v559: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:02 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:46:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:02 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:02 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:46:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:02 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:02 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:46:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:02 smithi152 ceph-mon[37154]: pgmap v559: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:02 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:46:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:02 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:02 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:46:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:02 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:02 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:46:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:02 smithi133 ceph-mon[36505]: pgmap v559: 97 pgs: 97 active+clean; 3.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:02 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:46:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:02 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:02 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:46:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:02 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:02 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:46:03.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.358+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa402cff0 con 0x7fafa41095e0 2024-03-04T07:46:03.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.360+0000 7faf9b400640 1 -- 172.21.15.116:0/4117400840 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7faf9c0098b0 con 0x7fafa41095e0 2024-03-04T07:46:03.360 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:46:03.361 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:46:03.361 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:46:03.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf880a1d80 msgr2=0x7faf880a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf880a1d80 0x7faf880a4240 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fafa419eb50 tx=0x7faf94005f90 comp rx=0 tx=0).stop 2024-03-04T07:46:03.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa41095e0 msgr2=0x7fafa41a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa41095e0 0x7fafa41a5220 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faf9c037040 tx=0x7faf9c035d10 comp rx=0 tx=0).stop 2024-03-04T07:46:03.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 shutdown_connections 2024-03-04T07:46:03.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafa41095e0 0x7fafa41a5220 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fafa4108c20 0x7fafa419e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7faf880a1d80 0x7faf880a4240 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 --2- 172.21.15.116:0/4117400840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafa4107a30 0x7fafa419dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 >> 172.21.15.116:0/4117400840 conn(0x7fafa40769e0 msgr2=0x7fafa410bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 shutdown_connections 2024-03-04T07:46:03.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.363+0000 7fafa3200640 1 -- 172.21.15.116:0/4117400840 wait complete. 2024-03-04T07:46:03.383 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:46:03.383 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:46:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 -- 172.21.15.116:0/3636977039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f41641027d0 msgr2=0x7f4164102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 --2- 172.21.15.116:0/3636977039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f41641027d0 0x7f4164102c50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f414c009ca0 tx=0x7f414c02c100 comp rx=0 tx=0).stop 2024-03-04T07:46:03.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 -- 172.21.15.116:0/3636977039 shutdown_connections 2024-03-04T07:46:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 --2- 172.21.15.116:0/3636977039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4164103190 0x7f4164109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 --2- 172.21.15.116:0/3636977039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f41641027d0 0x7f4164102c50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 --2- 172.21.15.116:0/3636977039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41641015e0 0x7f4164101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 -- 172.21.15.116:0/3636977039 >> 172.21.15.116:0/3636977039 conn(0x7f41640fcb70 msgr2=0x7f41640fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 -- 172.21.15.116:0/3636977039 shutdown_connections 2024-03-04T07:46:03.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.603+0000 7f4169600640 1 -- 172.21.15.116:0/3636977039 wait complete. 2024-03-04T07:46:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.604+0000 7f4169600640 1 Processor -- start 2024-03-04T07:46:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.604+0000 7f4169600640 1 -- start start 2024-03-04T07:46:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4169600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41641015e0 0x7f4164199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4169600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41641027d0 0x7f416419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4169600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4164103190 0x7f41641a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4169600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416410d260 con 0x7f41641015e0 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4169600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416410d0e0 con 0x7f4164103190 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4169600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416410d3e0 con 0x7f41641027d0 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41641015e0 0x7f4164199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41641015e0 0x7f4164199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35788/0 (socket says 172.21.15.116:35788) 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162a00640 1 -- 172.21.15.116:0/2844750870 learned_addr learned my addr 172.21.15.116:0/2844750870 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162000640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41641027d0 0x7f416419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4163400640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4164103190 0x7f41641a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162000640 1 -- 172.21.15.116:0/2844750870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4164103190 msgr2=0x7f41641a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162000640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4164103190 0x7f41641a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162000640 1 -- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41641015e0 msgr2=0x7f4164199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162000640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41641015e0 0x7f4164199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.605+0000 7f4162000640 1 -- 172.21.15.116:0/2844750870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41641a1710 con 0x7f41641027d0 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.606+0000 7f4162000640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41641027d0 0x7f416419a030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f414c02c640 tx=0x7f414c035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.606+0000 7f415b400640 1 -- 172.21.15.116:0/2844750870 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c039040 con 0x7f41641027d0 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.606+0000 7f415b400640 1 -- 172.21.15.116:0/2844750870 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f414c003970 con 0x7f41641027d0 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.606+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41641a19a0 con 0x7f41641027d0 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.606+0000 7f415b400640 1 -- 172.21.15.116:0/2844750870 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c004330 con 0x7f41641027d0 2024-03-04T07:46:03.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.606+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41641a1e90 con 0x7f41641027d0 2024-03-04T07:46:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.609+0000 7f416c385740 1 Processor -- start 2024-03-04T07:46:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.609+0000 7f415b400640 1 -- 172.21.15.116:0/2844750870 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f414c046030 con 0x7f41641027d0 2024-03-04T07:46:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f416c385740 1 -- start start 2024-03-04T07:46:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f416c385740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b130c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f416c385740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b18190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f416c385740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b1ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f416c385740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffe2a77d60 con 0x55ffe2b18800 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f416c385740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffe2a20cf0 con 0x55ffe2b13690 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f416c385740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffe2a73e00 con 0x55ffe2b12ca0 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f4162a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b18190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f4162000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b1ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f4162a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b18190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60278/0 (socket says 172.21.15.116:60278) 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f4162a00640 1 -- 172.21.15.116:0/650725735 learned_addr learned my addr 172.21.15.116:0/650725735 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f4163400640 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.610+0000 7f4162a00640 1 -- 172.21.15.116:0/650725735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 msgr2=0x55ffe2b130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4162a00640 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b130c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4162a00640 1 -- 172.21.15.116:0/650725735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 msgr2=0x55ffe2b1ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4162a00640 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b1ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4162a00640 1 -- 172.21.15.116:0/650725735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ffe278c3f0 con 0x55ffe2b13690 2024-03-04T07:46:03.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4162a00640 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b18190 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f415c009d60 tx=0x7f415c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90c0ca49ff40b76a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f415b400640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f41480a1d80 0x7f41480a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4153400640 1 -- 172.21.15.116:0/650725735 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c004230 con 0x55ffe2b13690 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4153400640 1 -- 172.21.15.116:0/650725735 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f415c0043d0 con 0x55ffe2b13690 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4153400640 1 -- 172.21.15.116:0/650725735 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c019890 con 0x55ffe2b13690 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f415b400640 1 -- 172.21.15.116:0/2844750870 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f414c0ef9f0 con 0x7f41641027d0 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.611+0000 7f4163400640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f41480a1d80 0x7f41480a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 -- 172.21.15.116:0/650725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 msgr2=0x55ffe2b18190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b18190 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f415c009d60 tx=0x7f415c011810 comp rx=0 tx=0).stop 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 -- 172.21.15.116:0/650725735 shutdown_connections 2024-03-04T07:46:03.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b1ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b18190 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 --2- 172.21.15.116:0/650725735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 -- 172.21.15.116:0/650725735 >> 172.21.15.116:0/650725735 conn(0x55ffe2b0e210 msgr2=0x55ffe2b0e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f4163400640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f41480a1d80 0x7f41480a4240 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55ffe2b13ef0 tx=0x7f4154005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 -- 172.21.15.116:0/650725735 shutdown_connections 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.612+0000 7f416c385740 1 -- 172.21.15.116:0/650725735 wait complete. 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 -- start start 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 Processor -- start 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b1d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b1d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b24730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffe2b1b7b0 con 0x55ffe2b18800 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffe2a16220 con 0x55ffe2b13690 2024-03-04T07:46:03.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffe2b1b930 con 0x55ffe2b12ca0 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.613+0000 7f416c385740 10 client.0 _ll_register_callbacks cb 0x55ffe278bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b1d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4163400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b24730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b1d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60290/0 (socket says 172.21.15.116:60290) 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 -- 172.21.15.116:0/164927401 learned_addr learned my addr 172.21.15.116:0/164927401 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162a00640 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b1d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 -- 172.21.15.116:0/164927401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 msgr2=0x55ffe2b1d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b1d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 -- 172.21.15.116:0/164927401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 msgr2=0x55ffe2b24730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b24730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 -- 172.21.15.116:0/164927401 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ffe28567d0 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4162000640 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b1d6f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f414c0bc700 tx=0x7f414c0f3040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb1b5653731771ef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4147e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4147e00640 1 -- 172.21.15.116:0/164927401 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c0bc2b0 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f4147e00640 1 -- 172.21.15.116:0/164927401 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f414c003b20 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.614+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f415c0043d0 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f4147e00640 1 -- 172.21.15.116:0/164927401 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f414c0bc0d0 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 10 client.26624 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ffe2b27aa0 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f4147e00640 1 -- 172.21.15.116:0/164927401 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f415c0043d0 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f4147e00640 1 -- 172.21.15.116:0/164927401 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f414c03c030 con 0x55ffe2b13690 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 10 client.26624 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 10 client.26624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:03.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 20 client.26624 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b29130 0x55ffe2b2b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 4 client.26624 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ffe2a852a0 con 0x55ffe2b29130 2024-03-04T07:46:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.615+0000 7f4162a00640 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b29130 0x55ffe2b2b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.616+0000 7f4162a00640 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b29130 0x55ffe2b2b590 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae7846a37aecd5ea server_cookie=fddc79ab3401c49c in_seq=0 out_seq=0 2024-03-04T07:46:03.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.616+0000 7f4147e00640 10 client.26624 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f4147e00640 1 -- 172.21.15.116:0/164927401 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ffe2a852a0 con 0x55ffe2b29130 2024-03-04T07:46:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f4147e00640 10 client.26624 handle_command_reply: tid=0 2024-03-04T07:46:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 client.26624 shutdown 2024-03-04T07:46:03.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 msgr2=0x55ffe2b1d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b1d6f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f414c0bc700 tx=0x7f414c0f3040 comp rx=0 tx=0).stop 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 shutdown_connections 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b29130 0x55ffe2b2b590 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b18800 0x55ffe2b24730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b13690 0x55ffe2b1d6f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f4147e00640 0 client.26624 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 --2- 172.21.15.116:0/164927401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b1d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 >> 172.21.15.116:0/164927401 conn(0x55ffe2b0e210 msgr2=0x55ffe2b0d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 shutdown_connections 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 1 -- 172.21.15.116:0/164927401 wait complete. 2024-03-04T07:46:03.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.617+0000 7f416c385740 20 client.26624 trim_cache size 0 max 16384 2024-03-04T07:46:03.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 Processor -- start 2024-03-04T07:46:03.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 -- start start 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b0e210 0x55ffe2b2fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b246f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b13690 0x55ffe2b30390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a8ca0 con 0x55ffe2b12ca0 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641ab280 con 0x55ffe2b0e210 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f416c385740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a92a0 con 0x55ffe2b13690 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f4162000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b246f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f4162a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b0e210 0x55ffe2b2fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f4162a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b0e210 0x55ffe2b2fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60292/0 (socket says 172.21.15.116:60292) 2024-03-04T07:46:03.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f4162a00640 1 -- 172.21.15.116:0/843789136 learned_addr learned my addr 172.21.15.116:0/843789136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f4162000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b246f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35820/0 (socket says 172.21.15.116:35820) 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.631+0000 7f4163400640 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b13690 0x55ffe2b30390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4162a00640 1 -- 172.21.15.116:0/843789136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b13690 msgr2=0x55ffe2b30390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4162a00640 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b13690 0x55ffe2b30390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4162a00640 1 -- 172.21.15.116:0/843789136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b12ca0 msgr2=0x55ffe2b246f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4162a00640 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b246f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4162a00640 1 -- 172.21.15.116:0/843789136 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ffe2b1bab0 con 0x55ffe2b0e210 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4162a00640 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b0e210 0x55ffe2b2fe50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f415c005fd0 tx=0x7f415c004300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5187df7018a1547 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4152a00640 1 -- 172.21.15.116:0/843789136 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c004370 con 0x55ffe2b0e210 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4152a00640 1 -- 172.21.15.116:0/843789136 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f415c005500 con 0x55ffe2b0e210 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.632+0000 7f4152a00640 1 -- 172.21.15.116:0/843789136 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c003810 con 0x55ffe2b0e210 2024-03-04T07:46:03.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 -- 172.21.15.116:0/843789136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b0e210 msgr2=0x55ffe2b2fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b0e210 0x55ffe2b2fe50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f415c005fd0 tx=0x7f415c004300 comp rx=0 tx=0).stop 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 -- 172.21.15.116:0/843789136 shutdown_connections 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b13690 0x55ffe2b30390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b246f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 --2- 172.21.15.116:0/843789136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b0e210 0x55ffe2b2fe50 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 -- 172.21.15.116:0/843789136 >> 172.21.15.116:0/843789136 conn(0x55ffe2b1cd70 msgr2=0x55ffe2b11d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 -- 172.21.15.116:0/843789136 shutdown_connections 2024-03-04T07:46:03.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 -- 172.21.15.116:0/843789136 wait complete. 2024-03-04T07:46:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 -- start start 2024-03-04T07:46:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.633+0000 7f416c385740 1 Processor -- start 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f416c385740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b0e210 0x55ffe2b230c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f416c385740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b23600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f416c385740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b13690 0x55ffe2b23b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f416c385740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a9900 con 0x55ffe2b13690 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f416c385740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641aac20 con 0x55ffe2b12ca0 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f416c385740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a9f60 con 0x55ffe2b0e210 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f416c385740 10 client.0 _ll_register_callbacks cb 0x7f414c009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b23600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4163400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b13690 0x55ffe2b23b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b23600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60304/0 (socket says 172.21.15.116:60304) 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 -- 172.21.15.116:0/3136911124 learned_addr learned my addr 172.21.15.116:0/3136911124 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162a00640 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b0e210 0x55ffe2b230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 -- 172.21.15.116:0/3136911124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b0e210 msgr2=0x55ffe2b230c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b0e210 0x55ffe2b230c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 -- 172.21.15.116:0/3136911124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b13690 msgr2=0x55ffe2b23b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b13690 0x55ffe2b23b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162000640 1 -- 172.21.15.116:0/3136911124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ffe2b84c20 con 0x55ffe2b12ca0 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.634+0000 7f4162a00640 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b0e210 0x55ffe2b230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f4162000640 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b23600 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f414c0ed590 tx=0x7f414c009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e25014da65d54f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f4147e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f4147e00640 1 -- 172.21.15.116:0/3136911124 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c0bc2b0 con 0x55ffe2b12ca0 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f4147e00640 1 -- 172.21.15.116:0/3136911124 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f414c0dc2b0 con 0x55ffe2b12ca0 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f415c005500 con 0x55ffe2b12ca0 2024-03-04T07:46:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f4147e00640 1 -- 172.21.15.116:0/3136911124 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f414c0c0910 con 0x55ffe2b12ca0 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 10 client.26630 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ffe2b85f50 con 0x55ffe2b12ca0 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f4147e00640 1 -- 172.21.15.116:0/3136911124 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f415c005500 con 0x55ffe2b12ca0 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f4147e00640 1 -- 172.21.15.116:0/3136911124 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f414c03c070 con 0x55ffe2b12ca0 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 10 client.26630 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 10 client.26630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 20 client.26630 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b18800 0x55ffe2b88750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 4 client.26630 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.635+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ffe2a7f5f0 con 0x55ffe2b18800 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f4162a00640 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b18800 0x55ffe2b88750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f4162a00640 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b18800 0x55ffe2b88750 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8195c1e617850ef3 server_cookie=aefeb64e6ced9ee2 in_seq=0 out_seq=0 2024-03-04T07:46:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f4147e00640 10 client.26630 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f4147e00640 1 -- 172.21.15.116:0/3136911124 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ffe2a7f5f0 con 0x55ffe2b18800 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f4147e00640 10 client.26630 handle_command_reply: tid=0 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f416c385740 1 client.26630 shutdown 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b12ca0 msgr2=0x55ffe2b23600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.636+0000 7f416c385740 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b23600 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f414c0ed590 tx=0x7f414c009870 comp rx=0 tx=0).stop 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 shutdown_connections 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ffe2b18800 0x55ffe2b88750 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffe2b13690 0x55ffe2b23b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ffe2b12ca0 0x55ffe2b23600 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 --2- 172.21.15.116:0/3136911124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ffe2b0e210 0x55ffe2b230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4147e00640 0 client.26630 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 >> 172.21.15.116:0/3136911124 conn(0x55ffe2b1cd70 msgr2=0x55ffe2b0cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 shutdown_connections 2024-03-04T07:46:03.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 1 -- 172.21.15.116:0/3136911124 wait complete. 2024-03-04T07:46:03.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f416c385740 20 client.26630 trim_cache size 0 max 16384 2024-03-04T07:46:03.644 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f41480a1d80 msgr2=0x7f41480a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f41480a1d80 0x7f41480a4240 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55ffe2b13ef0 tx=0x7f4154005f90 comp rx=0 tx=0).stop 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41641027d0 msgr2=0x7f416419a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41641027d0 0x7f416419a030 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f414c02c640 tx=0x7f414c035d90 comp rx=0 tx=0).stop 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 shutdown_connections 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4164103190 0x7f41641a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f41480a1d80 0x7f41480a4240 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41641027d0 0x7f416419a030 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 --2- 172.21.15.116:0/2844750870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41641015e0 0x7f4164199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.637+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 >> 172.21.15.116:0/2844750870 conn(0x7f41640fcb70 msgr2=0x7f4164073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.638+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 shutdown_connections 2024-03-04T07:46:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:03.638+0000 7f4169600640 1 -- 172.21.15.116:0/2844750870 wait complete. 2024-03-04T07:46:04.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:03 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4117400840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:04.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:03 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4117400840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:04.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:03 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4117400840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:04.656 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:46:04.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:46:04.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:46:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 -- 172.21.15.116:0/102596720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 msgr2=0x7f9600104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 --2- 172.21.15.116:0/102596720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 0x7f9600104e80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009900 tx=0x7f95ec02bc40 comp rx=0 tx=0).stop 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 -- 172.21.15.116:0/102596720 shutdown_connections 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 --2- 172.21.15.116:0/102596720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96001053c0 0x7f960010bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 --2- 172.21.15.116:0/102596720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 0x7f9600104e80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 --2- 172.21.15.116:0/102596720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9600103810 0x7f9600103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 -- 172.21.15.116:0/102596720 >> 172.21.15.116:0/102596720 conn(0x7f96000fed80 msgr2=0x7f96001011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 -- 172.21.15.116:0/102596720 shutdown_connections 2024-03-04T07:46:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.883+0000 7f9607800640 1 -- 172.21.15.116:0/102596720 wait complete. 2024-03-04T07:46:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.884+0000 7f9607800640 1 Processor -- start 2024-03-04T07:46:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9607800640 1 -- start start 2024-03-04T07:46:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9607800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9600103810 0x7f9600199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9607800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 0x7f960019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9607800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001053c0 0x7f96001a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9607800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960010f470 con 0x7f9600104a00 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9607800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960010f2f0 con 0x7f9600103810 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9607800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960010f5f0 con 0x7f96001053c0 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9604e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9600103810 0x7f9600199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f95ffe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 0x7f960019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9604e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9600103810 0x7f9600199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60328/0 (socket says 172.21.15.116:60328) 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.885+0000 7f9604e00640 1 -- 172.21.15.116:0/4048750626 learned_addr learned my addr 172.21.15.116:0/4048750626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9605800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001053c0 0x7f96001a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9604e00640 1 -- 172.21.15.116:0/4048750626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001053c0 msgr2=0x7f96001a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9604e00640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001053c0 0x7f96001a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9604e00640 1 -- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 msgr2=0x7f960019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9604e00640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 0x7f960019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9604e00640 1 -- 172.21.15.116:0/4048750626 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96001a1760 con 0x7f9600103810 2024-03-04T07:46:04.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9605800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001053c0 0x7f96001a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9604e00640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9600103810 0x7f9600199b40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f95f800ebc0 tx=0x7f95f800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f95fd600640 1 -- 172.21.15.116:0/4048750626 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f8003b70 con 0x7f9600103810 2024-03-04T07:46:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f95fd600640 1 -- 172.21.15.116:0/4048750626 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f8003d10 con 0x7f9600103810 2024-03-04T07:46:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96001a1a50 con 0x7f9600103810 2024-03-04T07:46:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f95fd600640 1 -- 172.21.15.116:0/4048750626 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f8019980 con 0x7f9600103810 2024-03-04T07:46:04.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.886+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96001a1f70 con 0x7f9600103810 2024-03-04T07:46:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.889+0000 7f95fd600640 1 -- 172.21.15.116:0/4048750626 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f95f8019b20 con 0x7f9600103810 2024-03-04T07:46:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 Processor -- start 2024-03-04T07:46:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 -- start start 2024-03-04T07:46:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aad34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aad85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aadb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c1a9e6e20 con 0x561c1aad30d0 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c1aa38a20 con 0x561c1aad3ac0 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f960a5e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c1a9e1b00 con 0x561c1aad8c30 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f9605800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aad34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f9604e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aad85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f9605800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aad34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35846/0 (socket says 172.21.15.116:35846) 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.890+0000 7f9605800640 1 -- 172.21.15.116:0/254194103 learned_addr learned my addr 172.21.15.116:0/254194103 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f95ffe00640 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aadb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f9605800640 1 -- 172.21.15.116:0/254194103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 msgr2=0x561c1aadb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f9605800640 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aadb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f9605800640 1 -- 172.21.15.116:0/254194103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 msgr2=0x561c1aad85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f9605800640 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aad85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f9605800640 1 -- 172.21.15.116:0/254194103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c1a9f6e10 con 0x561c1aad30d0 2024-03-04T07:46:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f9604e00640 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aad85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f9605800640 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aad34f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009cc0 tx=0x7f95f00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82676f4e9fdda97f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:04.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.891+0000 7f95fd600640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f95e40a1d80 0x7f95e40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f95fd600640 1 -- 172.21.15.116:0/4048750626 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f95f80ce6a0 con 0x7f9600103810 2024-03-04T07:46:04.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f9604e00640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f95e40a1d80 0x7f95e40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f95f6000640 1 -- 172.21.15.116:0/254194103 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0012450 con 0x561c1aad30d0 2024-03-04T07:46:04.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f95f6000640 1 -- 172.21.15.116:0/254194103 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f0012ad0 con 0x561c1aad30d0 2024-03-04T07:46:04.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f95f6000640 1 -- 172.21.15.116:0/254194103 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f000fc90 con 0x561c1aad30d0 2024-03-04T07:46:04.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f960a5e8740 1 -- 172.21.15.116:0/254194103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 msgr2=0x561c1aad34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f9604e00640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f95e40a1d80 0x7f95e40a4240 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f95f809b320 tx=0x7f95f80d4530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.892+0000 7f960a5e8740 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aad34f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009cc0 tx=0x7f95f00117b0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 -- 172.21.15.116:0/254194103 shutdown_connections 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aadb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aad85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 --2- 172.21.15.116:0/254194103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aad34f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 -- 172.21.15.116:0/254194103 >> 172.21.15.116:0/254194103 conn(0x561c1aace640 msgr2=0x561c1aacea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 -- 172.21.15.116:0/254194103 shutdown_connections 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 -- 172.21.15.116:0/254194103 wait complete. 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 -- start start 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.893+0000 7f960a5e8740 1 Processor -- start 2024-03-04T07:46:04.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aadd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad3ac0 0x561c1aadd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f960a5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c1a9f4f40 con 0x561c1aad3ac0 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f960a5e8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c1aa34ac0 con 0x561c1aad8c30 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f960a5e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c1aadbbe0 con 0x561c1aad30d0 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f960a5e8740 10 client.0 _ll_register_callbacks cb 0x561c1aa33aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad3ac0 0x561c1aadd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad3ac0 0x561c1aadd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35852/0 (socket says 172.21.15.116:35852) 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 -- 172.21.15.116:0/3310000815 learned_addr learned my addr 172.21.15.116:0/3310000815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f9605800640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aadd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f9605800640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 -- 172.21.15.116:0/3310000815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 msgr2=0x561c1aadd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aadd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 -- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 msgr2=0x561c1aae49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f95ffe00640 1 -- 172.21.15.116:0/3310000815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c1a8157e0 con 0x561c1aad3ac0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.894+0000 7f9605800640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f95ffe00640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad3ac0 0x561c1aadd990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009900 tx=0x7f95ec004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e9ae8673de0f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f95eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f95eaa00640 1 -- 172.21.15.116:0/3310000815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec005d50 con 0x561c1aad3ac0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f95eaa00640 1 -- 172.21.15.116:0/3310000815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f95ec005230 con 0x561c1aad3ac0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95f0012ad0 con 0x561c1aad3ac0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f95eaa00640 1 -- 172.21.15.116:0/3310000815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95ec005430 con 0x561c1aad3ac0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f960a5e8740 10 client.16515 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.895+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c1a875f70 con 0x561c1aad3ac0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f95eaa00640 1 -- 172.21.15.116:0/3310000815 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f0012ad0 con 0x561c1aad3ac0 2024-03-04T07:46:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f95eaa00640 1 -- 172.21.15.116:0/3310000815 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f95ec003850 con 0x561c1aad3ac0 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f960a5e8740 10 client.16515 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f960a5e8740 10 client.16515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f960a5e8740 20 client.16515 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f960a5e8740 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1aae9330 0x561c1aaeb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f960a5e8740 4 client.16515 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c1aa45f60 con 0x561c1aae9330 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.896+0000 7f9605800640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1aae9330 0x561c1aaeb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.897+0000 7f9605800640 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1aae9330 0x561c1aaeb750 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26146825f3323758 server_cookie=819b9f20996bf162 in_seq=0 out_seq=0 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.897+0000 7f95eaa00640 10 client.16515 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.897+0000 7f95eaa00640 1 -- 172.21.15.116:0/3310000815 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c1aa45f60 con 0x561c1aae9330 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.897+0000 7f95eaa00640 10 client.16515 handle_command_reply: tid=0 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.897+0000 7f960a5e8740 1 client.16515 shutdown 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad3ac0 msgr2=0x561c1aadd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad3ac0 0x561c1aadd990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009900 tx=0x7f95ec004250 comp rx=0 tx=0).stop 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 shutdown_connections 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1aae9330 0x561c1aaeb750 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad3ac0 0x561c1aadd990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 --2- 172.21.15.116:0/3310000815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aadd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 >> 172.21.15.116:0/3310000815 conn(0x561c1aace640 msgr2=0x561c1aacf890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f95eaa00640 0 client.16515 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 shutdown_connections 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 1 -- 172.21.15.116:0/3310000815 wait complete. 2024-03-04T07:46:04.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.898+0000 7f960a5e8740 20 client.16515 trim_cache size 0 max 16384 2024-03-04T07:46:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.918+0000 7f960a5e8740 1 Processor -- start 2024-03-04T07:46:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.918+0000 7f960a5e8740 1 -- start start 2024-03-04T07:46:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aacda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aae4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aad1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f960a5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001a8c70 con 0x561c1aad30d0 2024-03-04T07:46:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f960a5e8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001ab250 con 0x561c1aad3ac0 2024-03-04T07:46:04.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f960a5e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001a9270 con 0x561c1aad8c30 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f95ffe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aae4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f95ffe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aae4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60338/0 (socket says 172.21.15.116:60338) 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f95ffe00640 1 -- 172.21.15.116:0/4239227517 learned_addr learned my addr 172.21.15.116:0/4239227517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f9605800640 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aacda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f9605800640 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aad1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f95ffe00640 1 -- 172.21.15.116:0/4239227517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 msgr2=0x561c1aad1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f95ffe00640 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aad1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f95ffe00640 1 -- 172.21.15.116:0/4239227517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 msgr2=0x561c1aacda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.919+0000 7f95ffe00640 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aacda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.920+0000 7f95ffe00640 1 -- 172.21.15.116:0/4239227517 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c1a70d170 con 0x561c1aad3ac0 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.920+0000 7f9605800640 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aad1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.920+0000 7f95ffe00640 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aae4990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f95ec000c00 tx=0x7f95ec001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7e36fa1e9e1eae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.920+0000 7f95f5600640 1 -- 172.21.15.116:0/4239227517 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec005d50 con 0x561c1aad3ac0 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.920+0000 7f95f5600640 1 -- 172.21.15.116:0/4239227517 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95ec0057d0 con 0x561c1aad3ac0 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.920+0000 7f95f5600640 1 -- 172.21.15.116:0/4239227517 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec034440 con 0x561c1aad3ac0 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 -- 172.21.15.116:0/4239227517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 msgr2=0x561c1aae4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aae4990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f95ec000c00 tx=0x7f95ec001e70 comp rx=0 tx=0).stop 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 -- 172.21.15.116:0/4239227517 shutdown_connections 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad8c30 0x561c1aad1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad3ac0 0x561c1aae4990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 --2- 172.21.15.116:0/4239227517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aad30d0 0x561c1aacda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 -- 172.21.15.116:0/4239227517 >> 172.21.15.116:0/4239227517 conn(0x561c1aace640 msgr2=0x561c1aad18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 -- 172.21.15.116:0/4239227517 shutdown_connections 2024-03-04T07:46:04.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.921+0000 7f960a5e8740 1 -- 172.21.15.116:0/4239227517 wait complete. 2024-03-04T07:46:04.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 -- start start 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 Processor -- start 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aae43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aae2ab0 0x561c1aae2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001a98d0 con 0x561c1aae2ab0 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001aabf0 con 0x561c1aad8c30 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001a9f30 con 0x561c1aad30d0 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f960a5e8740 10 client.0 _ll_register_callbacks cb 0x7f95ec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f95ffe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f95ffe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60342/0 (socket says 172.21.15.116:60342) 2024-03-04T07:46:04.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.922+0000 7f95ffe00640 1 -- 172.21.15.116:0/1613083509 learned_addr learned my addr 172.21.15.116:0/1613083509 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f9605800640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aae2ab0 0x561c1aae2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f9605800640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aae43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95ffe00640 1 -- 172.21.15.116:0/1613083509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 msgr2=0x561c1aae43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95ffe00640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aae43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95ffe00640 1 -- 172.21.15.116:0/1613083509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aae2ab0 msgr2=0x561c1aae2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95ffe00640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aae2ab0 0x561c1aae2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95ffe00640 1 -- 172.21.15.116:0/1613083509 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c1aae3710 con 0x561c1aad8c30 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f9605800640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aae2ab0 0x561c1aae2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95ffe00640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae48f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f95ec02c7c0 tx=0x7f95ec02ced0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f97708fa062a8df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95eaa00640 1 -- 172.21.15.116:0/1613083509 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec005d50 con 0x561c1aad8c30 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f95eaa00640 1 -- 172.21.15.116:0/1613083509 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f95ec003800 con 0x561c1aad8c30 2024-03-04T07:46:04.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.923+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95ec0057d0 con 0x561c1aad8c30 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f95eaa00640 1 -- 172.21.15.116:0/1613083509 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95ec004250 con 0x561c1aad8c30 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 10 client.26648 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c1ab45ac0 con 0x561c1aad8c30 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f95eaa00640 1 -- 172.21.15.116:0/1613083509 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95ec0057d0 con 0x561c1aad8c30 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f95eaa00640 1 -- 172.21.15.116:0/1613083509 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f95ec051410 con 0x561c1aad8c30 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 10 client.26648 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 10 client.26648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 20 client.26648 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1ab46150 0x561c1ab48570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 4 client.26648 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561c1aa402b0 con 0x561c1ab46150 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.924+0000 7f9605800640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1ab46150 0x561c1ab48570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:04.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.925+0000 7f9605800640 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1ab46150 0x561c1ab48570 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=357b3fc58743fec server_cookie=9080c85dabc4234b in_seq=0 out_seq=0 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.925+0000 7f95eaa00640 10 client.26648 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f95eaa00640 1 -- 172.21.15.116:0/1613083509 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561c1aa402b0 con 0x561c1ab46150 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f95eaa00640 10 client.26648 handle_command_reply: tid=0 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 client.26648 shutdown 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 msgr2=0x561c1aae48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae48f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f95ec02c7c0 tx=0x7f95ec02ced0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 shutdown_connections 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c1ab46150 0x561c1ab48570 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c1aae2ab0 0x561c1aae2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c1aad8c30 0x561c1aae48f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f95eaa00640 0 client.26648 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 --2- 172.21.15.116:0/1613083509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c1aad30d0 0x561c1aae43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.926+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 >> 172.21.15.116:0/1613083509 conn(0x561c1aace640 msgr2=0x561c1aad2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 shutdown_connections 2024-03-04T07:46:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f960a5e8740 1 -- 172.21.15.116:0/1613083509 wait complete. 2024-03-04T07:46:04.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f960a5e8740 20 client.26648 trim_cache size 0 max 16384 2024-03-04T07:46:04.931 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:04.931 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:46:04.932 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "3ae8ca4f-1a62-4664-a709-bad3cf76cded", 2024-03-04T07:46:04.932 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:46:04.932 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f95e40a1d80 msgr2=0x7f95e40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f9607800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f95e40a1d80 0x7f95e40a4240 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f95f809b320 tx=0x7f95f80d4530 comp rx=0 tx=0).stop 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9600103810 msgr2=0x7f9600199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f9607800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9600103810 0x7f9600199b40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f95f800ebc0 tx=0x7f95f800c6a0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 shutdown_connections 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f9607800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f95e40a1d80 0x7f95e40a4240 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.927+0000 7f9607800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96001053c0 0x7f96001a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.928+0000 7f9607800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9600104a00 0x7f960019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.928+0000 7f9607800640 1 --2- 172.21.15.116:0/4048750626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9600103810 0x7f9600199b40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.928+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 >> 172.21.15.116:0/4048750626 conn(0x7f96000fed80 msgr2=0x7f9600107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:04.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.928+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 shutdown_connections 2024-03-04T07:46:04.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:04.928+0000 7f9607800640 1 -- 172.21.15.116:0/4048750626 wait complete. 2024-03-04T07:46:04.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:46:04.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:46:05.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:04 smithi116 ceph-mon[32150]: pgmap v560: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:46:05.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 -- 172.21.15.116:0/2254944746 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8100550 msgr2=0x7f46d8100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 --2- 172.21.15.116:0/2254944746 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8100550 0x7f46d8100970 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009900 tx=0x7f46c802bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:05.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 -- 172.21.15.116:0/2254944746 shutdown_connections 2024-03-04T07:46:05.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 --2- 172.21.15.116:0/2254944746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8102100 0x7f46d8108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 --2- 172.21.15.116:0/2254944746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46d8101740 0x7f46d8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 --2- 172.21.15.116:0/2254944746 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8100550 0x7f46d8100970 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 -- 172.21.15.116:0/2254944746 >> 172.21.15.116:0/2254944746 conn(0x7f46d80fbb00 msgr2=0x7f46d80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:05.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 -- 172.21.15.116:0/2254944746 shutdown_connections 2024-03-04T07:46:05.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.154+0000 7f46dea00640 1 -- 172.21.15.116:0/2254944746 wait complete. 2024-03-04T07:46:05.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.155+0000 7f46dea00640 1 Processor -- start 2024-03-04T07:46:05.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.155+0000 7f46dea00640 1 -- start start 2024-03-04T07:46:05.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8100550 0x7f46d8199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46d8101740 0x7f46d819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8102100 0x7f46d81a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d810c190 con 0x7f46d8100550 2024-03-04T07:46:05.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dea00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d810c010 con 0x7f46d8102100 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dea00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d810c310 con 0x7f46d8101740 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8100550 0x7f46d8199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8100550 0x7f46d8199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35886/0 (socket says 172.21.15.116:35886) 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 -- 172.21.15.116:0/3255227119 learned_addr learned my addr 172.21.15.116:0/3255227119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dca00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8102100 0x7f46d81a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46cfe00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46d8101740 0x7f46d819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 -- 172.21.15.116:0/3255227119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46d8101740 msgr2=0x7f46d819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46d8101740 0x7f46d819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 -- 172.21.15.116:0/3255227119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8102100 msgr2=0x7f46d81a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8102100 0x7f46d81a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46d7e00640 1 -- 172.21.15.116:0/3255227119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d81a1770 con 0x7f46d8100550 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46cfe00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46d8101740 0x7f46d819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.156+0000 7f46dca00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8102100 0x7f46d81a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:05.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.157+0000 7f46d7e00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8100550 0x7f46d8199bc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f46c8000c00 tx=0x7f46c802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:05.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.157+0000 7f46d5600640 1 -- 172.21.15.116:0/3255227119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8043020 con 0x7f46d8100550 2024-03-04T07:46:05.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.157+0000 7f46d5600640 1 -- 172.21.15.116:0/3255227119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c80054e0 con 0x7f46d8100550 2024-03-04T07:46:05.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.157+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d81a1a00 con 0x7f46d8100550 2024-03-04T07:46:05.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.157+0000 7f46d5600640 1 -- 172.21.15.116:0/3255227119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8003ce0 con 0x7f46d8100550 2024-03-04T07:46:05.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.157+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d81a1f70 con 0x7f46d8100550 2024-03-04T07:46:05.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.160+0000 7f46d5600640 1 -- 172.21.15.116:0/3255227119 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f46c8046020 con 0x7f46d8100550 2024-03-04T07:46:05.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 Processor -- start 2024-03-04T07:46:05.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 -- start start 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671e2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671e8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671eab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367110090 con 0x5583671e86b0 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367147d30 con 0x5583671e2b50 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46e15df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583670f0a30 con 0x5583671e3540 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46dca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671e2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46dca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671e2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60370/0 (socket says 172.21.15.116:60370) 2024-03-04T07:46:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46dca00640 1 -- 172.21.15.116:0/1149325231 learned_addr learned my addr 172.21.15.116:0/1149325231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.161+0000 7f46cfe00640 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671eab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46d7e00640 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671e8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46dca00640 1 -- 172.21.15.116:0/1149325231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 msgr2=0x5583671e8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46dca00640 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671e8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46dca00640 1 -- 172.21.15.116:0/1149325231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 msgr2=0x5583671eab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46dca00640 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671eab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46dca00640 1 -- 172.21.15.116:0/1149325231 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558366f267d0 con 0x5583671e2b50 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46cfe00640 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46d5600640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46bc0a1e00 0x7f46bc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46dca00640 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671e2f70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f46d00117e0 tx=0x7f46d0011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25169dd54b4f4e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46d5600640 1 -- 172.21.15.116:0/3255227119 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f46c80efeb0 con 0x7f46d8100550 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46d7e00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46bc0a1e00 0x7f46bc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46cd200640 1 -- 172.21.15.116:0/1149325231 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0011310 con 0x5583671e2b50 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46cd200640 1 -- 172.21.15.116:0/1149325231 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d0003e50 con 0x5583671e2b50 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.162+0000 7f46cd200640 1 -- 172.21.15.116:0/1149325231 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00045c0 con 0x5583671e2b50 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 -- 172.21.15.116:0/1149325231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 msgr2=0x5583671e2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671e2f70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f46d00117e0 tx=0x7f46d0011ca0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46d7e00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46bc0a1e00 0x7f46bc0a42c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f46c80bc8a0 tx=0x7f46c802bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 -- 172.21.15.116:0/1149325231 shutdown_connections 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671e8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 --2- 172.21.15.116:0/1149325231 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671e2f70 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 -- 172.21.15.116:0/1149325231 >> 172.21.15.116:0/1149325231 conn(0x5583671de0c0 msgr2=0x5583671de4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 -- 172.21.15.116:0/1149325231 shutdown_connections 2024-03-04T07:46:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.163+0000 7f46e15df740 1 -- 172.21.15.116:0/1149325231 wait complete. 2024-03-04T07:46:05.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 -- start start 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 Processor -- start 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e2b50 0x5583671ecf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671ed4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e86b0 0x5583671f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583670e60c0 con 0x5583671e2b50 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367143dd0 con 0x5583671e3540 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583671eb660 con 0x5583671e86b0 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.164+0000 7f46e15df740 10 client.0 _ll_register_callbacks cb 0x558367103860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46dca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e2b50 0x5583671ecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671ed4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671ed4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60372/0 (socket says 172.21.15.116:60372) 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 -- 172.21.15.116:0/4220718807 learned_addr learned my addr 172.21.15.116:0/4220718807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:05.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46dca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e2b50 0x5583671ecf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35908/0 (socket says 172.21.15.116:35908) 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46dca00640 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e86b0 0x5583671f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 -- 172.21.15.116:0/4220718807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e86b0 msgr2=0x5583671f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e86b0 0x5583671f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 -- 172.21.15.116:0/4220718807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e2b50 msgr2=0x5583671ecf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e2b50 0x5583671ecf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 -- 172.21.15.116:0/4220718807 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558366e5c3f0 con 0x5583671e3540 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46dca00640 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e86b0 0x5583671f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46cfe00640 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671ed4b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f46c000bf50 tx=0x7f46c0011870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3ab0a7b55d1b438 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.165+0000 7f46c5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46c5600640 1 -- 172.21.15.116:0/4220718807 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0004220 con 0x5583671e3540 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46c5600640 1 -- 172.21.15.116:0/4220718807 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f46c00043c0 con 0x5583671e3540 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46d0003e50 con 0x5583671e3540 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46c5600640 1 -- 172.21.15.116:0/4220718807 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46c00035f0 con 0x5583671e3540 2024-03-04T07:46:05.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 10 client.26660 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558366e1c0f0 con 0x5583671e3540 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46c5600640 1 -- 172.21.15.116:0/4220718807 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d0003e50 con 0x5583671e3540 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46c5600640 1 -- 172.21.15.116:0/4220718807 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f46c0012a30 con 0x5583671e3540 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 10 client.26660 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 10 client.26660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 20 client.26660 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5583671f8a20 0x5583671fae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 4 client.26660 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.166+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558367155270 con 0x5583671f8a20 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.167+0000 7f46dca00640 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5583671f8a20 0x5583671fae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.167+0000 7f46dca00640 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5583671f8a20 0x5583671fae60 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0f6760005a382d5 server_cookie=8e51f7be957a7618 in_seq=0 out_seq=0 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.167+0000 7f46c5600640 10 client.26660 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:05.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46c5600640 1 -- 172.21.15.116:0/4220718807 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558367155270 con 0x5583671f8a20 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46c5600640 10 client.26660 handle_command_reply: tid=0 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 client.26660 shutdown 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 msgr2=0x5583671ed4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671ed4b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f46c000bf50 tx=0x7f46c0011870 comp rx=0 tx=0).stop 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 shutdown_connections 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5583671f8a20 0x5583671fae60 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e86b0 0x5583671f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671ed4b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 --2- 172.21.15.116:0/4220718807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e2b50 0x5583671ecf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46c5600640 0 client.26660 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 >> 172.21.15.116:0/4220718807 conn(0x5583671de0c0 msgr2=0x5583671dd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 shutdown_connections 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 1 -- 172.21.15.116:0/4220718807 wait complete. 2024-03-04T07:46:05.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.168+0000 7f46e15df740 20 client.26660 trim_cache size 0 max 16384 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.185+0000 7f46e15df740 1 Processor -- start 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46e15df740 1 -- start start 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46e15df740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671dcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46e15df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46e15df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671dd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46e15df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d81a8cf0 con 0x5583671e86b0 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46e15df740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d81ab2d0 con 0x5583671e2b50 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46e15df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d81a92f0 con 0x5583671e3540 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46dca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671dcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46cfe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46dca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671dcad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60388/0 (socket says 172.21.15.116:60388) 2024-03-04T07:46:05.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46dca00640 1 -- 172.21.15.116:0/2079163843 learned_addr learned my addr 172.21.15.116:0/2079163843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:05.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.186+0000 7f46dca00640 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46dca00640 1 -- 172.21.15.116:0/2079163843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 msgr2=0x5583671f44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46dca00640 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671f44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46dca00640 1 -- 172.21.15.116:0/2079163843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 msgr2=0x5583671dd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46dca00640 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671dd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46dca00640 1 -- 172.21.15.116:0/2079163843 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583671e3dd0 con 0x5583671e2b50 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46dca00640 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671dcad0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f46d000a010 tx=0x7f46d000ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72dd66a3ff875579 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46c7e00640 1 -- 172.21.15.116:0/2079163843 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0023040 con 0x5583671e2b50 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46c7e00640 1 -- 172.21.15.116:0/2079163843 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d0024040 con 0x5583671e2b50 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.187+0000 7f46c7e00640 1 -- 172.21.15.116:0/2079163843 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0005720 con 0x5583671e2b50 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 -- 172.21.15.116:0/2079163843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 msgr2=0x5583671dcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671dcad0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f46d000a010 tx=0x7f46d000ff70 comp rx=0 tx=0).stop 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 -- 172.21.15.116:0/2079163843 shutdown_connections 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671dd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e3540 0x5583671f44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 --2- 172.21.15.116:0/2079163843 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e2b50 0x5583671dcad0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 -- 172.21.15.116:0/2079163843 >> 172.21.15.116:0/2079163843 conn(0x5583671de0c0 msgr2=0x5583671e1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 -- 172.21.15.116:0/2079163843 shutdown_connections 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 -- 172.21.15.116:0/2079163843 wait complete. 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 -- start start 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.188+0000 7f46e15df740 1 Processor -- start 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46e15df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e2b50 0x5583671f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46e15df740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671f3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46e15df740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671f3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46e15df740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d81a9950 con 0x5583671e86b0 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46e15df740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d81aac70 con 0x5583671e3540 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46e15df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d81a9fb0 con 0x5583671e2b50 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46e15df740 10 client.0 _ll_register_callbacks cb 0x7f46c000bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60400/0 (socket says 172.21.15.116:60400) 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 -- 172.21.15.116:0/2570626470 learned_addr learned my addr 172.21.15.116:0/2570626470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46dca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46dca00640 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e2b50 0x5583671f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 -- 172.21.15.116:0/2570626470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e2b50 msgr2=0x5583671f2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e2b50 0x5583671f2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 -- 172.21.15.116:0/2570626470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 msgr2=0x5583671f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.189+0000 7f46cfe00640 1 -- 172.21.15.116:0/2570626470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55836725cb00 con 0x5583671e3540 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46cfe00640 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671f3240 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006010 tx=0x7f46c0005700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=366229f373d875f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46c5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46c5600640 1 -- 172.21.15.116:0/2570626470 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0034070 con 0x5583671e3540 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46c5600640 1 -- 172.21.15.116:0/2570626470 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f46c001ba80 con 0x5583671e3540 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46d0024040 con 0x5583671e3540 2024-03-04T07:46:05.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46c5600640 1 -- 172.21.15.116:0/2570626470 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46c001bc80 con 0x5583671e3540 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46e15df740 10 client.26666 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55836725de50 con 0x5583671e3540 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.190+0000 7f46c5600640 1 -- 172.21.15.116:0/2570626470 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d0024040 con 0x5583671e3540 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46c5600640 1 -- 172.21.15.116:0/2570626470 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f46c0037020 con 0x5583671e3540 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46e15df740 10 client.26666 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46e15df740 10 client.26666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46e15df740 20 client.26666 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46e15df740 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55836725e7a0 0x558367260bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46e15df740 4 client.26666 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55836714f5c0 con 0x55836725e7a0 2024-03-04T07:46:05.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46dca00640 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55836725e7a0 0x558367260bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46dca00640 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55836725e7a0 0x558367260bc0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0327db604efdea8 server_cookie=bd290e9e305c248a in_seq=0 out_seq=0 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46c5600640 10 client.26666 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.191+0000 7f46c5600640 1 -- 172.21.15.116:0/2570626470 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55836714f5c0 con 0x55836725e7a0 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46c5600640 10 client.26666 handle_command_reply: tid=0 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 client.26666 shutdown 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 msgr2=0x5583671f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671f3240 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006010 tx=0x7f46c0005700 comp rx=0 tx=0).stop 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 shutdown_connections 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55836725e7a0 0x558367260bc0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5583671e86b0 0x5583671f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5583671e3540 0x5583671f3240 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46c5600640 0 client.26666 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 --2- 172.21.15.116:0/2570626470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5583671e2b50 0x5583671f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 >> 172.21.15.116:0/2570626470 conn(0x5583671de0c0 msgr2=0x5583671dcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 shutdown_connections 2024-03-04T07:46:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 1 -- 172.21.15.116:0/2570626470 wait complete. 2024-03-04T07:46:05.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.192+0000 7f46e15df740 20 client.26666 trim_cache size 0 max 16384 2024-03-04T07:46:05.199 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:05.199 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:46:05.199 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:46:05.199 INFO:teuthology.orchestra.run.smithi116.stdout: "3ae8ca4f-1a62-4664-a709-bad3cf76cded": { 2024-03-04T07:46:05.199 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:46:05.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "3ae8ca4f-1a62-4664-a709-bad3cf76cded", 2024-03-04T07:46:05.200 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:46:05.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:46:05.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:46:05.200 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46bc0a1e00 msgr2=0x7f46bc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46bc0a1e00 0x7f46bc0a42c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f46c80bc8a0 tx=0x7f46c802bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8100550 msgr2=0x7f46d8199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8100550 0x7f46d8199bc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f46c8000c00 tx=0x7f46c802c6c0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 shutdown_connections 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f46d8102100 0x7f46d81a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f46bc0a1e00 0x7f46bc0a42c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46d8101740 0x7f46d819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 --2- 172.21.15.116:0/3255227119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d8100550 0x7f46d8199bc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 >> 172.21.15.116:0/3255227119 conn(0x7f46d80fbb00 msgr2=0x7f46d80fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 shutdown_connections 2024-03-04T07:46:05.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:05.193+0000 7f46dea00640 1 -- 172.21.15.116:0/3255227119 wait complete. 2024-03-04T07:46:05.209 INFO:tasks.cephfs.filesystem:scrub status for tag:3ae8ca4f-1a62-4664-a709-bad3cf76cded - {'path': '/', 'tag': '3ae8ca4f-1a62-4664-a709-bad3cf76cded', 'options': 'recursive,force'} 2024-03-04T07:46:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:04 smithi152 ceph-mon[37154]: pgmap v560: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:46:05.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:04 smithi133 ceph-mon[36505]: pgmap v560: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-03-04T07:46:06.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:05 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:46:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:05 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:05 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:46:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:05 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:06.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:05 smithi116 ceph-mon[32150]: pgmap v561: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:46:06.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:05 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:46:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:05 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:05 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:46:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:05 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:05 smithi152 ceph-mon[37154]: pgmap v561: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:46:06.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:05 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:46:06.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:05 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:06.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:05 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:46:06.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:05 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:06.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:05 smithi133 ceph-mon[36505]: pgmap v561: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:46:07.669 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.669 INFO:tasks.workunit.client.0.smithi116.stdout:189/600: open fd = 3 2024-03-04T07:46:07.669 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.669 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.669 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.669 INFO:tasks.workunit.client.0.smithi116.stdout:190/600: open fd = 3 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:191/600: open fd = 3 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:192/600: open fd = 3 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:193/600: open fd = 3 2024-03-04T07:46:07.670 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:194/600: open fd = 3 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:195/600: open fd = 3 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:196/600: open fd = 3 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:197/600: open fd = 3 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:198/600: open fd = 3 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.672 INFO:tasks.workunit.client.0.smithi116.stdout:199/600: open fd = 3 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:200/600: open fd = 3 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:201/600: open fd = 3 2024-03-04T07:46:07.673 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.674 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.674 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.674 INFO:tasks.workunit.client.0.smithi116.stdout:202/600: open fd = 3 2024-03-04T07:46:07.674 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.675 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.675 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.675 INFO:tasks.workunit.client.0.smithi116.stdout:203/600: open fd = 3 2024-03-04T07:46:07.675 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.675 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:204/600: open fd = 3 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:205/600: open fd = 3 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.676 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:206/600: open fd = 3 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:207/600: open fd = 3 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.677 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:208/600: open fd = 3 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:209/600: open fd = 3 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.678 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:210/600: open fd = 3 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:211/600: open fd = 3 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:212/600: open fd = 3 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.679 INFO:tasks.workunit.client.0.smithi116.stdout:213/600: open fd = 3 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:214/600: open fd = 3 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:215/600: open fd = 3 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:216/600: open fd = 3 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.680 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:217/600: open fd = 3 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:218/600: open fd = 3 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:219/600: open fd = 3 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.681 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:220/600: open fd = 3 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:221/600: open fd = 3 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:222/600: open fd = 3 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:223/600: open fd = 3 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.682 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.683 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.683 INFO:tasks.workunit.client.0.smithi116.stdout:224/600: open fd = 3 2024-03-04T07:46:07.683 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.683 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.683 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.683 INFO:tasks.workunit.client.0.smithi116.stdout:225/600: open fd = 3 2024-03-04T07:46:07.683 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:226/600: open fd = 3 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:227/600: open fd = 3 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:228/600: open fd = 3 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.684 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:229/600: open fd = 3 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:230/600: open fd = 3 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:231/600: open fd = 3 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:232/600: open fd = 3 2024-03-04T07:46:07.685 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:233/600: open fd = 3 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:234/600: open fd = 3 2024-03-04T07:46:07.686 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:235/600: open fd = 3 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:236/600: open fd = 3 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:237/600: open fd = 3 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.687 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:238/600: open fd = 3 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:239/600: open fd = 3 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:240/600: open fd = 3 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:241/600: open fd = 3 2024-03-04T07:46:07.688 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:242/600: open fd = 3 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:243/600: open fd = 3 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:244/600: open fd = 3 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.689 INFO:tasks.workunit.client.0.smithi116.stdout:245/600: open fd = 3 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:246/600: open fd = 3 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:247/600: open fd = 3 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:248/600: open fd = 3 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.690 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:249/600: open fd = 3 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:250/600: open fd = 3 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:07.691 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:46:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:08 smithi116 ceph-mon[32150]: pgmap v562: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:46:09.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:08 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:46:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:08 smithi152 ceph-mon[37154]: pgmap v562: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:46:09.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:08 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:46:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:08 smithi133 ceph-mon[36505]: pgmap v562: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:46:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:08 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:46:11.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:10 smithi116 ceph-mon[32150]: pgmap v563: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:46:11.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:10 smithi152 ceph-mon[37154]: pgmap v563: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:46:11.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:10 smithi133 ceph-mon[36505]: pgmap v563: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:46:13.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:12 smithi116 ceph-mon[32150]: pgmap v564: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:46:13.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:12 smithi152 ceph-mon[37154]: pgmap v564: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:46:13.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:12 smithi133 ceph-mon[36505]: pgmap v564: 97 pgs: 97 active+clean; 3.9 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 -- 172.21.15.116:0/2148905213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc107c30 msgr2=0x7f28bc108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 --2- 172.21.15.116:0/2148905213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc107c30 0x7f28bc108050 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009900 tx=0x7f28ac02bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 -- 172.21.15.116:0/2148905213 shutdown_connections 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 --2- 172.21.15.116:0/2148905213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28bc1097e0 0x7f28bc110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 --2- 172.21.15.116:0/2148905213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28bc108e20 0x7f28bc1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 --2- 172.21.15.116:0/2148905213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc107c30 0x7f28bc108050 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 -- 172.21.15.116:0/2148905213 >> 172.21.15.116:0/2148905213 conn(0x7f28bc076be0 msgr2=0x7f28bc079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 -- 172.21.15.116:0/2148905213 shutdown_connections 2024-03-04T07:46:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.890+0000 7f28c2000640 1 -- 172.21.15.116:0/2148905213 wait complete. 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.891+0000 7f28c2000640 1 Processor -- start 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28c2000640 1 -- start start 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28c2000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28bc107c30 0x7f28bc19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28c2000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28bc108e20 0x7f28bc19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28c2000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc1097e0 0x7f28bc1a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28c2000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc113900 con 0x7f28bc108e20 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28c2000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc113780 con 0x7f28bc1097e0 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28c2000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc113a80 con 0x7f28bc107c30 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28bc108e20 0x7f28bc19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28bbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc1097e0 0x7f28bc1a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28bc108e20 0x7f28bc19e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51174/0 (socket says 172.21.15.116:51174) 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28baa00640 1 -- 172.21.15.116:0/1221135534 learned_addr learned my addr 172.21.15.116:0/1221135534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.892+0000 7f28bb400640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28bc107c30 0x7f28bc19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28bbe00640 1 -- 172.21.15.116:0/1221135534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28bc107c30 msgr2=0x7f28bc19df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28bbe00640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28bc107c30 0x7f28bc19df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28bbe00640 1 -- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28bc108e20 msgr2=0x7f28bc19e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28bbe00640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28bc108e20 0x7f28bc19e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28bbe00640 1 -- 172.21.15.116:0/1221135534 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc1a5b60 con 0x7f28bc1097e0 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28bbe00640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc1097e0 0x7f28bc1a54f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009fd0 tx=0x7f28b400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28b3e00640 1 -- 172.21.15.116:0/1221135534 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b400ce20 con 0x7f28bc1097e0 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28b3e00640 1 -- 172.21.15.116:0/1221135534 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28b4005020 con 0x7f28bc1097e0 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28b3e00640 1 -- 172.21.15.116:0/1221135534 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4010720 con 0x7f28bc1097e0 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc1a5e50 con 0x7f28bc1097e0 2024-03-04T07:46:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.893+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc1a62c0 con 0x7f28bc1097e0 2024-03-04T07:46:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.896+0000 7f28b3e00640 1 -- 172.21.15.116:0/1221135534 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f28b4005540 con 0x7f28bc1097e0 2024-03-04T07:46:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.896+0000 7f28c4e35740 1 Processor -- start 2024-03-04T07:46:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28c4e35740 1 -- start start 2024-03-04T07:46:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91897850 0x559f91897c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f9189cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f9189f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28c4e35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f917fd370 con 0x559f9189d3b0 2024-03-04T07:46:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28c4e35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f917a6550 con 0x559f91897850 2024-03-04T07:46:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28c4e35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f917f9410 con 0x559f91898240 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28baa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91897850 0x559f91897c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28baa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91897850 0x559f91897c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60420/0 (socket says 172.21.15.116:60420) 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28baa00640 1 -- 172.21.15.116:0/1370602698 learned_addr learned my addr 172.21.15.116:0/1370602698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28baa00640 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f9189f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.897+0000 7f28bb400640 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f9189cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28b3e00640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f28a00a1e20 0x7f28a00a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28baa00640 1 -- 172.21.15.116:0/1370602698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 msgr2=0x559f9189cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28baa00640 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f9189cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28baa00640 1 -- 172.21.15.116:0/1370602698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 msgr2=0x559f9189f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28baa00640 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f9189f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28baa00640 1 -- 172.21.15.116:0/1370602698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f917ba3c0 con 0x559f91897850 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28b3e00640 1 -- 172.21.15.116:0/1221135534 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f28b40ce3a0 con 0x7f28bc1097e0 2024-03-04T07:46:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28bbe00640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f28a00a1e20 0x7f28a00a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28baa00640 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91897850 0x559f91897c70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f28a80118d0 tx=0x7f28a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea1079f0172cd19a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.898+0000 7f28a7e00640 1 -- 172.21.15.116:0/1370602698 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8005020 con 0x559f91897850 2024-03-04T07:46:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28a7e00640 1 -- 172.21.15.116:0/1370602698 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28a80047a0 con 0x559f91897850 2024-03-04T07:46:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28a7e00640 1 -- 172.21.15.116:0/1370602698 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a800fa20 con 0x559f91897850 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 -- 172.21.15.116:0/1370602698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91897850 msgr2=0x559f91897c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91897850 0x559f91897c70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f28a80118d0 tx=0x7f28a80046c0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28bbe00640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f28a00a1e20 0x7f28a00a42e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f28b4005ba0 tx=0x7f28b4005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 -- 172.21.15.116:0/1370602698 shutdown_connections 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f9189f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f9189cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 --2- 172.21.15.116:0/1370602698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91897850 0x559f91897c70 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 -- 172.21.15.116:0/1370602698 >> 172.21.15.116:0/1370602698 conn(0x559f91892dc0 msgr2=0x559f918931c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 -- 172.21.15.116:0/1370602698 shutdown_connections 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.899+0000 7f28c4e35740 1 -- 172.21.15.116:0/1370602698 wait complete. 2024-03-04T07:46:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.900+0000 7f28c4e35740 1 -- start start 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.900+0000 7f28c4e35740 1 Processor -- start 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28c4e35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f918a0360 con 0x559f91897850 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28c4e35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f917b99b0 con 0x559f9189d3b0 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28c4e35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f918a04e0 con 0x559f91898240 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28c4e35740 10 client.0 _ll_register_callbacks cb 0x559f917f5340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51194/0 (socket says 172.21.15.116:51194) 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28baa00640 1 -- 172.21.15.116:0/3533034341 learned_addr learned my addr 172.21.15.116:0/3533034341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28bb400640 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28baa00640 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28bb400640 1 -- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 msgr2=0x559f918a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28bb400640 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28bb400640 1 -- 172.21.15.116:0/3533034341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 msgr2=0x559f918a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28bb400640 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.901+0000 7f28bb400640 1 -- 172.21.15.116:0/3533034341 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f91512210 con 0x559f91898240 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28bb400640 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a2100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009230 tx=0x7f28ac004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf9d96a827a8ea03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28a4c00640 1 -- 172.21.15.116:0/3533034341 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac005d50 con 0x559f91898240 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28a4c00640 1 -- 172.21.15.116:0/3533034341 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f28ac005040 con 0x559f91898240 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28a80047a0 con 0x559f91898240 2024-03-04T07:46:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28a4c00640 1 -- 172.21.15.116:0/3533034341 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28ac005240 con 0x559f91898240 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28c4e35740 10 client.26581 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.902+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559f915de620 con 0x559f91898240 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28a4c00640 1 -- 172.21.15.116:0/3533034341 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28a80047a0 con 0x559f91898240 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28a4c00640 1 -- 172.21.15.116:0/3533034341 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f28ac0035f0 con 0x559f91898240 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28c4e35740 10 client.26581 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28c4e35740 10 client.26581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28c4e35740 20 client.26581 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f918ad9b0 0x559f918afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28c4e35740 4 client.26581 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f9180a8b0 con 0x559f918ad9b0 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.903+0000 7f28baa00640 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f918ad9b0 0x559f918afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.904+0000 7f28baa00640 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f918ad9b0 0x559f918afdc0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=148ece5284dc3216 server_cookie=a2ba50d7613c1741 in_seq=0 out_seq=0 2024-03-04T07:46:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.904+0000 7f28a4c00640 10 client.26581 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.904+0000 7f28a4c00640 1 -- 172.21.15.116:0/3533034341 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559f9180a8b0 con 0x559f918ad9b0 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.904+0000 7f28a4c00640 10 client.26581 handle_command_reply: tid=0 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 client.26581 shutdown 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 msgr2=0x559f918a2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a2100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009230 tx=0x7f28ac004270 comp rx=0 tx=0).stop 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 shutdown_connections 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f918ad9b0 0x559f918afdc0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a2100 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28a4c00640 0 client.26581 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3533034341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 >> 172.21.15.116:0/3533034341 conn(0x559f91892dc0 msgr2=0x559f91891fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 shutdown_connections 2024-03-04T07:46:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 1 -- 172.21.15.116:0/3533034341 wait complete. 2024-03-04T07:46:14.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.905+0000 7f28c4e35740 20 client.26581 trim_cache size 0 max 16384 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 Processor -- start 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 -- start start 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91897850 0x559f918995a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91898240 0x559f918a9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f91899ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1ad110 con 0x559f9189d3b0 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1af6f0 con 0x559f91898240 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28c4e35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1ad710 con 0x559f91897850 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.920+0000 7f28bb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91898240 0x559f918a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91898240 0x559f918a9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60440/0 (socket says 172.21.15.116:60440) 2024-03-04T07:46:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 -- 172.21.15.116:0/3151940655 learned_addr learned my addr 172.21.15.116:0/3151940655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f91899ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28baa00640 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91897850 0x559f918995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 -- 172.21.15.116:0/3151940655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91897850 msgr2=0x559f918995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91897850 0x559f918995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 -- 172.21.15.116:0/3151940655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 msgr2=0x559f91899ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f91899ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 -- 172.21.15.116:0/3151940655 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f918b0440 con 0x559f91898240 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28baa00640 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f91899ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28bb400640 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91898240 0x559f918a9100 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f28ac003a60 tx=0x7f28ac000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46f0a05ba39e7c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28a7400640 1 -- 172.21.15.116:0/3151940655 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac059070 con 0x559f91898240 2024-03-04T07:46:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28a7400640 1 -- 172.21.15.116:0/3151940655 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28ac003bd0 con 0x559f91898240 2024-03-04T07:46:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.921+0000 7f28a7400640 1 -- 172.21.15.116:0/3151940655 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac004c40 con 0x559f91898240 2024-03-04T07:46:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 -- 172.21.15.116:0/3151940655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91898240 msgr2=0x559f918a9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91898240 0x559f918a9100 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f28ac003a60 tx=0x7f28ac000bc0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 -- 172.21.15.116:0/3151940655 shutdown_connections 2024-03-04T07:46:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9189d3b0 0x559f91899ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f91898240 0x559f918a9100 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 --2- 172.21.15.116:0/3151940655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91897850 0x559f918995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 -- 172.21.15.116:0/3151940655 >> 172.21.15.116:0/3151940655 conn(0x559f91892dc0 msgr2=0x559f918931a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 -- 172.21.15.116:0/3151940655 shutdown_connections 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 -- 172.21.15.116:0/3151940655 wait complete. 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.922+0000 7f28c4e35740 1 -- start start 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 1 Processor -- start 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1add70 con 0x559f91897850 2024-03-04T07:46:14.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1af090 con 0x559f9189d3b0 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1ae3d0 con 0x559f91898240 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28c4e35740 10 client.0 _ll_register_callbacks cb 0x7f28ac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28baa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51204/0 (socket says 172.21.15.116:51204) 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28baa00640 1 -- 172.21.15.116:0/2300414374 learned_addr learned my addr 172.21.15.116:0/2300414374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28bb400640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.923+0000 7f28baa00640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28baa00640 1 -- 172.21.15.116:0/2300414374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 msgr2=0x559f918a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28baa00640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28baa00640 1 -- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 msgr2=0x559f918a7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28baa00640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28baa00640 1 -- 172.21.15.116:0/2300414374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f918a85c0 con 0x559f91897850 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28bb400640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28baa00640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a6a80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f28a8011710 tx=0x7f28a8013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca78bf5250f921c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28a4c00640 1 -- 172.21.15.116:0/2300414374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80047a0 con 0x559f91897850 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28a4c00640 1 -- 172.21.15.116:0/2300414374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f28a8004240 con 0x559f91897850 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.924+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28ac003bd0 con 0x559f91897850 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28a4c00640 1 -- 172.21.15.116:0/2300414374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28a8004460 con 0x559f91897850 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 10 client.16557 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559f919094d0 con 0x559f91897850 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28a4c00640 1 -- 172.21.15.116:0/2300414374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28ac003bd0 con 0x559f91897850 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28a4c00640 1 -- 172.21.15.116:0/2300414374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f28a801d030 con 0x559f91897850 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 10 client.16557 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 10 client.16557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:14.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 20 client.16557 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f91909c50 0x559f9190c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 4 client.16557 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:46:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559f91804c00 con 0x559f91909c50 2024-03-04T07:46:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.925+0000 7f28bb400640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f91909c50 0x559f9190c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.926+0000 7f28bb400640 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f91909c50 0x559f9190c070 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e52e82058bfe234f server_cookie=5c3ec656c9a8d68 in_seq=0 out_seq=0 2024-03-04T07:46:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.926+0000 7f28a4c00640 10 client.16557 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.941+0000 7f28a4c00640 1 -- 172.21.15.116:0/2300414374 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559f91804c00 con 0x559f91909c50 2024-03-04T07:46:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28a4c00640 10 client.16557 handle_command_reply: tid=0 2024-03-04T07:46:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 client.16557 shutdown 2024-03-04T07:46:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 msgr2=0x559f918a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a6a80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f28a8011710 tx=0x7f28a8013040 comp rx=0 tx=0).stop 2024-03-04T07:46:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 shutdown_connections 2024-03-04T07:46:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f91909c50 0x559f9190c070 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f9189d3b0 0x559f918a7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f91898240 0x559f918a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 --2- 172.21.15.116:0/2300414374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f91897850 0x559f918a6a80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 >> 172.21.15.116:0/2300414374 conn(0x559f91892dc0 msgr2=0x559f918a1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28a4c00640 0 client.16557 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 shutdown_connections 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 1 -- 172.21.15.116:0/2300414374 wait complete. 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.942+0000 7f28c4e35740 20 client.16557 trim_cache size 0 max 16384 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:46:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:46:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:46:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:46:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f28a00a1e20 msgr2=0x7f28a00a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f28a00a1e20 0x7f28a00a42e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f28b4005ba0 tx=0x7f28b4005940 comp rx=0 tx=0).stop 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc1097e0 msgr2=0x7f28bc1a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc1097e0 0x7f28bc1a54f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009fd0 tx=0x7f28b400ef10 comp rx=0 tx=0).stop 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 shutdown_connections 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f28bc1097e0 0x7f28bc1a54f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28bc108e20 0x7f28bc19e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f28a00a1e20 0x7f28a00a42e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 --2- 172.21.15.116:0/1221135534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28bc107c30 0x7f28bc19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.943+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 >> 172.21.15.116:0/1221135534 conn(0x7f28bc076be0 msgr2=0x7f28bc10c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.944+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 shutdown_connections 2024-03-04T07:46:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:14.944+0000 7f28c2000640 1 -- 172.21.15.116:0/1221135534 wait complete. 2024-03-04T07:46:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:14 smithi116 ceph-mon[32150]: pgmap v565: 97 pgs: 97 active+clean; 4.0 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-03-04T07:46:15.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:14 smithi152 ceph-mon[37154]: pgmap v565: 97 pgs: 97 active+clean; 4.0 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-03-04T07:46:15.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:14 smithi133 ceph-mon[36505]: pgmap v565: 97 pgs: 97 active+clean; 4.0 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-03-04T07:46:17.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:16 smithi152 ceph-mon[37154]: pgmap v566: 97 pgs: 97 active+clean; 4.0 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-04T07:46:17.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:16 smithi133 ceph-mon[36505]: pgmap v566: 97 pgs: 97 active+clean; 4.0 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-04T07:46:17.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:16 smithi116 ceph-mon[32150]: pgmap v566: 97 pgs: 97 active+clean; 4.0 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-04T07:46:19.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:18 smithi152 ceph-mon[37154]: pgmap v567: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:19.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:18 smithi133 ceph-mon[36505]: pgmap v567: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:18 smithi116 ceph-mon[32150]: pgmap v567: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:21.021 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:20 smithi152 ceph-mon[37154]: pgmap v568: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:46:21.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:20 smithi133 ceph-mon[36505]: pgmap v568: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:46:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:20 smithi116 ceph-mon[32150]: pgmap v568: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:46:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:22 smithi133 ceph-mon[36505]: pgmap v569: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:46:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:22 smithi116 ceph-mon[32150]: pgmap v569: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:46:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:22 smithi152 ceph-mon[37154]: pgmap v569: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:46:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:24 smithi133 ceph-mon[36505]: pgmap v570: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:24 smithi116 ceph-mon[32150]: pgmap v570: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:25.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:24 smithi152 ceph-mon[37154]: pgmap v570: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:26 smithi133 ceph-mon[36505]: pgmap v571: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:26 smithi116 ceph-mon[32150]: pgmap v571: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:27.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:26 smithi152 ceph-mon[37154]: pgmap v571: 97 pgs: 97 active+clean; 4.1 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 -- 172.21.15.116:0/4235880010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 msgr2=0x7ff0d81092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 --2- 172.21.15.116:0/4235880010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d81092a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009900 tx=0x7ff0c002bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 -- 172.21.15.116:0/4235880010 shutdown_connections 2024-03-04T07:46:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 --2- 172.21.15.116:0/4235880010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d81097e0 0x7ff0d8110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 --2- 172.21.15.116:0/4235880010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d81092a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 --2- 172.21.15.116:0/4235880010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d8107c30 0x7ff0d8108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 -- 172.21.15.116:0/4235880010 >> 172.21.15.116:0/4235880010 conn(0x7ff0d8076be0 msgr2=0x7ff0d8079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 -- 172.21.15.116:0/4235880010 shutdown_connections 2024-03-04T07:46:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.086+0000 7ff0dec00640 1 -- 172.21.15.116:0/4235880010 wait complete. 2024-03-04T07:46:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.087+0000 7ff0dec00640 1 Processor -- start 2024-03-04T07:46:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.087+0000 7ff0dec00640 1 -- start start 2024-03-04T07:46:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0dec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8107c30 0x7ff0d819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0dec00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0dec00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d81097e0 0x7ff0d81a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0dec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81138b0 con 0x7ff0d8107c30 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0dec00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d8113730 con 0x7ff0d8108e20 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0dec00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d8113a30 con 0x7ff0d81097e0 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8107c30 0x7ff0d819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0cf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8107c30 0x7ff0d819dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48152/0 (socket says 172.21.15.116:48152) 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0cf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44598/0 (socket says 172.21.15.116:44598) 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0d7e00640 1 -- 172.21.15.116:0/3678917117 learned_addr learned my addr 172.21.15.116:0/3678917117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0dcc00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d81097e0 0x7ff0d81a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0cf400640 1 -- 172.21.15.116:0/3678917117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d81097e0 msgr2=0x7ff0d81a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0cf400640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d81097e0 0x7ff0d81a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0cf400640 1 -- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8107c30 msgr2=0x7ff0d819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0cf400640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8107c30 0x7ff0d819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0cf400640 1 -- 172.21.15.116:0/3678917117 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d81a5ae0 con 0x7ff0d8108e20 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.088+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8107c30 0x7ff0d819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.089+0000 7ff0cf400640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d819e400 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009230 tx=0x7ff0c002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.089+0000 7ff0d5600640 1 -- 172.21.15.116:0/3678917117 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0041070 con 0x7ff0d8108e20 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.089+0000 7ff0d5600640 1 -- 172.21.15.116:0/3678917117 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c0004450 con 0x7ff0d8108e20 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.089+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d81a5d70 con 0x7ff0d8108e20 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.089+0000 7ff0d5600640 1 -- 172.21.15.116:0/3678917117 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0003df0 con 0x7ff0d8108e20 2024-03-04T07:46:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.089+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d81a6250 con 0x7ff0d8108e20 2024-03-04T07:46:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.092+0000 7ff0e19b9740 1 Processor -- start 2024-03-04T07:46:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.092+0000 7ff0d5600640 1 -- 172.21.15.116:0/3678917117 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff0c0044070 con 0x7ff0d8108e20 2024-03-04T07:46:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.092+0000 7ff0e19b9740 1 -- start start 2024-03-04T07:46:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f625c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f62d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56107f58b370 con 0x56107f625850 2024-03-04T07:46:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56107f534550 con 0x56107f62b3b0 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56107f587410 con 0x56107f626240 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0dcc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f625c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0cf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f62d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0cf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f62d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44614/0 (socket says 172.21.15.116:44614) 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0cf400640 1 -- 172.21.15.116:0/788053438 learned_addr learned my addr 172.21.15.116:0/788053438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0dcc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f625c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48160/0 (socket says 172.21.15.116:48160) 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0cf400640 1 -- 172.21.15.116:0/788053438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 msgr2=0x56107f62ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.093+0000 7ff0cf400640 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0cf400640 1 -- 172.21.15.116:0/788053438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 msgr2=0x56107f625c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0cf400640 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f625c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0cf400640 1 -- 172.21.15.116:0/788053438 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56107f5483c0 con 0x56107f62b3b0 2024-03-04T07:46:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0cf400640 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f62d840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00f5a90 tx=0x7ff0c00f5ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=402d669d48a348cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0d5600640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0bc0a1ea0 0x7ff0bc0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0cd200640 1 -- 172.21.15.116:0/788053438 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00fe070 con 0x56107f62b3b0 2024-03-04T07:46:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0cd200640 1 -- 172.21.15.116:0/788053438 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c00bc510 con 0x56107f62b3b0 2024-03-04T07:46:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0cd200640 1 -- 172.21.15.116:0/788053438 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00e3d20 con 0x56107f62b3b0 2024-03-04T07:46:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0d5600640 1 -- 172.21.15.116:0/3678917117 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff0c00ef770 con 0x7ff0d8108e20 2024-03-04T07:46:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.094+0000 7ff0dcc00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0bc0a1ea0 0x7ff0bc0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 -- 172.21.15.116:0/788053438 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 msgr2=0x56107f62d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f62d840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00f5a90 tx=0x7ff0c00f5ac0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0dcc00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0bc0a1ea0 0x7ff0bc0a4360 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x56107f626aa0 tx=0x7ff0d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 -- 172.21.15.116:0/788053438 shutdown_connections 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f62d840 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/788053438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f625c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 -- 172.21.15.116:0/788053438 >> 172.21.15.116:0/788053438 conn(0x56107f620dc0 msgr2=0x56107f6211c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 -- 172.21.15.116:0/788053438 shutdown_connections 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.095+0000 7ff0e19b9740 1 -- 172.21.15.116:0/788053438 wait complete. 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 -- start start 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 Processor -- start 2024-03-04T07:46:28.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f62f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f636d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56107f62e120 con 0x56107f62b3b0 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56107f5479b0 con 0x56107f625850 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56107f62e2a0 con 0x56107f626240 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.096+0000 7ff0e19b9740 10 client.0 _ll_register_callbacks cb 0x56107f583340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f62f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f62f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44622/0 (socket says 172.21.15.116:44622) 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 -- 172.21.15.116:0/844523094 learned_addr learned my addr 172.21.15.116:0/844523094 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0cf400640 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0dcc00640 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f636d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 -- 172.21.15.116:0/844523094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 msgr2=0x56107f62fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 -- 172.21.15.116:0/844523094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 msgr2=0x56107f636d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f636d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 -- 172.21.15.116:0/844523094 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56107f2a0210 con 0x56107f625850 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f62f810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800fe60 tx=0x7ff0c8012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42cbb65f8d72a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0c5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.097+0000 7ff0c5600640 1 -- 172.21.15.116:0/844523094 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8003970 con 0x56107f625850 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0c5600640 1 -- 172.21.15.116:0/844523094 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff0c8003b10 con 0x56107f625850 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0c00bc510 con 0x56107f625850 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0c5600640 1 -- 172.21.15.116:0/844523094 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0c8003d10 con 0x56107f625850 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 10 client.26696 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56107f36c620 con 0x56107f625850 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0c5600640 1 -- 172.21.15.116:0/844523094 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c00bc510 con 0x56107f625850 2024-03-04T07:46:28.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0c5600640 1 -- 172.21.15.116:0/844523094 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff0c80106a0 con 0x56107f625850 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 10 client.26696 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 10 client.26696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 20 client.26696 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f63b6c0 0x56107f63daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 4 client.26696 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.098+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56107f5988b0 con 0x56107f63b6c0 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.099+0000 7ff0cf400640 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f63b6c0 0x56107f63daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.099+0000 7ff0cf400640 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f63b6c0 0x56107f63daf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6773bfee22adf728 server_cookie=472d21a52b44784d in_seq=0 out_seq=0 2024-03-04T07:46:28.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.099+0000 7ff0c5600640 10 client.26696 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0c5600640 1 -- 172.21.15.116:0/844523094 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56107f5988b0 con 0x56107f63b6c0 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0c5600640 10 client.26696 handle_command_reply: tid=0 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 client.26696 shutdown 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 msgr2=0x56107f62f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f62f810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800fe60 tx=0x7ff0c8012820 comp rx=0 tx=0).stop 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 shutdown_connections 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f63b6c0 0x56107f63daf0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f636d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f62fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/844523094 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f62f810 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0c5600640 0 client.26696 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 >> 172.21.15.116:0/844523094 conn(0x56107f620dc0 msgr2=0x56107f61ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 shutdown_connections 2024-03-04T07:46:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.100+0000 7ff0e19b9740 1 -- 172.21.15.116:0/844523094 wait complete. 2024-03-04T07:46:28.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.101+0000 7ff0e19b9740 20 client.26696 trim_cache size 0 max 16384 2024-03-04T07:46:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.115+0000 7ff0e19b9740 1 Processor -- start 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0e19b9740 1 -- start start 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f624020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f636d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f624560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81acff0 con 0x56107f625850 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81af5d0 con 0x56107f62b3b0 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81ad5f0 con 0x56107f626240 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f624020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f624020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48182/0 (socket says 172.21.15.116:48182) 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2692470071 learned_addr learned my addr 172.21.15.116:0/2692470071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:28.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0dcc00640 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f624560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.116+0000 7ff0cf400640 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f636d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2692470071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 msgr2=0x56107f636d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f636d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2692470071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 msgr2=0x56107f624560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f624560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2692470071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56107f626620 con 0x56107f625850 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0dcc00640 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f624560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f624020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8000f80 tx=0x7ff0c8010ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f687d366c4749fca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0c7e00640 1 -- 172.21.15.116:0/2692470071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8035070 con 0x56107f625850 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0c7e00640 1 -- 172.21.15.116:0/2692470071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c8003970 con 0x56107f625850 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.117+0000 7ff0c7e00640 1 -- 172.21.15.116:0/2692470071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c802c040 con 0x56107f625850 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2692470071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 msgr2=0x56107f624020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f624020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8000f80 tx=0x7ff0c8010ee0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2692470071 shutdown_connections 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f62b3b0 0x56107f624560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f636d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2692470071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f625850 0x56107f624020 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2692470071 >> 172.21.15.116:0/2692470071 conn(0x56107f620dc0 msgr2=0x56107f62aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2692470071 shutdown_connections 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2692470071 wait complete. 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.118+0000 7ff0e19b9740 1 -- start start 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 1 Processor -- start 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f634650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f634b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f635b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81adc50 con 0x56107f62b3b0 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81aef70 con 0x56107f625850 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d81ae2b0 con 0x56107f626240 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0e19b9740 10 client.0 _ll_register_callbacks cb 0x7ff0c800f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0dcc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f635b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f634650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f634650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44644/0 (socket says 172.21.15.116:44644) 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2071907377 learned_addr learned my addr 172.21.15.116:0/2071907377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0dcc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f635b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48194/0 (socket says 172.21.15.116:48194) 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.119+0000 7ff0cf400640 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f634b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2071907377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 msgr2=0x56107f634b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f634b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2071907377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 msgr2=0x56107f635b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f635b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0d7e00640 1 -- 172.21.15.116:0/2071907377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56107f636120 con 0x56107f625850 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0d7e00640 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f634650 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005f40 tx=0x7ff0c8012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cad7bbff07a5ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0c5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0c5600640 1 -- 172.21.15.116:0/2071907377 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8035070 con 0x56107f625850 2024-03-04T07:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0c8003970 con 0x56107f625850 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0c5600640 1 -- 172.21.15.116:0/2071907377 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff0c8005420 con 0x56107f625850 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0c5600640 1 -- 172.21.15.116:0/2071907377 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0c802c040 con 0x56107f625850 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0e19b9740 10 client.26702 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.120+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56107f697170 con 0x56107f625850 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0c5600640 1 -- 172.21.15.116:0/2071907377 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0c8003970 con 0x56107f625850 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0c5600640 1 -- 172.21.15.116:0/2071907377 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff0c80057f0 con 0x56107f625850 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0e19b9740 10 client.26702 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0e19b9740 10 client.26702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0e19b9740 20 client.26702 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f697a00 0x56107f699e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0e19b9740 4 client.26702 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56107f592c00 con 0x56107f697a00 2024-03-04T07:46:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.121+0000 7ff0cf400640 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f697a00 0x56107f699e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.122+0000 7ff0cf400640 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f697a00 0x56107f699e20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5588557cfd290b77 server_cookie=4c06b66d694c18bc in_seq=0 out_seq=0 2024-03-04T07:46:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.122+0000 7ff0c5600640 10 client.26702 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0c5600640 1 -- 172.21.15.116:0/2071907377 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56107f592c00 con 0x56107f697a00 2024-03-04T07:46:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0c5600640 10 client.26702 handle_command_reply: tid=0 2024-03-04T07:46:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 client.26702 shutdown 2024-03-04T07:46:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 msgr2=0x56107f634650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f634650 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005f40 tx=0x7ff0c8012820 comp rx=0 tx=0).stop 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 shutdown_connections 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56107f697a00 0x56107f699e20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56107f62b3b0 0x56107f635b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56107f626240 0x56107f634b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 --2- 172.21.15.116:0/2071907377 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56107f625850 0x56107f634650 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0c5600640 0 client.26702 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.135+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 >> 172.21.15.116:0/2071907377 conn(0x56107f620dc0 msgr2=0x56107f624020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.136+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 shutdown_connections 2024-03-04T07:46:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.136+0000 7ff0e19b9740 1 -- 172.21.15.116:0/2071907377 wait complete. 2024-03-04T07:46:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.136+0000 7ff0e19b9740 20 client.26702 trim_cache size 0 max 16384 2024-03-04T07:46:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:46:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:46:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:46:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:46:28.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0bc0a1ea0 msgr2=0x7ff0bc0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0bc0a1ea0 0x7ff0bc0a4360 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x56107f626aa0 tx=0x7ff0d0005f90 comp rx=0 tx=0).stop 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 msgr2=0x7ff0d819e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d819e400 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009230 tx=0x7ff0c002c650 comp rx=0 tx=0).stop 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 shutdown_connections 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff0bc0a1ea0 0x7ff0bc0a4360 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0d81097e0 0x7ff0d81a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff0d8108e20 0x7ff0d819e400 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 --2- 172.21.15.116:0/3678917117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0d8107c30 0x7ff0d819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 >> 172.21.15.116:0/3678917117 conn(0x7ff0d8076be0 msgr2=0x7ff0d810c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 shutdown_connections 2024-03-04T07:46:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:28.137+0000 7ff0dec00640 1 -- 172.21.15.116:0/3678917117 wait complete. 2024-03-04T07:46:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:28 smithi133 ceph-mon[36505]: pgmap v572: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-03-04T07:46:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:28 smithi116 ceph-mon[32150]: pgmap v572: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-03-04T07:46:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:28 smithi152 ceph-mon[37154]: pgmap v572: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-03-04T07:46:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:30 smithi133 ceph-mon[36505]: pgmap v573: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:46:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:30 smithi116 ceph-mon[32150]: pgmap v573: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:46:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:30 smithi152 ceph-mon[37154]: pgmap v573: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:46:33.061 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:32 smithi116 ceph-mon[32150]: pgmap v574: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:33.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:32 smithi133 ceph-mon[36505]: pgmap v574: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:32 smithi152 ceph-mon[37154]: pgmap v574: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:34 smithi133 ceph-mon[36505]: pgmap v575: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:35.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:46:35.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:46:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:34 smithi116 ceph-mon[32150]: pgmap v575: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:34 smithi152 ceph-mon[37154]: pgmap v575: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:35.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 -- 172.21.15.116:0/23353260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1049e0 msgr2=0x7f8fbc104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/23353260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1049e0 0x7f8fbc104e60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009900 tx=0x7f8fa402bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:35.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 -- 172.21.15.116:0/23353260 shutdown_connections 2024-03-04T07:46:35.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/23353260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fbc1053a0 0x7f8fbc10bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/23353260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1049e0 0x7f8fbc104e60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/23353260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fbc1037f0 0x7f8fbc103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 -- 172.21.15.116:0/23353260 >> 172.21.15.116:0/23353260 conn(0x7f8fbc0fed80 msgr2=0x7f8fbc1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.424+0000 7f8fc0e00640 1 -- 172.21.15.116:0/23353260 shutdown_connections 2024-03-04T07:46:35.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.425+0000 7f8fc0e00640 1 -- 172.21.15.116:0/23353260 wait complete. 2024-03-04T07:46:35.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 Processor -- start 2024-03-04T07:46:35.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 -- start start 2024-03-04T07:46:35.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fbc1037f0 0x7f8fbc199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fbc1049e0 0x7f8fbc19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1053a0 0x7f8fbc1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc10f450 con 0x7f8fbc1049e0 2024-03-04T07:46:35.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc10f2d0 con 0x7f8fbc1053a0 2024-03-04T07:46:35.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fc0e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc10f5d0 con 0x7f8fbc1037f0 2024-03-04T07:46:35.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fba800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1053a0 0x7f8fbc1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fb3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fbc1049e0 0x7f8fbc19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fb3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fbc1049e0 0x7f8fbc19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48218/0 (socket says 172.21.15.116:48218) 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fba800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1053a0 0x7f8fbc1a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44662/0 (socket says 172.21.15.116:44662) 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.426+0000 7f8fb3e00640 1 -- 172.21.15.116:0/27858135 learned_addr learned my addr 172.21.15.116:0/27858135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fbc1037f0 0x7f8fbc199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fba800640 1 -- 172.21.15.116:0/27858135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fbc1037f0 msgr2=0x7f8fbc199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fba800640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fbc1037f0 0x7f8fbc199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fba800640 1 -- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fbc1049e0 msgr2=0x7f8fbc19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fba800640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fbc1049e0 0x7f8fbc19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fba800640 1 -- 172.21.15.116:0/27858135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc1a1710 con 0x7f8fbc1053a0 2024-03-04T07:46:35.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fba800640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1053a0 0x7f8fbc1a10a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400e940 tx=0x7f8fb400ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fb2a00640 1 -- 172.21.15.116:0/27858135 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4004d20 con 0x7f8fbc1053a0 2024-03-04T07:46:35.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fb2a00640 1 -- 172.21.15.116:0/27858135 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fb4005020 con 0x7f8fbc1053a0 2024-03-04T07:46:35.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fb2a00640 1 -- 172.21.15.116:0/27858135 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40116b0 con 0x7f8fbc1053a0 2024-03-04T07:46:35.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.427+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc1a1a00 con 0x7f8fbc1053a0 2024-03-04T07:46:35.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.428+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc1a1ed0 con 0x7f8fbc1053a0 2024-03-04T07:46:35.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fb2a00640 1 -- 172.21.15.116:0/27858135 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8fb40054d0 con 0x7f8fbc1053a0 2024-03-04T07:46:35.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 Processor -- start 2024-03-04T07:46:35.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 -- start start 2024-03-04T07:46:35.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff53f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff59040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff5bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ffe81090 con 0x5625fff54540 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ffeb8d30 con 0x5625fff53b50 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.431+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ffe61a30 con 0x5625fff596b0 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff53f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff53f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44668/0 (socket says 172.21.15.116:44668) 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 -- 172.21.15.116:0/2927991445 learned_addr learned my addr 172.21.15.116:0/2927991445 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff59040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff5bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 -- 172.21.15.116:0/2927991445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 msgr2=0x5625fff5bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff5bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 -- 172.21.15.116:0/2927991445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 msgr2=0x5625fff59040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff59040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 -- 172.21.15.116:0/2927991445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625ffc977d0 con 0x5625fff53b50 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb2a00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fa00a1d80 0x7f8fa00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb2a00640 1 -- 172.21.15.116:0/27858135 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8fb40cde40 con 0x7f8fbc1053a0 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff53f70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4005c40 tx=0x7f8fa4005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e75b3f4e91c937a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.432+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fa00a1d80 0x7f8fa00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.433+0000 7f8fab400640 1 -- 172.21.15.116:0/2927991445 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa40046c0 con 0x5625fff53b50 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.433+0000 7f8fab400640 1 -- 172.21.15.116:0/2927991445 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004ed0 con 0x5625fff53b50 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.433+0000 7f8fab400640 1 -- 172.21.15.116:0/2927991445 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4003850 con 0x5625fff53b50 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 -- 172.21.15.116:0/2927991445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 msgr2=0x5625fff53f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fa00a1d80 0x7f8fa00a4240 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40096c0 tx=0x7f8fa4045040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff53f70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4005c40 tx=0x7f8fa4005b70 comp rx=0 tx=0).stop 2024-03-04T07:46:35.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 -- 172.21.15.116:0/2927991445 shutdown_connections 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff5bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff59040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/2927991445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff53f70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 -- 172.21.15.116:0/2927991445 >> 172.21.15.116:0/2927991445 conn(0x5625fff4f0c0 msgr2=0x5625fff4f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 -- 172.21.15.116:0/2927991445 shutdown_connections 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 -- 172.21.15.116:0/2927991445 wait complete. 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.434+0000 7f8fc3c28740 1 -- start start 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 1 Processor -- start 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff53b50 0x5625fff5df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff5e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff596b0 0x5625fff654a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ffe570c0 con 0x5625fff54540 2024-03-04T07:46:35.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625ffeb4dd0 con 0x5625fff596b0 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625fff5c660 con 0x5625fff53b50 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fc3c28740 10 client.0 _ll_register_callbacks cb 0x5625ffe74860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fb3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff5e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fba800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff596b0 0x5625fff654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fba800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff596b0 0x5625fff654a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44676/0 (socket says 172.21.15.116:44676) 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fba800640 1 -- 172.21.15.116:0/4116103273 learned_addr learned my addr 172.21.15.116:0/4116103273 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.435+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff53b50 0x5625fff5df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fba800640 1 -- 172.21.15.116:0/4116103273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff53b50 msgr2=0x5625fff5df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fba800640 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff53b50 0x5625fff5df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fba800640 1 -- 172.21.15.116:0/4116103273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 msgr2=0x5625fff5e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fba800640 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff5e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fba800640 1 -- 172.21.15.116:0/4116103273 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625ffbcd3f0 con 0x5625fff596b0 2024-03-04T07:46:35.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fba800640 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff596b0 0x5625fff654a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40a2da0 tx=0x7f8fb409abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61029a2b5fa63d2e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8f9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8f9fe00640 1 -- 172.21.15.116:0/4116103273 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb409be60 con 0x5625fff596b0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8f9fe00640 1 -- 172.21.15.116:0/4116103273 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8fb409a030 con 0x5625fff596b0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fa4004ed0 con 0x5625fff596b0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8f9fe00640 1 -- 172.21.15.116:0/4116103273 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fb401f070 con 0x5625fff596b0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fc3c28740 10 client.26714 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.436+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5625ffb8d0f0 con 0x5625fff596b0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8f9fe00640 1 -- 172.21.15.116:0/4116103273 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004ed0 con 0x5625fff596b0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8f9fe00640 1 -- 172.21.15.116:0/4116103273 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8fb400f040 con 0x5625fff596b0 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8fc3c28740 10 client.26714 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8fc3c28740 10 client.26714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8fc3c28740 20 client.26714 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fff69a10 0x5625fff6be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8fc3c28740 4 client.26714 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625ffec6270 con 0x5625fff69a10 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.437+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fff69a10 0x5625fff6be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.438+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fff69a10 0x5625fff6be70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf8ea62ede48e844 server_cookie=a76a2f953b0db8fc in_seq=0 out_seq=0 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.438+0000 7f8f9fe00640 10 client.26714 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.438+0000 7f8f9fe00640 1 -- 172.21.15.116:0/4116103273 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5625ffec6270 con 0x5625fff69a10 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.438+0000 7f8f9fe00640 10 client.26714 handle_command_reply: tid=0 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.438+0000 7f8fc3c28740 1 client.26714 shutdown 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff596b0 msgr2=0x5625fff654a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff596b0 0x5625fff654a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40a2da0 tx=0x7f8fb409abe0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 shutdown_connections 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fff69a10 0x5625fff6be70 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff596b0 0x5625fff654a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff5e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/4116103273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff53b50 0x5625fff5df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8f9fe00640 0 client.26714 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:35.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 >> 172.21.15.116:0/4116103273 conn(0x5625fff4f0c0 msgr2=0x5625fff4e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 shutdown_connections 2024-03-04T07:46:35.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 1 -- 172.21.15.116:0/4116103273 wait complete. 2024-03-04T07:46:35.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.439+0000 7f8fc3c28740 20 client.26714 trim_cache size 0 max 16384 2024-03-04T07:46:35.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 Processor -- start 2024-03-04T07:46:35.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 -- start start 2024-03-04T07:46:35.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff5d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff65460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff4dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1a8c50 con 0x5625fff54540 2024-03-04T07:46:35.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1ab230 con 0x5625fff53b50 2024-03-04T07:46:35.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.456+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1a9250 con 0x5625fff596b0 2024-03-04T07:46:35.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff65460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff5d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff65460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48238/0 (socket says 172.21.15.116:48238) 2024-03-04T07:46:35.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 -- 172.21.15.116:0/3684935792 learned_addr learned my addr 172.21.15.116:0/3684935792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:35.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff5d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44686/0 (socket says 172.21.15.116:44686) 2024-03-04T07:46:35.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fba800640 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff4dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 -- 172.21.15.116:0/3684935792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 msgr2=0x5625fff4dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff4dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 -- 172.21.15.116:0/3684935792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 msgr2=0x5625fff5d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff5d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb3e00640 1 -- 172.21.15.116:0/3684935792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625fff53790 con 0x5625fff54540 2024-03-04T07:46:35.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.457+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff5d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:35.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.458+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff65460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4036040 tx=0x7f8fa4053000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1e543351db704d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.458+0000 7f8faaa00640 1 -- 172.21.15.116:0/3684935792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa405d070 con 0x5625fff54540 2024-03-04T07:46:35.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.458+0000 7f8faaa00640 1 -- 172.21.15.116:0/3684935792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004ed0 con 0x5625fff54540 2024-03-04T07:46:35.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.458+0000 7f8faaa00640 1 -- 172.21.15.116:0/3684935792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4003850 con 0x5625fff54540 2024-03-04T07:46:35.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 -- 172.21.15.116:0/3684935792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 msgr2=0x5625fff65460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff65460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4036040 tx=0x7f8fa4053000 comp rx=0 tx=0).stop 2024-03-04T07:46:35.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 -- 172.21.15.116:0/3684935792 shutdown_connections 2024-03-04T07:46:35.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff596b0 0x5625fff4dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff54540 0x5625fff65460 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/3684935792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff5d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 -- 172.21.15.116:0/3684935792 >> 172.21.15.116:0/3684935792 conn(0x5625fff4f0c0 msgr2=0x5625fff55a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 -- 172.21.15.116:0/3684935792 shutdown_connections 2024-03-04T07:46:35.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 -- 172.21.15.116:0/3684935792 wait complete. 2024-03-04T07:46:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 -- start start 2024-03-04T07:46:35.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.459+0000 7f8fc3c28740 1 Processor -- start 2024-03-04T07:46:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff65020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff54540 0x5625fff636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fc3c28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff596b0 0x5625fff63c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1a98b0 con 0x5625fff596b0 2024-03-04T07:46:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1aabd0 con 0x5625fff53b50 2024-03-04T07:46:35.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fc3c28740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1a9f10 con 0x5625fff54540 2024-03-04T07:46:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fc3c28740 10 client.0 _ll_register_callbacks cb 0x7f8fb409a7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff65020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff65020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44700/0 (socket says 172.21.15.116:44700) 2024-03-04T07:46:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 -- 172.21.15.116:0/1870370862 learned_addr learned my addr 172.21.15.116:0/1870370862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fba800640 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff596b0 0x5625fff63c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff54540 0x5625fff636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 -- 172.21.15.116:0/1870370862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff54540 msgr2=0x5625fff636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff54540 0x5625fff636f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 -- 172.21.15.116:0/1870370862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff596b0 msgr2=0x5625fff63c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:35.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff596b0 0x5625fff63c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.460+0000 7f8fb9e00640 1 -- 172.21.15.116:0/1870370862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625fff64170 con 0x5625fff53b50 2024-03-04T07:46:35.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8fb9e00640 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff65020 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8fac009e50 tx=0x7f8fac020cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b961a3890ceb2bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8f9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:35.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8f9fe00640 1 -- 172.21.15.116:0/1870370862 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac0108e0 con 0x5625fff53b50 2024-03-04T07:46:35.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8f9fe00640 1 -- 172.21.15.116:0/1870370862 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8fac010a80 con 0x5625fff53b50 2024-03-04T07:46:35.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fa4004ed0 con 0x5625fff53b50 2024-03-04T07:46:35.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8f9fe00640 1 -- 172.21.15.116:0/1870370862 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fac030050 con 0x5625fff53b50 2024-03-04T07:46:35.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8fc3c28740 10 client.26720 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:35.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.461+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5625fffc6000 con 0x5625fff53b50 2024-03-04T07:46:35.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8f9fe00640 1 -- 172.21.15.116:0/1870370862 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004ed0 con 0x5625fff53b50 2024-03-04T07:46:35.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8f9fe00640 1 -- 172.21.15.116:0/1870370862 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8fac0283c0 con 0x5625fff53b50 2024-03-04T07:46:35.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8fc3c28740 10 client.26720 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:35.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8fc3c28740 10 client.26720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:35.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8fc3c28740 20 client.26720 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:35.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fffc6940 0x5625fffc8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8fc3c28740 4 client.26720 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:46:35.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5625ffec05c0 con 0x5625fffc6940 2024-03-04T07:46:35.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.462+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fffc6940 0x5625fffc8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8f9fe00640 10 client.26720 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fb3e00640 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fffc6940 0x5625fffc8d60 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf5b3d2c80b045d6 server_cookie=a5dc044207c9b866 in_seq=0 out_seq=0 2024-03-04T07:46:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8f9fe00640 1 -- 172.21.15.116:0/1870370862 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5625ffec05c0 con 0x5625fffc6940 2024-03-04T07:46:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8f9fe00640 10 client.26720 handle_command_reply: tid=0 2024-03-04T07:46:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 client.26720 shutdown 2024-03-04T07:46:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 msgr2=0x5625fff65020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff65020 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8fac009e50 tx=0x7f8fac020cf0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 shutdown_connections 2024-03-04T07:46:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5625fffc6940 0x5625fffc8d60 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5625fff596b0 0x5625fff63c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5625fff54540 0x5625fff636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 --2- 172.21.15.116:0/1870370862 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5625fff53b50 0x5625fff65020 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8f9fe00640 0 client.26720 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 >> 172.21.15.116:0/1870370862 conn(0x5625fff4f0c0 msgr2=0x5625fff52320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 shutdown_connections 2024-03-04T07:46:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 1 -- 172.21.15.116:0/1870370862 wait complete. 2024-03-04T07:46:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.463+0000 7f8fc3c28740 20 client.26720 trim_cache size 0 max 16384 2024-03-04T07:46:35.480 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:46:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:46:35.480 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fa00a1d80 msgr2=0x7f8fa00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fa00a1d80 0x7f8fa00a4240 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40096c0 tx=0x7f8fa4045040 comp rx=0 tx=0).stop 2024-03-04T07:46:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1053a0 msgr2=0x7f8fbc1a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1053a0 0x7f8fbc1a10a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400e940 tx=0x7f8fb400ee00 comp rx=0 tx=0).stop 2024-03-04T07:46:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 shutdown_connections 2024-03-04T07:46:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fbc1053a0 0x7f8fbc1a10a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fbc1049e0 0x7f8fbc19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fa00a1d80 0x7f8fa00a4240 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 --2- 172.21.15.116:0/27858135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8fbc1037f0 0x7f8fbc199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 >> 172.21.15.116:0/27858135 conn(0x7f8fbc0fed80 msgr2=0x7f8fbc107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 shutdown_connections 2024-03-04T07:46:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.464+0000 7f8fc0e00640 1 -- 172.21.15.116:0/27858135 wait complete. 2024-03-04T07:46:35.485 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:46:35.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:46:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 -- 172.21.15.116:0/2878640845 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60105c50 msgr2=0x7f8e60108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 --2- 172.21.15.116:0/2878640845 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60105c50 0x7f8e60108040 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009ca0 tx=0x7f8e5802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 -- 172.21.15.116:0/2878640845 shutdown_connections 2024-03-04T07:46:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 --2- 172.21.15.116:0/2878640845 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60105c50 0x7f8e60108040 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 --2- 172.21.15.116:0/2878640845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60100f50 0x7f8e601013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 --2- 172.21.15.116:0/2878640845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60067b30 0x7f8e60100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 -- 172.21.15.116:0/2878640845 >> 172.21.15.116:0/2878640845 conn(0x7f8e600fadd0 msgr2=0x7f8e600fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 -- 172.21.15.116:0/2878640845 shutdown_connections 2024-03-04T07:46:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.740+0000 7f8e66a00640 1 -- 172.21.15.116:0/2878640845 wait complete. 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.741+0000 7f8e66a00640 1 Processor -- start 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.741+0000 7f8e66a00640 1 -- start start 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e66a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60067b30 0x7f8e601978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e66a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60100f50 0x7f8e60197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e66a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60105c50 0x7f8e6019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e66a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6010c1a0 con 0x7f8e60067b30 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e66a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6010c020 con 0x7f8e60100f50 2024-03-04T07:46:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e66a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6010c320 con 0x7f8e60105c50 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e5fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60067b30 0x7f8e601978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e57400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60100f50 0x7f8e60197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e5fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60067b30 0x7f8e601978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48264/0 (socket says 172.21.15.116:48264) 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e5fe00640 1 -- 172.21.15.116:0/3990774442 learned_addr learned my addr 172.21.15.116:0/3990774442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e57400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60100f50 0x7f8e60197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44732/0 (socket says 172.21.15.116:44732) 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e64a00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60105c50 0x7f8e6019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e57400640 1 -- 172.21.15.116:0/3990774442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60105c50 msgr2=0x7f8e6019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e57400640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60105c50 0x7f8e6019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e57400640 1 -- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60067b30 msgr2=0x7f8e601978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e57400640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60067b30 0x7f8e601978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.742+0000 7f8e57400640 1 -- 172.21.15.116:0/3990774442 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6019f4c0 con 0x7f8e60100f50 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.743+0000 7f8e57400640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60100f50 0x7f8e60197de0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e9c0 tx=0x7f8e4800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.743+0000 7f8e5d600640 1 -- 172.21.15.116:0/3990774442 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800ccc0 con 0x7f8e60100f50 2024-03-04T07:46:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.743+0000 7f8e5d600640 1 -- 172.21.15.116:0/3990774442 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e480035f0 con 0x7f8e60100f50 2024-03-04T07:46:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.743+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6019f7b0 con 0x7f8e60100f50 2024-03-04T07:46:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.743+0000 7f8e5d600640 1 -- 172.21.15.116:0/3990774442 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800ce60 con 0x7f8e60100f50 2024-03-04T07:46:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.743+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6019fd00 con 0x7f8e60100f50 2024-03-04T07:46:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.744+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e60105350 con 0x7f8e60100f50 2024-03-04T07:46:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.746+0000 7f8e5d600640 1 -- 172.21.15.116:0/3990774442 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8e480037a0 con 0x7f8e60100f50 2024-03-04T07:46:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.747+0000 7f8e5d600640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1d80 0x7f8e440a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.747+0000 7f8e5d600640 1 -- 172.21.15.116:0/3990774442 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8e480ce150 con 0x7f8e60100f50 2024-03-04T07:46:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.747+0000 7f8e5fe00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1d80 0x7f8e440a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.748+0000 7f8e5fe00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1d80 0x7f8e440a4240 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005fd0 tx=0x7f8e5000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:35.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.756+0000 7f8e5d600640 1 -- 172.21.15.116:0/3990774442 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8e4809b540 con 0x7f8e60100f50 2024-03-04T07:46:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.993+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6002cff0 con 0x7f8e60100f50 2024-03-04T07:46:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.995+0000 7f8e5d600640 1 -- 172.21.15.116:0/3990774442 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f8e4809ae40 con 0x7f8e60100f50 2024-03-04T07:46:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:46:35.996 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:46:35.996 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:46:35.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.997+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1d80 msgr2=0x7f8e440a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.997+0000 7f8e66a00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1d80 0x7f8e440a4240 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8e50005fd0 tx=0x7f8e5000b410 comp rx=0 tx=0).stop 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.997+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60100f50 msgr2=0x7f8e60197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.997+0000 7f8e66a00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60100f50 0x7f8e60197de0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e9c0 tx=0x7f8e4800ee80 comp rx=0 tx=0).stop 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 shutdown_connections 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1d80 0x7f8e440a4240 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60105c50 0x7f8e6019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60100f50 0x7f8e60197de0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 --2- 172.21.15.116:0/3990774442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60067b30 0x7f8e601978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 >> 172.21.15.116:0/3990774442 conn(0x7f8e600fadd0 msgr2=0x7f8e60108ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 shutdown_connections 2024-03-04T07:46:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:35.998+0000 7f8e66a00640 1 -- 172.21.15.116:0/3990774442 wait complete. 2024-03-04T07:46:36.017 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:46:36.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:46:36.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 -- 172.21.15.116:0/1691223205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 msgr2=0x7f5a60103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 --2- 172.21.15.116:0/1691223205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a60103af0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009900 tx=0x7f5a5002bc40 comp rx=0 tx=0).stop 2024-03-04T07:46:36.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 -- 172.21.15.116:0/1691223205 shutdown_connections 2024-03-04T07:46:36.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 --2- 172.21.15.116:0/1691223205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a60105280 0x7f5a6010bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 --2- 172.21.15.116:0/1691223205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a601048c0 0x7f5a60104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 --2- 172.21.15.116:0/1691223205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a60103af0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 -- 172.21.15.116:0/1691223205 >> 172.21.15.116:0/1691223205 conn(0x7f5a600fec40 msgr2=0x7f5a60101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 -- 172.21.15.116:0/1691223205 shutdown_connections 2024-03-04T07:46:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.230+0000 7f5a66400640 1 -- 172.21.15.116:0/1691223205 wait complete. 2024-03-04T07:46:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.231+0000 7f5a66400640 1 Processor -- start 2024-03-04T07:46:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.231+0000 7f5a66400640 1 -- start start 2024-03-04T07:46:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a66400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a601999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a66400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a601048c0 0x7f5a60199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a66400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a60105280 0x7f5a601a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a66400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6010f3a0 con 0x7f5a601048c0 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a66400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6010f220 con 0x7f5a601036d0 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a66400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6010f520 con 0x7f5a60105280 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a601999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a57e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a601048c0 0x7f5a60199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a601999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36518/0 (socket says 172.21.15.116:36518) 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65000640 1 -- 172.21.15.116:0/2542856426 learned_addr learned my addr 172.21.15.116:0/2542856426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65a00640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a60105280 0x7f5a601a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65000640 1 -- 172.21.15.116:0/2542856426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a60105280 msgr2=0x7f5a601a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65000640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a60105280 0x7f5a601a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65000640 1 -- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a601048c0 msgr2=0x7f5a60199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.232+0000 7f5a65000640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a601048c0 0x7f5a60199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a65000640 1 -- 172.21.15.116:0/2542856426 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a601a1600 con 0x7f5a601036d0 2024-03-04T07:46:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a57e00640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a601048c0 0x7f5a60199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a65000640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a601999e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5a5002efd0 tx=0x7f5a50005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a5e000640 1 -- 172.21.15.116:0/2542856426 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a50041070 con 0x7f5a601036d0 2024-03-04T07:46:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a5e000640 1 -- 172.21.15.116:0/2542856426 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a500044d0 con 0x7f5a601036d0 2024-03-04T07:46:36.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a601a1890 con 0x7f5a601036d0 2024-03-04T07:46:36.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a5e000640 1 -- 172.21.15.116:0/2542856426 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a50034440 con 0x7f5a601036d0 2024-03-04T07:46:36.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.233+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a601a1da0 con 0x7f5a601036d0 2024-03-04T07:46:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.236+0000 7f5a5e000640 1 -- 172.21.15.116:0/2542856426 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5a50044070 con 0x7f5a601036d0 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.236+0000 7f5a6a7c8740 1 Processor -- start 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.236+0000 7f5a6a7c8740 1 -- start start 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff94683290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94688360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff946889d0 0x55ff9468ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff945e7d60 con 0x55ff94683860 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff94590cf0 con 0x55ff94682e70 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff945e3e00 con 0x55ff946889d0 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff94683290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff94683290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36528/0 (socket says 172.21.15.116:36528) 2024-03-04T07:46:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 -- 172.21.15.116:0/296465967 learned_addr learned my addr 172.21.15.116:0/296465967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65000640 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94688360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a57e00640 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff946889d0 0x55ff9468ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 -- 172.21.15.116:0/296465967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff946889d0 msgr2=0x55ff9468ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff946889d0 0x55ff9468ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 -- 172.21.15.116:0/296465967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 msgr2=0x55ff94688360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94688360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.237+0000 7f5a65a00640 1 -- 172.21.15.116:0/296465967 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff942fc3f0 con 0x55ff94682e70 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.238+0000 7f5a65a00640 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff94683290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5a58010370 tx=0x7f5a58010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50835598fac56269 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.238+0000 7f5a5e000640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a440a1e00 0x7f5a440a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.238+0000 7f5a56000640 1 -- 172.21.15.116:0/296465967 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58016070 con 0x55ff94682e70 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.238+0000 7f5a56000640 1 -- 172.21.15.116:0/296465967 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a58004d20 con 0x55ff94682e70 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.238+0000 7f5a56000640 1 -- 172.21.15.116:0/296465967 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58005480 con 0x55ff94682e70 2024-03-04T07:46:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.238+0000 7f5a5e000640 1 -- 172.21.15.116:0/2542856426 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5a500ef900 con 0x7f5a601036d0 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.238+0000 7f5a65000640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a440a1e00 0x7f5a440a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/296465967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 msgr2=0x55ff94683290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff94683290 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5a58010370 tx=0x7f5a58010830 comp rx=0 tx=0).stop 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a65000640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a440a1e00 0x7f5a440a42c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5a500bc3a0 tx=0x7f5a50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/296465967 shutdown_connections 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff946889d0 0x55ff9468ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94688360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/296465967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff94683290 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/296465967 >> 172.21.15.116:0/296465967 conn(0x55ff9467e3e0 msgr2=0x55ff9467e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/296465967 shutdown_connections 2024-03-04T07:46:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/296465967 wait complete. 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.239+0000 7f5a6a7c8740 1 -- start start 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 1 Processor -- start 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff9468d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff9468d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff946889d0 0x55ff94694920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9468b980 con 0x55ff94683860 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff94586220 con 0x55ff946889d0 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff9468bb00 con 0x55ff94682e70 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a6a7c8740 10 client.0 _ll_register_callbacks cb 0x55ff942fbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a57e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff9468d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a65a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff9468d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a65a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff9468d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:57872/0 (socket says 172.21.15.116:57872) 2024-03-04T07:46:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a65a00640 1 -- 172.21.15.116:0/2535785752 learned_addr learned my addr 172.21.15.116:0/2535785752 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.240+0000 7f5a65a00640 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff946889d0 0x55ff94694920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a57e00640 1 -- 172.21.15.116:0/2535785752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 msgr2=0x55ff9468d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a57e00640 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff9468d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a57e00640 1 -- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff946889d0 msgr2=0x55ff94694920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a57e00640 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff946889d0 0x55ff94694920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a57e00640 1 -- 172.21.15.116:0/2535785752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff943c67d0 con 0x55ff94683860 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a65a00640 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff9468d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a57e00640 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff9468d8e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5a6019aef0 tx=0x7f5a48012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d64d93d9a53f2ea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a4ea00640 1 -- 172.21.15.116:0/2535785752 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48004050 con 0x55ff94683860 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.241+0000 7f5a4ea00640 1 -- 172.21.15.116:0/2535785752 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5a480041f0 con 0x55ff94683860 2024-03-04T07:46:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a58004d20 con 0x55ff94683860 2024-03-04T07:46:36.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a4ea00640 1 -- 172.21.15.116:0/2535785752 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a480043f0 con 0x55ff94683860 2024-03-04T07:46:36.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 10 client.16611 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:36.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ff94697d60 con 0x55ff94683860 2024-03-04T07:46:36.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a4ea00640 1 -- 172.21.15.116:0/2535785752 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a58004d20 con 0x55ff94683860 2024-03-04T07:46:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a4ea00640 1 -- 172.21.15.116:0/2535785752 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5a48003ca0 con 0x55ff94683860 2024-03-04T07:46:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 10 client.16611 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 10 client.16611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 20 client.16611 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff94699460 0x55ff9469b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 4 client.16611 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.242+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff945f52a0 con 0x55ff94699460 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.243+0000 7f5a65a00640 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff94699460 0x55ff9469b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.243+0000 7f5a65a00640 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff94699460 0x55ff9469b880 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e0e04bb3c7b6c4f server_cookie=35bbab1110c9e054 in_seq=0 out_seq=0 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.243+0000 7f5a4ea00640 10 client.16611 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a4ea00640 1 -- 172.21.15.116:0/2535785752 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ff945f52a0 con 0x55ff94699460 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a4ea00640 10 client.16611 handle_command_reply: tid=0 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 client.16611 shutdown 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 msgr2=0x55ff9468d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff9468d8e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5a6019aef0 tx=0x7f5a48012e70 comp rx=0 tx=0).stop 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 shutdown_connections 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff94699460 0x55ff9469b880 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff946889d0 0x55ff94694920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff9468d8e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a4ea00640 0 client.16611 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/2535785752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff9468d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 >> 172.21.15.116:0/2535785752 conn(0x55ff9467e3e0 msgr2=0x55ff9467d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 shutdown_connections 2024-03-04T07:46:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/2535785752 wait complete. 2024-03-04T07:46:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.244+0000 7f5a6a7c8740 20 client.16611 trim_cache size 0 max 16384 2024-03-04T07:46:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 Processor -- start 2024-03-04T07:46:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 -- start start 2024-03-04T07:46:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff9467e3e0 0x55ff9468bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff946948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94681d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601a8ac0 con 0x55ff94683860 2024-03-04T07:46:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601ab0a0 con 0x55ff9467e3e0 2024-03-04T07:46:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601a90c0 con 0x55ff94682e70 2024-03-04T07:46:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a65a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff9467e3e0 0x55ff9468bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a65a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff9467e3e0 0x55ff9468bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36544/0 (socket says 172.21.15.116:36544) 2024-03-04T07:46:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a65a00640 1 -- 172.21.15.116:0/1339820825 learned_addr learned my addr 172.21.15.116:0/1339820825 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.258+0000 7f5a57e00640 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff946948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a65a00640 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94681d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a65a00640 1 -- 172.21.15.116:0/1339820825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 msgr2=0x55ff946948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a65a00640 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff946948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a65a00640 1 -- 172.21.15.116:0/1339820825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 msgr2=0x55ff94681d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a65a00640 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94681d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a65a00640 1 -- 172.21.15.116:0/1339820825 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff9469bf00 con 0x55ff9467e3e0 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a65a00640 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff9467e3e0 0x55ff9468bed0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5a580098b0 tx=0x7f5a58010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e9b766dc68ce39f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a55600640 1 -- 172.21.15.116:0/1339820825 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58016040 con 0x55ff9467e3e0 2024-03-04T07:46:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a55600640 1 -- 172.21.15.116:0/1339820825 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a5802ed20 con 0x55ff9467e3e0 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.259+0000 7f5a55600640 1 -- 172.21.15.116:0/1339820825 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a580054a0 con 0x55ff9467e3e0 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/1339820825 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff9467e3e0 msgr2=0x55ff9468bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff9467e3e0 0x55ff9468bed0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5a580098b0 tx=0x7f5a58010830 comp rx=0 tx=0).stop 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/1339820825 shutdown_connections 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff94683860 0x55ff94681d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94682e70 0x55ff946948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/1339820825 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff9467e3e0 0x55ff9468bed0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/1339820825 >> 172.21.15.116:0/1339820825 conn(0x55ff9468c5b0 msgr2=0x55ff94681640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/1339820825 shutdown_connections 2024-03-04T07:46:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/1339820825 wait complete. 2024-03-04T07:46:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.260+0000 7f5a6a7c8740 1 -- start start 2024-03-04T07:46:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 1 Processor -- start 2024-03-04T07:46:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff9467e3e0 0x55ff94694470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff946949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94683860 0x55ff94692b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601a9720 con 0x55ff9467e3e0 2024-03-04T07:46:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601aaa40 con 0x55ff94682e70 2024-03-04T07:46:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601a9d80 con 0x55ff94683860 2024-03-04T07:46:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a6a7c8740 10 client.0 _ll_register_callbacks cb 0x7f5a48009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a57e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff946949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a65a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff9467e3e0 0x55ff94694470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a57e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff946949b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36556/0 (socket says 172.21.15.116:36556) 2024-03-04T07:46:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.261+0000 7f5a57e00640 1 -- 172.21.15.116:0/3535895369 learned_addr learned my addr 172.21.15.116:0/3535895369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a65a00640 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94683860 0x55ff94692b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a57e00640 1 -- 172.21.15.116:0/3535895369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94683860 msgr2=0x55ff94692b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a57e00640 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94683860 0x55ff94692b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a57e00640 1 -- 172.21.15.116:0/3535895369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff9467e3e0 msgr2=0x55ff94694470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a57e00640 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff9467e3e0 0x55ff94694470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a57e00640 1 -- 172.21.15.116:0/3535895369 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff94693380 con 0x55ff94682e70 2024-03-04T07:46:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a65a00640 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94683860 0x55ff94692b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:46:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a57e00640 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff946949b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5a480104b0 tx=0x7f5a48005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865d4703124f653c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a4ea00640 1 -- 172.21.15.116:0/3535895369 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a48036060 con 0x55ff94682e70 2024-03-04T07:46:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a5802ed20 con 0x55ff94682e70 2024-03-04T07:46:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.262+0000 7f5a4ea00640 1 -- 172.21.15.116:0/3535895369 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5a480106a0 con 0x55ff94682e70 2024-03-04T07:46:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a4ea00640 1 -- 172.21.15.116:0/3535895369 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a480108a0 con 0x55ff94682e70 2024-03-04T07:46:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 10 client.26744 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ff946f59f0 con 0x55ff94682e70 2024-03-04T07:46:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a4ea00640 1 -- 172.21.15.116:0/3535895369 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a5802ed20 con 0x55ff94682e70 2024-03-04T07:46:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a4ea00640 1 -- 172.21.15.116:0/3535895369 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5a4803b070 con 0x55ff94682e70 2024-03-04T07:46:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 10 client.26744 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 10 client.26744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:36.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 20 client.26744 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff946889d0 0x55ff946f8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 4 client.26744 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ff945ef5f0 con 0x55ff946889d0 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.263+0000 7f5a65a00640 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff946889d0 0x55ff946f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a65a00640 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff946889d0 0x55ff946f8360 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc33d4972cc86ca8 server_cookie=157bf56d246c1032 in_seq=0 out_seq=0 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a4ea00640 10 client.26744 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a4ea00640 1 -- 172.21.15.116:0/3535895369 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ff945ef5f0 con 0x55ff946889d0 2024-03-04T07:46:36.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a4ea00640 10 client.26744 handle_command_reply: tid=0 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 client.26744 shutdown 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 msgr2=0x55ff946949b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff946949b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5a480104b0 tx=0x7f5a48005270 comp rx=0 tx=0).stop 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 shutdown_connections 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ff946889d0 0x55ff946f8360 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ff94683860 0x55ff94692b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ff94682e70 0x55ff946949b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a4ea00640 0 client.26744 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 --2- 172.21.15.116:0/3535895369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff9467e3e0 0x55ff94694470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 >> 172.21.15.116:0/3535895369 conn(0x55ff9468c5b0 msgr2=0x55ff94681ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 shutdown_connections 2024-03-04T07:46:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 1 -- 172.21.15.116:0/3535895369 wait complete. 2024-03-04T07:46:36.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.264+0000 7f5a6a7c8740 20 client.26744 trim_cache size 0 max 16384 2024-03-04T07:46:36.280 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:46:36.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a440a1e00 msgr2=0x7f5a440a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a440a1e00 0x7f5a440a42c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5a500bc3a0 tx=0x7f5a50009260 comp rx=0 tx=0).stop 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 msgr2=0x7f5a601999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a601999e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5a5002efd0 tx=0x7f5a50005d50 comp rx=0 tx=0).stop 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 shutdown_connections 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5a440a1e00 0x7f5a440a42c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a60105280 0x7f5a601a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a601048c0 0x7f5a60199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 --2- 172.21.15.116:0/2542856426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5a601036d0 0x7f5a601999e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 >> 172.21.15.116:0/2542856426 conn(0x7f5a600fec40 msgr2=0x7f5a60107b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 shutdown_connections 2024-03-04T07:46:36.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:36.265+0000 7f5a66400640 1 -- 172.21.15.116:0/2542856426 wait complete. 2024-03-04T07:46:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:36 smithi133 ceph-mon[36505]: pgmap v576: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:46:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:36 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3990774442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:37.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:36 smithi116 ceph-mon[32150]: pgmap v576: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:46:37.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:36 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3990774442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:36 smithi152 ceph-mon[37154]: pgmap v576: 97 pgs: 97 active+clean; 4.2 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:46:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:36 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3990774442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:37.285 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:46:37.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:46:37.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:46:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.509+0000 7fe3a8a00640 1 -- 172.21.15.116:0/2224523454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a4103190 msgr2=0x7fe3a4109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.509+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/2224523454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a4103190 0x7fe3a4109a20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009ca0 tx=0x7fe39c02c0f0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.509+0000 7fe3a8a00640 1 -- 172.21.15.116:0/2224523454 shutdown_connections 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.509+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/2224523454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a4103190 0x7fe3a4109a20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.509+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/2224523454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a41027d0 0x7fe3a4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.509+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/2224523454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3a41015e0 0x7fe3a4101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.509+0000 7fe3a8a00640 1 -- 172.21.15.116:0/2224523454 >> 172.21.15.116:0/2224523454 conn(0x7fe3a40fcb70 msgr2=0x7fe3a40fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.510+0000 7fe3a8a00640 1 -- 172.21.15.116:0/2224523454 shutdown_connections 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.510+0000 7fe3a8a00640 1 -- 172.21.15.116:0/2224523454 wait complete. 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.510+0000 7fe3a8a00640 1 Processor -- start 2024-03-04T07:46:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a8a00640 1 -- start start 2024-03-04T07:46:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a41015e0 0x7fe3a41978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a41027d0 0x7fe3a4197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a8a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3a4103190 0x7fe3a419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a410d260 con 0x7fe3a41015e0 2024-03-04T07:46:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a8a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a410d0e0 con 0x7fe3a41027d0 2024-03-04T07:46:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a8a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a410d3e0 con 0x7fe3a4103190 2024-03-04T07:46:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a41027d0 0x7fe3a4197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a41015e0 0x7fe3a41978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a41027d0 0x7fe3a4197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36568/0 (socket says 172.21.15.116:36568) 2024-03-04T07:46:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a1400640 1 -- 172.21.15.116:0/1313408881 learned_addr learned my addr 172.21.15.116:0/1313408881 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.511+0000 7fe3a2800640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3a4103190 0x7fe3a419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a1400640 1 -- 172.21.15.116:0/1313408881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3a4103190 msgr2=0x7fe3a419ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a1400640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3a4103190 0x7fe3a419ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a1400640 1 -- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a41015e0 msgr2=0x7fe3a41978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a1400640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a41015e0 0x7fe3a41978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a1400640 1 -- 172.21.15.116:0/1313408881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a419f4c0 con 0x7fe3a41027d0 2024-03-04T07:46:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a41015e0 0x7fe3a41978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a1400640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a41027d0 0x7fe3a4197de0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e9f0 tx=0x7fe39000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe39aa00640 1 -- 172.21.15.116:0/1313408881 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39000cd80 con 0x7fe3a41027d0 2024-03-04T07:46:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe39aa00640 1 -- 172.21.15.116:0/1313408881 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe390010910 con 0x7fe3a41027d0 2024-03-04T07:46:37.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe39aa00640 1 -- 172.21.15.116:0/1313408881 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390018990 con 0x7fe3a41027d0 2024-03-04T07:46:37.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a419f7b0 con 0x7fe3a41027d0 2024-03-04T07:46:37.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.512+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a419fcb0 con 0x7fe3a41027d0 2024-03-04T07:46:37.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.515+0000 7fe39aa00640 1 -- 172.21.15.116:0/1313408881 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe390010ac0 con 0x7fe3a41027d0 2024-03-04T07:46:37.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 Processor -- start 2024-03-04T07:46:37.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 -- start start 2024-03-04T07:46:37.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480acb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480ad09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480ad34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574809dae20 con 0x557480acb4e0 2024-03-04T07:46:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557480a2ca20 con 0x557480ad1040 2024-03-04T07:46:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574809d5b00 con 0x557480acbed0 2024-03-04T07:46:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480ad34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3a2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480acb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480ad34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36582/0 (socket says 172.21.15.116:36582) 2024-03-04T07:46:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3a1400640 1 -- 172.21.15.116:0/3721013915 learned_addr learned my addr 172.21.15.116:0/3721013915 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.516+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480ad09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe3a1400640 1 -- 172.21.15.116:0/3721013915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 msgr2=0x557480ad09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe3a1400640 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480ad09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe3a1400640 1 -- 172.21.15.116:0/3721013915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 msgr2=0x557480acb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe3a1400640 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480acb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe3a1400640 1 -- 172.21.15.116:0/3721013915 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574809eae10 con 0x557480ad1040 2024-03-04T07:46:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe39aa00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3880a1e00 0x7fe3880a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe3a1400640 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480ad34d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe3900d6210 tx=0x7fe3900d66d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77daa79008f4e2cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe39aa00640 1 -- 172.21.15.116:0/1313408881 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe3900ce050 con 0x7fe3a41027d0 2024-03-04T07:46:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe3a2800640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3880a1e00 0x7fe3880a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe38f000640 1 -- 172.21.15.116:0/3721013915 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900970d0 con 0x557480ad1040 2024-03-04T07:46:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.517+0000 7fe38f000640 1 -- 172.21.15.116:0/3721013915 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe39009ae30 con 0x557480ad1040 2024-03-04T07:46:37.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe38f000640 1 -- 172.21.15.116:0/3721013915 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900d1bc0 con 0x557480ad1040 2024-03-04T07:46:37.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/3721013915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 msgr2=0x557480ad34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3a2800640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3880a1e00 0x7fe3880a42c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe39c02c630 tx=0x7fe39c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480ad34d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe3900d6210 tx=0x7fe3900d66d0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/3721013915 shutdown_connections 2024-03-04T07:46:37.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480ad34d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480ad09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/3721013915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480acb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.518+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/3721013915 >> 172.21.15.116:0/3721013915 conn(0x557480ac6a50 msgr2=0x557480ac6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.519+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/3721013915 shutdown_connections 2024-03-04T07:46:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.519+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/3721013915 wait complete. 2024-03-04T07:46:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.519+0000 7fe3ab7fa740 1 -- start start 2024-03-04T07:46:37.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.519+0000 7fe3ab7fa740 1 Processor -- start 2024-03-04T07:46:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574809e8f40 con 0x557480acb4e0 2024-03-04T07:46:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557480a28ac0 con 0x557480acbed0 2024-03-04T07:46:37.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557480ad3ff0 con 0x557480ad1040 2024-03-04T07:46:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3ab7fa740 10 client.0 _ll_register_callbacks cb 0x557480a27aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36586/0 (socket says 172.21.15.116:36586) 2024-03-04T07:46:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3a1400640 1 -- 172.21.15.116:0/2342763796 learned_addr learned my addr 172.21.15.116:0/2342763796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.520+0000 7fe3a2800640 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3a1400640 1 -- 172.21.15.116:0/2342763796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 msgr2=0x557480ac2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3a1400640 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3a1400640 1 -- 172.21.15.116:0/2342763796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 msgr2=0x557480ac58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3a1400640 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3a1400640 1 -- 172.21.15.116:0/2342763796 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574808097e0 con 0x557480acbed0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3a1400640 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac5de0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe39009f090 tx=0x7fe39000cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e332eb30bb1588b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe383400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe383400640 1 -- 172.21.15.116:0/2342763796 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39009a790 con 0x557480acbed0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe383400640 1 -- 172.21.15.116:0/2342763796 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe39009b090 con 0x557480acbed0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe39009ae30 con 0x557480acbed0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe383400640 1 -- 172.21.15.116:0/2342763796 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe39009a970 con 0x557480acbed0 2024-03-04T07:46:37.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.521+0000 7fe3ab7fa740 10 client.26756 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557480869f70 con 0x557480acbed0 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe383400640 1 -- 172.21.15.116:0/2342763796 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe39009ae30 con 0x557480acbed0 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe383400640 1 -- 172.21.15.116:0/2342763796 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3900d00d0 con 0x557480acbed0 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3ab7fa740 10 client.26756 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3ab7fa740 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3ab7fa740 20 client.26756 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480ae5720 0x557480ae7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3ab7fa740 4 client.26756 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557480a39f60 con 0x557480ae5720 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.522+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480ae5720 0x557480ae7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.523+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480ae5720 0x557480ae7b10 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=416f827f7a03ff6d server_cookie=507ae81d284f4888 in_seq=0 out_seq=0 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.523+0000 7fe383400640 10 client.26756 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.523+0000 7fe383400640 1 -- 172.21.15.116:0/2342763796 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557480a39f60 con 0x557480ae5720 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.523+0000 7fe383400640 10 client.26756 handle_command_reply: tid=0 2024-03-04T07:46:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.523+0000 7fe3ab7fa740 1 client.26756 shutdown 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 msgr2=0x557480ac5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac5de0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe39009f090 tx=0x7fe39000cf20 comp rx=0 tx=0).stop 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 shutdown_connections 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480ae5720 0x557480ae7b10 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac5de0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe383400640 0 client.26756 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/2342763796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 >> 172.21.15.116:0/2342763796 conn(0x557480ac6a50 msgr2=0x557480ac7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 shutdown_connections 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/2342763796 wait complete. 2024-03-04T07:46:37.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.524+0000 7fe3ab7fa740 20 client.26756 trim_cache size 0 max 16384 2024-03-04T07:46:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.541+0000 7fe3ab7fa740 1 Processor -- start 2024-03-04T07:46:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3ab7fa740 1 -- start start 2024-03-04T07:46:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a41a48b0 con 0x557480acb4e0 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a41a6e90 con 0x557480acbed0 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a41a4eb0 con 0x557480ad1040 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36588/0 (socket says 172.21.15.116:36588) 2024-03-04T07:46:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3a1400640 1 -- 172.21.15.116:0/842304188 learned_addr learned my addr 172.21.15.116:0/842304188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3a2800640 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3a1400640 1 -- 172.21.15.116:0/842304188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 msgr2=0x557480ac1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.542+0000 7fe3a1400640 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe3a1400640 1 -- 172.21.15.116:0/842304188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 msgr2=0x557480ac11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe3a1400640 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe3a1400640 1 -- 172.21.15.116:0/842304188 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557480701170 con 0x557480acbed0 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe3a2800640 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe3a1400640 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac1710 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe3900990e0 tx=0x7fe390097770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9170c574bfd22d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe38e600640 1 -- 172.21.15.116:0/842304188 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39009ac00 con 0x557480acbed0 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe38e600640 1 -- 172.21.15.116:0/842304188 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe39009eed0 con 0x557480acbed0 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.543+0000 7fe38e600640 1 -- 172.21.15.116:0/842304188 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900a3090 con 0x557480acbed0 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/842304188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 msgr2=0x557480ac1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac1710 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe3900990e0 tx=0x7fe390097770 comp rx=0 tx=0).stop 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/842304188 shutdown_connections 2024-03-04T07:46:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480ad1040 0x557480ac1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480acbed0 0x557480ac1710 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/842304188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480ac11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/842304188 >> 172.21.15.116:0/842304188 conn(0x557480ac6a50 msgr2=0x557480ac6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/842304188 shutdown_connections 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/842304188 wait complete. 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 -- start start 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.544+0000 7fe3ab7fa740 1 Processor -- start 2024-03-04T07:46:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480af62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480af6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3ab7fa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480af6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a41a5510 con 0x557480acb4e0 2024-03-04T07:46:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a41a6830 con 0x557480ad1040 2024-03-04T07:46:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3ab7fa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a41a5b70 con 0x557480acbed0 2024-03-04T07:46:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3ab7fa740 10 client.0 _ll_register_callbacks cb 0x7fe390005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3a1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480af62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480af6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3a2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480af6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3a1400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480af6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:57934/0 (socket says 172.21.15.116:57934) 2024-03-04T07:46:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3a1400640 1 -- 172.21.15.116:0/200037835 learned_addr learned my addr 172.21.15.116:0/200037835 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.545+0000 7fe3a2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480af6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36594/0 (socket says 172.21.15.116:36594) 2024-03-04T07:46:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3a1e00640 1 -- 172.21.15.116:0/200037835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 msgr2=0x557480af6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480af6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3a1e00640 1 -- 172.21.15.116:0/200037835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 msgr2=0x557480af6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480af6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3a1e00640 1 -- 172.21.15.116:0/200037835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557480b402f0 con 0x557480acb4e0 2024-03-04T07:46:37.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3a1400640 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480af6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3a1e00640 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480af62d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x557480ae8220 tx=0x7fe394004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0b14498697413a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe383400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe383400640 1 -- 172.21.15.116:0/200037835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39401f700 con 0x557480acb4e0 2024-03-04T07:46:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe383400640 1 -- 172.21.15.116:0/200037835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe39401f8a0 con 0x557480acb4e0 2024-03-04T07:46:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.546+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557480a342b0 con 0x557480acb4e0 2024-03-04T07:46:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe383400640 1 -- 172.21.15.116:0/200037835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe39401faa0 con 0x557480acb4e0 2024-03-04T07:46:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 10 client.16635 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557480b416a0 con 0x557480acb4e0 2024-03-04T07:46:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe383400640 1 -- 172.21.15.116:0/200037835 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557480a342b0 con 0x557480acb4e0 2024-03-04T07:46:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe383400640 1 -- 172.21.15.116:0/200037835 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe394010430 con 0x557480acb4e0 2024-03-04T07:46:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 10 client.16635 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 20 client.16635 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480b41f80 0x557480b443a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 4 client.16635 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5574809b2a40 con 0x557480b41f80 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.547+0000 7fe3a1400640 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480b41f80 0x557480b443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.548+0000 7fe3a1400640 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480b41f80 0x557480b443a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f3386a002c97df7 server_cookie=971c40bfa34f1c66 in_seq=0 out_seq=0 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.548+0000 7fe383400640 10 client.16635 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe383400640 1 -- 172.21.15.116:0/200037835 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5574809b2a40 con 0x557480b41f80 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe383400640 10 client.16635 handle_command_reply: tid=0 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 client.16635 shutdown 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 msgr2=0x557480af62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480af62d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x557480ae8220 tx=0x7fe394004410 comp rx=0 tx=0).stop 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 shutdown_connections 2024-03-04T07:46:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557480b41f80 0x557480b443a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557480ad1040 0x557480af6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557480acbed0 0x557480af6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 --2- 172.21.15.116:0/200037835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557480acb4e0 0x557480af62d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 >> 172.21.15.116:0/200037835 conn(0x557480ac6a50 msgr2=0x557480ac11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe383400640 0 client.16635 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 shutdown_connections 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 1 -- 172.21.15.116:0/200037835 wait complete. 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.549+0000 7fe3ab7fa740 20 client.16635 trim_cache size 0 max 16384 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:37.563 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:46:37.564 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "3f615617-ba92-479b-b6ed-1c46adccee35", 2024-03-04T07:46:37.564 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:46:37.564 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3880a1e00 msgr2=0x7fe3880a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3880a1e00 0x7fe3880a42c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe39c02c630 tx=0x7fe39c032590 comp rx=0 tx=0).stop 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a41027d0 msgr2=0x7fe3a4197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a41027d0 0x7fe3a4197de0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e9f0 tx=0x7fe39000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 shutdown_connections 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3880a1e00 0x7fe3880a42c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3a4103190 0x7fe3a419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3a41027d0 0x7fe3a4197de0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 --2- 172.21.15.116:0/1313408881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a41015e0 0x7fe3a41978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 >> 172.21.15.116:0/1313408881 conn(0x7fe3a40fcb70 msgr2=0x7fe3a4105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 shutdown_connections 2024-03-04T07:46:37.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.550+0000 7fe3a8a00640 1 -- 172.21.15.116:0/1313408881 wait complete. 2024-03-04T07:46:37.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:46:37.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:46:37.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.768+0000 7ff6b6600640 1 -- 172.21.15.116:0/1048227165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 msgr2=0x7ff6b0108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.768+0000 7ff6b6600640 1 --2- 172.21.15.116:0/1048227165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b0108010 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009900 tx=0x7ff6a002bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.768+0000 7ff6b6600640 1 -- 172.21.15.116:0/1048227165 shutdown_connections 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.768+0000 7ff6b6600640 1 --2- 172.21.15.116:0/1048227165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b01097a0 0x7ff6b0110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.768+0000 7ff6b6600640 1 --2- 172.21.15.116:0/1048227165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0108de0 0x7ff6b0109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.768+0000 7ff6b6600640 1 --2- 172.21.15.116:0/1048227165 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b0108010 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.769+0000 7ff6b6600640 1 -- 172.21.15.116:0/1048227165 >> 172.21.15.116:0/1048227165 conn(0x7ff6b0076b90 msgr2=0x7ff6b0078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.769+0000 7ff6b6600640 1 -- 172.21.15.116:0/1048227165 shutdown_connections 2024-03-04T07:46:37.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.769+0000 7ff6b6600640 1 -- 172.21.15.116:0/1048227165 wait complete. 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 Processor -- start 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 -- start start 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0108de0 0x7ff6b019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b01097a0 0x7ff6b01a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0113890 con 0x7ff6b01097a0 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0113710 con 0x7ff6b0107bf0 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.770+0000 7ff6b6600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0113a10 con 0x7ff6b0108de0 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b019df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36616/0 (socket says 172.21.15.116:36616) 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 -- 172.21.15.116:0/2024998392 learned_addr learned my addr 172.21.15.116:0/2024998392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6afe00640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b01097a0 0x7ff6b01a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6aea00640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0108de0 0x7ff6b019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 -- 172.21.15.116:0/2024998392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0108de0 msgr2=0x7ff6b019e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0108de0 0x7ff6b019e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 -- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b01097a0 msgr2=0x7ff6b01a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b01097a0 0x7ff6b01a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 -- 172.21.15.116:0/2024998392 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b01a5b30 con 0x7ff6b0107bf0 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6aea00640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0108de0 0x7ff6b019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.771+0000 7ff6af400640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b019df10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0000c00 tx=0x7ff6a002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.772+0000 7ff6a7e00640 1 -- 172.21.15.116:0/2024998392 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0043020 con 0x7ff6b0107bf0 2024-03-04T07:46:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.772+0000 7ff6a7e00640 1 -- 172.21.15.116:0/2024998392 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6a00054e0 con 0x7ff6b0107bf0 2024-03-04T07:46:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.772+0000 7ff6a7e00640 1 -- 172.21.15.116:0/2024998392 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0003c40 con 0x7ff6b0107bf0 2024-03-04T07:46:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.772+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b01a5dc0 con 0x7ff6b0107bf0 2024-03-04T07:46:37.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.772+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b01a6230 con 0x7ff6b0107bf0 2024-03-04T07:46:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.775+0000 7ff6a7e00640 1 -- 172.21.15.116:0/2024998392 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff6a0046020 con 0x7ff6b0107bf0 2024-03-04T07:46:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.775+0000 7ff6b93b3740 1 Processor -- start 2024-03-04T07:46:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.775+0000 7ff6b93b3740 1 -- start start 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.775+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfab50 0x556713dfaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.775+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e00040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e02b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556713d28090 con 0x556713dfb540 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556713d5fd30 con 0x556713dfab50 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556713d08a30 con 0x556713e006b0 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6af400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e00040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6afe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfab50 0x556713dfaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6af400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e00040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47396/0 (socket says 172.21.15.116:47396) 2024-03-04T07:46:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6af400640 1 -- 172.21.15.116:0/2560254456 learned_addr learned my addr 172.21.15.116:0/2560254456 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6aea00640 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e02b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6afe00640 1 -- 172.21.15.116:0/2560254456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 msgr2=0x556713e02b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6afe00640 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e02b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6afe00640 1 -- 172.21.15.116:0/2560254456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 msgr2=0x556713e00040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6afe00640 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e00040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6afe00640 1 -- 172.21.15.116:0/2560254456 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556713b3e7d0 con 0x556713dfab50 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6aea00640 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e02b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6a7e00640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff6940a1e00 0x7ff6940a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.776+0000 7ff6af400640 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e00040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.777+0000 7ff6a7e00640 1 -- 172.21.15.116:0/2024998392 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff6a00efdf0 con 0x7ff6b0107bf0 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.777+0000 7ff6afe00640 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfab50 0x556713dfaf70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009f00 tx=0x7ff6a8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14005591f3e2c85f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.777+0000 7ff6af400640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff6940a1e00 0x7ff6940a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.777+0000 7ff6a4c00640 1 -- 172.21.15.116:0/2560254456 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8004d20 con 0x556713dfab50 2024-03-04T07:46:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.777+0000 7ff6a4c00640 1 -- 172.21.15.116:0/2560254456 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6a8004030 con 0x556713dfab50 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.777+0000 7ff6a4c00640 1 -- 172.21.15.116:0/2560254456 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8005260 con 0x556713dfab50 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 -- 172.21.15.116:0/2560254456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfab50 msgr2=0x556713dfaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6af400640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff6940a1e00 0x7ff6940a42c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff6a002c170 tx=0x7ff6a00f3830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfab50 0x556713dfaf70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009f00 tx=0x7ff6a8011810 comp rx=0 tx=0).stop 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 -- 172.21.15.116:0/2560254456 shutdown_connections 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e02b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e00040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/2560254456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfab50 0x556713dfaf70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 -- 172.21.15.116:0/2560254456 >> 172.21.15.116:0/2560254456 conn(0x556713df60c0 msgr2=0x556713df64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 -- 172.21.15.116:0/2560254456 shutdown_connections 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 -- 172.21.15.116:0/2560254456 wait complete. 2024-03-04T07:46:37.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.778+0000 7ff6b93b3740 1 -- start start 2024-03-04T07:46:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 1 Processor -- start 2024-03-04T07:46:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfab50 0x556713e04f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e05490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713e006b0 0x556713e0c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556713cfe0c0 con 0x556713dfb540 2024-03-04T07:46:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556713d5bdd0 con 0x556713e006b0 2024-03-04T07:46:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556713e03660 con 0x556713dfab50 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6b93b3740 10 client.0 _ll_register_callbacks cb 0x556713d1b860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e05490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6afe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713e006b0 0x556713e0c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e05490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47398/0 (socket says 172.21.15.116:47398) 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6aea00640 1 -- 172.21.15.116:0/1751253550 learned_addr learned my addr 172.21.15.116:0/1751253550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.779+0000 7ff6afe00640 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfab50 0x556713e04f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff6aea00640 1 -- 172.21.15.116:0/1751253550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfab50 msgr2=0x556713e04f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff6aea00640 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfab50 0x556713e04f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff6aea00640 1 -- 172.21.15.116:0/1751253550 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713e006b0 msgr2=0x556713e0c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff6aea00640 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713e006b0 0x556713e0c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff6aea00640 1 -- 172.21.15.116:0/1751253550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556713a743f0 con 0x556713dfb540 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff6aea00640 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e05490 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff69c013790 tx=0x7ff69c013c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a97e8ff8c25cdf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff699600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff699600640 1 -- 172.21.15.116:0/1751253550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c020060 con 0x556713dfb540 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff699600640 1 -- 172.21.15.116:0/1751253550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff69c011d70 con 0x556713dfb540 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6a8004030 con 0x556713dfb540 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.780+0000 7ff699600640 1 -- 172.21.15.116:0/1751253550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff69c00f450 con 0x556713dfb540 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 10 client.16647 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556713a340f0 con 0x556713dfb540 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff699600640 1 -- 172.21.15.116:0/1751253550 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6a8004030 con 0x556713dfb540 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff699600640 1 -- 172.21.15.116:0/1751253550 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff69c025070 con 0x556713dfb540 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 10 client.16647 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 20 client.16647 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e10ab0 0x556713e12ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 4 client.16647 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556713d6d270 con 0x556713e10ab0 2024-03-04T07:46:37.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.781+0000 7ff6afe00640 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e10ab0 0x556713e12ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.782+0000 7ff6afe00640 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e10ab0 0x556713e12ed0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebf7917ae505c2bc server_cookie=7008f04d758eda11 in_seq=0 out_seq=0 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.782+0000 7ff699600640 10 client.16647 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.782+0000 7ff699600640 1 -- 172.21.15.116:0/1751253550 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556713d6d270 con 0x556713e10ab0 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.782+0000 7ff699600640 10 client.16647 handle_command_reply: tid=0 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 client.16647 shutdown 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 msgr2=0x556713e05490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e05490 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff69c013790 tx=0x7ff69c013c50 comp rx=0 tx=0).stop 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 shutdown_connections 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e10ab0 0x556713e12ed0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713e006b0 0x556713e0c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfb540 0x556713e05490 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/1751253550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfab50 0x556713e04f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff699600640 0 client.16647 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 >> 172.21.15.116:0/1751253550 conn(0x556713df60c0 msgr2=0x556713df5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 shutdown_connections 2024-03-04T07:46:37.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 1 -- 172.21.15.116:0/1751253550 wait complete. 2024-03-04T07:46:37.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.783+0000 7ff6b93b3740 20 client.16647 trim_cache size 0 max 16384 2024-03-04T07:46:37.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.800+0000 7ff6b93b3740 1 Processor -- start 2024-03-04T07:46:37.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6b93b3740 1 -- start start 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713df60c0 0x556713df9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfab50 0x556713e0c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfb540 0x556713e177b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01ad0c0 con 0x556713dfab50 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01af6a0 con 0x556713df60c0 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01ad6c0 con 0x556713dfb540 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6afe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713df60c0 0x556713df9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfab50 0x556713e0c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6afe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713df60c0 0x556713df9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36646/0 (socket says 172.21.15.116:36646) 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6afe00640 1 -- 172.21.15.116:0/150488243 learned_addr learned my addr 172.21.15.116:0/150488243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.801+0000 7ff6afe00640 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfb540 0x556713e177b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.802+0000 7ff6aea00640 1 -- 172.21.15.116:0/150488243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfb540 msgr2=0x556713e177b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.802+0000 7ff6aea00640 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfb540 0x556713e177b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.802+0000 7ff6aea00640 1 -- 172.21.15.116:0/150488243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713df60c0 msgr2=0x556713df9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.802+0000 7ff6aea00640 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713df60c0 0x556713df9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.802+0000 7ff6aea00640 1 -- 172.21.15.116:0/150488243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556713e13530 con 0x556713dfab50 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.802+0000 7ff6afe00640 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713df60c0 0x556713df9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.802+0000 7ff6aea00640 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfab50 0x556713e0c490 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff69c01b870 tx=0x7ff69c005930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63eee6b8235496a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.803+0000 7ff69be00640 1 -- 172.21.15.116:0/150488243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c02b030 con 0x556713dfab50 2024-03-04T07:46:37.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.803+0000 7ff69be00640 1 -- 172.21.15.116:0/150488243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0055f0 con 0x556713dfab50 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.803+0000 7ff69be00640 1 -- 172.21.15.116:0/150488243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c000e60 con 0x556713dfab50 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.803+0000 7ff6b93b3740 1 -- 172.21.15.116:0/150488243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfab50 msgr2=0x556713e0c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.803+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfab50 0x556713e0c490 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff69c01b870 tx=0x7ff69c005930 comp rx=0 tx=0).stop 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 -- 172.21.15.116:0/150488243 shutdown_connections 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713dfb540 0x556713e177b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713dfab50 0x556713e0c490 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/150488243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713df60c0 0x556713df9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 -- 172.21.15.116:0/150488243 >> 172.21.15.116:0/150488243 conn(0x556713e04b10 msgr2=0x556713df9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 -- 172.21.15.116:0/150488243 shutdown_connections 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 -- 172.21.15.116:0/150488243 wait complete. 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 -- start start 2024-03-04T07:46:37.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.804+0000 7ff6b93b3740 1 Processor -- start 2024-03-04T07:46:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713df60c0 0x556713e0bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfb540 0x556713e0a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6b93b3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e0ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01add20 con 0x556713df60c0 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01af040 con 0x556713dfb540 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6b93b3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01ae380 con 0x556713e006b0 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6b93b3740 10 client.0 _ll_register_callbacks cb 0x7ff69c009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713df60c0 0x556713e0bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfb540 0x556713e0a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713df60c0 0x556713e0bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47422/0 (socket says 172.21.15.116:47422) 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfb540 0x556713e0a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36654/0 (socket says 172.21.15.116:36654) 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 -- 172.21.15.116:0/3254963012 learned_addr learned my addr 172.21.15.116:0/3254963012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e0ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 -- 172.21.15.116:0/3254963012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 msgr2=0x556713e0ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e0ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 -- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfb540 msgr2=0x556713e0a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfb540 0x556713e0a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6afe00640 1 -- 172.21.15.116:0/3254963012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556713e0b060 con 0x556713df60c0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.805+0000 7ff6aea00640 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfb540 0x556713e0a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff6afe00640 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713df60c0 0x556713e0bf80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009270 tx=0x7ff6a8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751c1bbbbc52f8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff699600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff699600640 1 -- 172.21.15.116:0/3254963012 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a801e040 con 0x556713df60c0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff699600640 1 -- 172.21.15.116:0/3254963012 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff6a8004ab0 con 0x556713df60c0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff69c0055f0 con 0x556713df60c0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff699600640 1 -- 172.21.15.116:0/3254963012 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6a8004cb0 con 0x556713df60c0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff6b93b3740 10 client.16653 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556713e6d0a0 con 0x556713df60c0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.806+0000 7ff699600640 1 -- 172.21.15.116:0/3254963012 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0055f0 con 0x556713df60c0 2024-03-04T07:46:37.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff699600640 1 -- 172.21.15.116:0/3254963012 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff6a8024040 con 0x556713df60c0 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff6b93b3740 10 client.16653 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff6b93b3740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff6b93b3740 20 client.16653 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e6d910 0x556713e6fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff6b93b3740 4 client.16653 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556713d675c0 con 0x556713e6d910 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.807+0000 7ff6aea00640 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e6d910 0x556713e6fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6aea00640 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e6d910 0x556713e6fd30 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=484176d662bd6e38 server_cookie=d6e0ec1735d7b232 in_seq=0 out_seq=0 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff699600640 10 client.16653 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff699600640 1 -- 172.21.15.116:0/3254963012 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556713d675c0 con 0x556713e6d910 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff699600640 10 client.16653 handle_command_reply: tid=0 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 client.16653 shutdown 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713df60c0 msgr2=0x556713e0bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713df60c0 0x556713e0bf80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009270 tx=0x7ff6a8009500 comp rx=0 tx=0).stop 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 shutdown_connections 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556713e6d910 0x556713e6fd30 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556713e006b0 0x556713e0ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556713dfb540 0x556713e0a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 --2- 172.21.15.116:0/3254963012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556713df60c0 0x556713e0bf80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff699600640 0 client.16653 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 >> 172.21.15.116:0/3254963012 conn(0x556713e04b10 msgr2=0x556713df9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 shutdown_connections 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 1 -- 172.21.15.116:0/3254963012 wait complete. 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.808+0000 7ff6b93b3740 20 client.16653 trim_cache size 0 max 16384 2024-03-04T07:46:37.814 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:37.815 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:46:37.815 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:46:37.815 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:37.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff6940a1e00 msgr2=0x7ff6940a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff6940a1e00 0x7ff6940a42c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff6a002c170 tx=0x7ff6a00f3830 comp rx=0 tx=0).stop 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 msgr2=0x7ff6b019df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b019df10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0000c00 tx=0x7ff6a002c650 comp rx=0 tx=0).stop 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 shutdown_connections 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b01097a0 0x7ff6b01a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff6940a1e00 0x7ff6940a42c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0108de0 0x7ff6b019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 --2- 172.21.15.116:0/2024998392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff6b0107bf0 0x7ff6b019df10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 >> 172.21.15.116:0/2024998392 conn(0x7ff6b0076b90 msgr2=0x7ff6b010c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 shutdown_connections 2024-03-04T07:46:37.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:37.809+0000 7ff6b6600640 1 -- 172.21.15.116:0/2024998392 wait complete. 2024-03-04T07:46:37.829 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:46:37.829 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.027+0000 7f2e13200640 1 -- 172.21.15.116:0/2025045718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e140ffc90 msgr2=0x7f2e141020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.027+0000 7f2e13200640 1 --2- 172.21.15.116:0/2025045718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e140ffc90 0x7f2e141020c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009900 tx=0x7f2e0402bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.028+0000 7f2e13200640 1 -- 172.21.15.116:0/2025045718 shutdown_connections 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.028+0000 7f2e13200640 1 --2- 172.21.15.116:0/2025045718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e14104fb0 0x7f2e14105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.028+0000 7f2e13200640 1 --2- 172.21.15.116:0/2025045718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e14102600 0x7f2e14104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.028+0000 7f2e13200640 1 --2- 172.21.15.116:0/2025045718 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e140ffc90 0x7f2e141020c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.028+0000 7f2e13200640 1 -- 172.21.15.116:0/2025045718 >> 172.21.15.116:0/2025045718 conn(0x7f2e140fb890 msgr2=0x7f2e140fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.028+0000 7f2e13200640 1 -- 172.21.15.116:0/2025045718 shutdown_connections 2024-03-04T07:46:38.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.028+0000 7f2e13200640 1 -- 172.21.15.116:0/2025045718 wait complete. 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 Processor -- start 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 -- start start 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e140ffc90 0x7f2e1419dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e14102600 0x7f2e1419e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e14104fb0 0x7f2e141a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e141136e0 con 0x7f2e14102600 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14113560 con 0x7f2e14104fb0 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e13200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14113860 con 0x7f2e140ffc90 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e11e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e140ffc90 0x7f2e1419dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e12800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e14104fb0 0x7f2e141a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e11e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e140ffc90 0x7f2e1419dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58018/0 (socket says 172.21.15.116:58018) 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e11e00640 1 -- 172.21.15.116:0/220348408 learned_addr learned my addr 172.21.15.116:0/220348408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.029+0000 7f2e09400640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e14102600 0x7f2e1419e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e12800640 1 -- 172.21.15.116:0/220348408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e140ffc90 msgr2=0x7f2e1419dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e12800640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e140ffc90 0x7f2e1419dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e12800640 1 -- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e14102600 msgr2=0x7f2e1419e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e12800640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e14102600 0x7f2e1419e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e12800640 1 -- 172.21.15.116:0/220348408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e141a5960 con 0x7f2e14104fb0 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e09400640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e14102600 0x7f2e1419e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e12800640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e14104fb0 0x7f2e141a52f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009fd0 tx=0x7f2e0c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e0b400640 1 -- 172.21.15.116:0/220348408 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c00ce20 con 0x7f2e14104fb0 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e0b400640 1 -- 172.21.15.116:0/220348408 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e0c005020 con 0x7f2e14104fb0 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e141a5c50 con 0x7f2e14104fb0 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e0b400640 1 -- 172.21.15.116:0/220348408 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c010720 con 0x7f2e14104fb0 2024-03-04T07:46:38.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.030+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e141a61a0 con 0x7f2e14104fb0 2024-03-04T07:46:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.032+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e140fd380 con 0x7f2e14104fb0 2024-03-04T07:46:38.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.033+0000 7f2e0b400640 1 -- 172.21.15.116:0/220348408 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2e0c005540 con 0x7f2e14104fb0 2024-03-04T07:46:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.035+0000 7f2e0b400640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2df80a1d80 0x7f2df80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.035+0000 7f2e0b400640 1 -- 172.21.15.116:0/220348408 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2e0c0ce440 con 0x7f2e14104fb0 2024-03-04T07:46:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.035+0000 7f2e11e00640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2df80a1d80 0x7f2df80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.036+0000 7f2e11e00640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2df80a1d80 0x7f2df80a4240 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2e0402c8e0 tx=0x7f2e0402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.038+0000 7f2e0b400640 1 -- 172.21.15.116:0/220348408 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2e0c09b830 con 0x7f2e14104fb0 2024-03-04T07:46:38.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.276+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e1402cff0 con 0x7f2e14104fb0 2024-03-04T07:46:38.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.277+0000 7f2e0b400640 1 -- 172.21.15.116:0/220348408 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f2e0c09b130 con 0x7f2e14104fb0 2024-03-04T07:46:38.278 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:46:38.278 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:46:38.278 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:46:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2df80a1d80 msgr2=0x7f2df80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2df80a1d80 0x7f2df80a4240 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2e0402c8e0 tx=0x7f2e0402df90 comp rx=0 tx=0).stop 2024-03-04T07:46:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e14104fb0 msgr2=0x7f2e141a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e14104fb0 0x7f2e141a52f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009fd0 tx=0x7f2e0c00ef10 comp rx=0 tx=0).stop 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 shutdown_connections 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2e14104fb0 0x7f2e141a52f0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e14102600 0x7f2e1419e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2df80a1d80 0x7f2df80a4240 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 --2- 172.21.15.116:0/220348408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e140ffc90 0x7f2e1419dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 >> 172.21.15.116:0/220348408 conn(0x7f2e140fb890 msgr2=0x7f2e14102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 shutdown_connections 2024-03-04T07:46:38.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.280+0000 7f2e13200640 1 -- 172.21.15.116:0/220348408 wait complete. 2024-03-04T07:46:38.300 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:46:38.300 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:46:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 -- 172.21.15.116:0/3162552426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 msgr2=0x7f947410bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 --2- 172.21.15.116:0/3162552426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 0x7f947410bc50 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f946c009860 tx=0x7f946c02bb70 comp rx=0 tx=0).stop 2024-03-04T07:46:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 -- 172.21.15.116:0/3162552426 shutdown_connections 2024-03-04T07:46:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 --2- 172.21.15.116:0/3162552426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 0x7f947410bc50 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 --2- 172.21.15.116:0/3162552426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474104a00 0x7f9474104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 --2- 172.21.15.116:0/3162552426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474103810 0x7f9474103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 -- 172.21.15.116:0/3162552426 >> 172.21.15.116:0/3162552426 conn(0x7f94740fed80 msgr2=0x7f94741011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 -- 172.21.15.116:0/3162552426 shutdown_connections 2024-03-04T07:46:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.507+0000 7f947a600640 1 -- 172.21.15.116:0/3162552426 wait complete. 2024-03-04T07:46:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.508+0000 7f947a600640 1 Processor -- start 2024-03-04T07:46:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.508+0000 7f947a600640 1 -- start start 2024-03-04T07:46:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.508+0000 7f947a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474103810 0x7f9474199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.508+0000 7f947a600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474104a00 0x7f9474199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.508+0000 7f947a600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 0x7f94741a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.508+0000 7f947a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947410f470 con 0x7f9474103810 2024-03-04T07:46:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f947a600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947410f2f0 con 0x7f94741053c0 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f947a600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947410f5f0 con 0x7f9474104a00 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474103810 0x7f9474199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474103810 0x7f9474199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47444/0 (socket says 172.21.15.116:47444) 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 -- 172.21.15.116:0/2781427982 learned_addr learned my addr 172.21.15.116:0/2781427982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473e00640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 0x7f94741a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9472a00640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474104a00 0x7f9474199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 -- 172.21.15.116:0/2781427982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474104a00 msgr2=0x7f9474199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474104a00 0x7f9474199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 -- 172.21.15.116:0/2781427982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 msgr2=0x7f94741a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 0x7f94741a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 -- 172.21.15.116:0/2781427982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94741a16a0 con 0x7f9474103810 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9472a00640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474104a00 0x7f9474199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473e00640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 0x7f94741a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.509+0000 7f9473400640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474103810 0x7f9474199a80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9464009fd0 tx=0x7f946400c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.510+0000 7f946be00640 1 -- 172.21.15.116:0/2781427982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464011470 con 0x7f9474103810 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.510+0000 7f946be00640 1 -- 172.21.15.116:0/2781427982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9464011af0 con 0x7f9474103810 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.510+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94741a1990 con 0x7f9474103810 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.510+0000 7f946be00640 1 -- 172.21.15.116:0/2781427982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946401ab90 con 0x7f9474103810 2024-03-04T07:46:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.510+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94741a1ee0 con 0x7f9474103810 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.514+0000 7f947d424740 1 Processor -- start 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.514+0000 7f947d424740 1 -- start start 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.514+0000 7f946be00640 1 -- 172.21.15.116:0/2781427982 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9464011610 con 0x7f9474103810 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f947d424740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e79890c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f947d424740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e798e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f947d424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e798e800 0x55d9e7990c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f947d424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e78edd60 con 0x55d9e798e800 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f947d424740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e7896cf0 con 0x55d9e7989690 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f947d424740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e78e9e00 con 0x55d9e7988ca0 2024-03-04T07:46:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e798e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e798e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36686/0 (socket says 172.21.15.116:36686) 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 -- 172.21.15.116:0/1817351976 learned_addr learned my addr 172.21.15.116:0/1817351976 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9472a00640 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e798e800 0x55d9e7990c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473e00640 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e79890c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 -- 172.21.15.116:0/1817351976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 msgr2=0x55d9e79890c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e79890c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 -- 172.21.15.116:0/1817351976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e798e800 msgr2=0x55d9e7990c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e798e800 0x55d9e7990c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9473400640 1 -- 172.21.15.116:0/1817351976 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9e76023f0 con 0x55d9e7989690 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.515+0000 7f9472a00640 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e798e800 0x55d9e7990c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.516+0000 7f9473400640 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e798e190 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f94640d65d0 tx=0x7f94640d6a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9cb8404b9914db server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.516+0000 7f946be00640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94580a1d80 0x7f94580a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.516+0000 7f9468c00640 1 -- 172.21.15.116:0/1817351976 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640e2070 con 0x55d9e7989690 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.516+0000 7f9468c00640 1 -- 172.21.15.116:0/1817351976 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f946409b840 con 0x55d9e7989690 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.516+0000 7f9468c00640 1 -- 172.21.15.116:0/1817351976 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640bc5c0 con 0x55d9e7989690 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.516+0000 7f9473e00640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94580a1d80 0x7f94580a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.516+0000 7f946be00640 1 -- 172.21.15.116:0/2781427982 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f94640ce410 con 0x7f9474103810 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 -- 172.21.15.116:0/1817351976 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 msgr2=0x55d9e798e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e798e190 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f94640d65d0 tx=0x7f94640d6a90 comp rx=0 tx=0).stop 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 -- 172.21.15.116:0/1817351976 shutdown_connections 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e798e800 0x55d9e7990c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e798e190 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 --2- 172.21.15.116:0/1817351976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e79890c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 -- 172.21.15.116:0/1817351976 >> 172.21.15.116:0/1817351976 conn(0x55d9e7984210 msgr2=0x55d9e7984610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f9473e00640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94580a1d80 0x7f94580a4240 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f946c02c180 tx=0x7f946c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 -- 172.21.15.116:0/1817351976 shutdown_connections 2024-03-04T07:46:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.517+0000 7f947d424740 1 -- 172.21.15.116:0/1817351976 wait complete. 2024-03-04T07:46:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 -- start start 2024-03-04T07:46:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 Processor -- start 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e79931c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e7993700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e799a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e79917b0 con 0x55d9e7988ca0 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e788c220 con 0x55d9e7989690 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e7991930 con 0x55d9e798e800 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f947d424740 10 client.0 _ll_register_callbacks cb 0x55d9e7601be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.518+0000 7f9472a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e7993700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e79931c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9472a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e7993700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36696/0 (socket says 172.21.15.116:36696) 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473e00640 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e799a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9472a00640 1 -- 172.21.15.116:0/1742053925 learned_addr learned my addr 172.21.15.116:0/1742053925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473400640 1 -- 172.21.15.116:0/1742053925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 msgr2=0x55d9e799a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473400640 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e799a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473400640 1 -- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 msgr2=0x55d9e7993700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473400640 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e7993700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473400640 1 -- 172.21.15.116:0/1742053925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9e76cc7d0 con 0x55d9e7988ca0 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9472a00640 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e7993700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9473400640 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e79931c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f94640d65d0 tx=0x7f94640d6f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b1c7f21cb810b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9461600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.519+0000 7f9461600640 1 -- 172.21.15.116:0/1742053925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640bca50 con 0x55d9e7988ca0 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f9461600640 1 -- 172.21.15.116:0/1742053925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9464099440 con 0x55d9e7988ca0 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f946409b840 con 0x55d9e7988ca0 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f9461600640 1 -- 172.21.15.116:0/1742053925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94640bd6e0 con 0x55d9e7988ca0 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 10 client.16671 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:38.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d9e79917b0 con 0x55d9e7988ca0 2024-03-04T07:46:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f9461600640 1 -- 172.21.15.116:0/1742053925 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f946409b840 con 0x55d9e7988ca0 2024-03-04T07:46:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f9461600640 1 -- 172.21.15.116:0/1742053925 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9464022030 con 0x55d9e7988ca0 2024-03-04T07:46:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 10 client.16671 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 20 client.16671 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e799ec30 0x55d9e79a1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 4 client.16671 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.520+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9e78fb2a0 con 0x55d9e799ec30 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.521+0000 7f9472a00640 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e799ec30 0x55d9e79a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.521+0000 7f9472a00640 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e799ec30 0x55d9e79a1020 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ae14dc26fed5bb9 server_cookie=1e07d2fa2f398c5e in_seq=0 out_seq=0 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.521+0000 7f9461600640 10 client.16671 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f9461600640 1 -- 172.21.15.116:0/1742053925 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d9e78fb2a0 con 0x55d9e799ec30 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f9461600640 10 client.16671 handle_command_reply: tid=0 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 client.16671 shutdown 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 msgr2=0x55d9e79931c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e79931c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f94640d65d0 tx=0x7f94640d6f90 comp rx=0 tx=0).stop 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 shutdown_connections 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e799ec30 0x55d9e79a1020 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e799a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e7993700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f9461600640 0 client.16671 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 --2- 172.21.15.116:0/1742053925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e79931c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 >> 172.21.15.116:0/1742053925 conn(0x55d9e7984210 msgr2=0x55d9e79834b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 shutdown_connections 2024-03-04T07:46:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 1 -- 172.21.15.116:0/1742053925 wait complete. 2024-03-04T07:46:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.522+0000 7f947d424740 20 client.16671 trim_cache size 0 max 16384 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 Processor -- start 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 -- start start 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e7993440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e799a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e79a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741a8ce0 con 0x55d9e7988ca0 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741ab2c0 con 0x55d9e7989690 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f947d424740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741a92e0 con 0x55d9e798e800 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f9473400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e7993440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f9472a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e799a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f9472a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e799a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36708/0 (socket says 172.21.15.116:36708) 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f9473400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e7993440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47474/0 (socket says 172.21.15.116:47474) 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f9472a00640 1 -- 172.21.15.116:0/1604417938 learned_addr learned my addr 172.21.15.116:0/1604417938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.536+0000 7f9473e00640 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e79a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9472a00640 1 -- 172.21.15.116:0/1604417938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 msgr2=0x55d9e79a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9472a00640 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e79a5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9472a00640 1 -- 172.21.15.116:0/1604417938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 msgr2=0x55d9e7993440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9472a00640 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e7993440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9472a00640 1 -- 172.21.15.116:0/1604417938 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9e7984f80 con 0x55d9e7989690 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9473400640 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e7993440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9472a00640 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e799a700 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x55d9e79a1790 tx=0x7f945c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bad029f285bd5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9463e00640 1 -- 172.21.15.116:0/1604417938 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c020c70 con 0x55d9e7989690 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9463e00640 1 -- 172.21.15.116:0/1604417938 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f945c020e10 con 0x55d9e7989690 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.537+0000 7f9463e00640 1 -- 172.21.15.116:0/1604417938 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c0047a0 con 0x55d9e7989690 2024-03-04T07:46:38.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 -- 172.21.15.116:0/1604417938 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 msgr2=0x55d9e799a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e799a700 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x55d9e79a1790 tx=0x7f945c009260 comp rx=0 tx=0).stop 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 -- 172.21.15.116:0/1604417938 shutdown_connections 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e798e800 0x55d9e79a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e7989690 0x55d9e799a700 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 --2- 172.21.15.116:0/1604417938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7988ca0 0x55d9e7993440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 -- 172.21.15.116:0/1604417938 >> 172.21.15.116:0/1604417938 conn(0x55d9e7984210 msgr2=0x55d9e7987d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 -- 172.21.15.116:0/1604417938 shutdown_connections 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 -- 172.21.15.116:0/1604417938 wait complete. 2024-03-04T07:46:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.538+0000 7f947d424740 1 -- start start 2024-03-04T07:46:38.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 1 Processor -- start 2024-03-04T07:46:38.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e799a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7989690 0x55d9e7998da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e798e800 0x55d9e79993b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741a9940 con 0x55d9e7989690 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741aac60 con 0x55d9e798e800 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741a9fa0 con 0x55d9e7988ca0 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f947d424740 10 client.0 _ll_register_callbacks cb 0x7f94640e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f9473400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e799a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f9473e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e798e800 0x55d9e79993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f9472a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7989690 0x55d9e7998da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f9473400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e799a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58088/0 (socket says 172.21.15.116:58088) 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.539+0000 7f9473400640 1 -- 172.21.15.116:0/1382766905 learned_addr learned my addr 172.21.15.116:0/1382766905 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:38.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9473e00640 1 -- 172.21.15.116:0/1382766905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 msgr2=0x55d9e799a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9473e00640 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e799a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9473e00640 1 -- 172.21.15.116:0/1382766905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7989690 msgr2=0x55d9e7998da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9473e00640 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7989690 0x55d9e7998da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9473e00640 1 -- 172.21.15.116:0/1382766905 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9e7999be0 con 0x55d9e798e800 2024-03-04T07:46:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9472a00640 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7989690 0x55d9e7998da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9473400640 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e799a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9473e00640 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e798e800 0x55d9e79993b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f946c009780 tx=0x7f946c02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6262d8c905ac7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9461600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9461600640 1 -- 172.21.15.116:0/1382766905 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c052070 con 0x55d9e798e800 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9461600640 1 -- 172.21.15.116:0/1382766905 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f946c02c920 con 0x55d9e798e800 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f945c020e10 con 0x55d9e798e800 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.540+0000 7f9461600640 1 -- 172.21.15.116:0/1382766905 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f946c039040 con 0x55d9e798e800 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 10 client.26804 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d9e79fb5b0 con 0x55d9e798e800 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f9461600640 1 -- 172.21.15.116:0/1382766905 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f945c020e10 con 0x55d9e798e800 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f9461600640 1 -- 172.21.15.116:0/1382766905 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f946c04b050 con 0x55d9e798e800 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 10 client.26804 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 10 client.26804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 20 client.26804 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e79fbee0 0x55d9e79fe300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 4 client.26804 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d9e78f55f0 con 0x55d9e79fbee0 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.541+0000 7f9473400640 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e79fbee0 0x55d9e79fe300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f9461600640 10 client.26804 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f9473400640 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e79fbee0 0x55d9e79fe300 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c19db211a9b7c4a6 server_cookie=be8070b6cad10a7f in_seq=0 out_seq=0 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f9461600640 1 -- 172.21.15.116:0/1382766905 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d9e78f55f0 con 0x55d9e79fbee0 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f9461600640 10 client.26804 handle_command_reply: tid=0 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 client.26804 shutdown 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e798e800 msgr2=0x55d9e79993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e798e800 0x55d9e79993b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f946c009780 tx=0x7f946c02bb70 comp rx=0 tx=0).stop 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 shutdown_connections 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d9e79fbee0 0x55d9e79fe300 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d9e798e800 0x55d9e79993b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f9461600640 0 client.26804 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d9e7989690 0x55d9e7998da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 --2- 172.21.15.116:0/1382766905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9e7988ca0 0x55d9e799a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 >> 172.21.15.116:0/1382766905 conn(0x55d9e7984210 msgr2=0x55d9e7987d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 shutdown_connections 2024-03-04T07:46:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 1 -- 172.21.15.116:0/1382766905 wait complete. 2024-03-04T07:46:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.542+0000 7f947d424740 20 client.26804 trim_cache size 0 max 16384 2024-03-04T07:46:38.553 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:46:38.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.543+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94580a1d80 msgr2=0x7f94580a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.543+0000 7f947a600640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94580a1d80 0x7f94580a4240 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f946c02c180 tx=0x7f946c02e040 comp rx=0 tx=0).stop 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.543+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474103810 msgr2=0x7f9474199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.543+0000 7f947a600640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474103810 0x7f9474199a80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9464009fd0 tx=0x7f946400c270 comp rx=0 tx=0).stop 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 shutdown_connections 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94741053c0 0x7f94741a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94580a1d80 0x7f94580a4240 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9474104a00 0x7f9474199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 --2- 172.21.15.116:0/2781427982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9474103810 0x7f9474199a80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 >> 172.21.15.116:0/2781427982 conn(0x7f94740fed80 msgr2=0x7f9474107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 shutdown_connections 2024-03-04T07:46:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:38.544+0000 7f947a600640 1 -- 172.21.15.116:0/2781427982 wait complete. 2024-03-04T07:46:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:38 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:46:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:38 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:38 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:46:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:38 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:38 smithi133 ceph-mon[36505]: pgmap v577: 97 pgs: 97 active+clean; 4.3 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:38 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:46:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:38 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/220348408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:38 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:46:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:38 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:38 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:46:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:38 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:38 smithi116 ceph-mon[32150]: pgmap v577: 97 pgs: 97 active+clean; 4.3 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:38 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:46:39.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:38 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/220348408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:38 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:46:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:38 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:38 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:46:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:38 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:38 smithi152 ceph-mon[37154]: pgmap v577: 97 pgs: 97 active+clean; 4.3 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:46:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:38 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:46:39.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:38 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/220348408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:46:39.562 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:46:39.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:46:39.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:46:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 -- 172.21.15.116:0/3124629860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 msgr2=0x7f87dc102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3124629860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 0x7f87dc102b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009900 tx=0x7f87c402bc40 comp rx=0 tx=0).stop 2024-03-04T07:46:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 -- 172.21.15.116:0/3124629860 shutdown_connections 2024-03-04T07:46:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3124629860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87dc103060 0x7f87dc1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3124629860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 0x7f87dc102b20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3124629860 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc1014b0 0x7f87dc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 -- 172.21.15.116:0/3124629860 >> 172.21.15.116:0/3124629860 conn(0x7f87dc0fca40 msgr2=0x7f87dc0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 -- 172.21.15.116:0/3124629860 shutdown_connections 2024-03-04T07:46:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.780+0000 7f87dbe00640 1 -- 172.21.15.116:0/3124629860 wait complete. 2024-03-04T07:46:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 Processor -- start 2024-03-04T07:46:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 -- start start 2024-03-04T07:46:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87dc1014b0 0x7f87dc1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 0x7f87dc199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc103060 0x7f87dc1a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc10d190 con 0x7f87dc1014b0 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc10d010 con 0x7f87dc103060 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.781+0000 7f87dbe00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc10d310 con 0x7f87dc1026a0 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc103060 0x7f87dc1a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87daa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87dc1014b0 0x7f87dc1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc103060 0x7f87dc1a0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36734/0 (socket says 172.21.15.116:36734) 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 -- 172.21.15.116:0/3070617008 learned_addr learned my addr 172.21.15.116:0/3070617008 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87da000640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 0x7f87dc199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 -- 172.21.15.116:0/3070617008 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 msgr2=0x7f87dc199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 0x7f87dc199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 -- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87dc1014b0 msgr2=0x7f87dc1999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87dc1014b0 0x7f87dc1999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 -- 172.21.15.116:0/3070617008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc1a15c0 con 0x7f87dc103060 2024-03-04T07:46:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87db400640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc103060 0x7f87dc1a0f50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f87d400eb10 tx=0x7f87d400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87e0a00640 1 -- 172.21.15.116:0/3070617008 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4004b20 con 0x7f87dc103060 2024-03-04T07:46:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc1a18b0 con 0x7f87dc103060 2024-03-04T07:46:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87e0a00640 1 -- 172.21.15.116:0/3070617008 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87d4004cc0 con 0x7f87dc103060 2024-03-04T07:46:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.782+0000 7f87e0a00640 1 -- 172.21.15.116:0/3070617008 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4010760 con 0x7f87dc103060 2024-03-04T07:46:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.783+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc1a1d80 con 0x7f87dc103060 2024-03-04T07:46:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 Processor -- start 2024-03-04T07:46:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e0a00640 1 -- 172.21.15.116:0/3070617008 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f87d4005020 con 0x7f87dc103060 2024-03-04T07:46:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 -- start start 2024-03-04T07:46:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a386d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a3d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a87894be20 con 0x55a878a3de10 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a87899da20 con 0x55a878a38ca0 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a878946b00 con 0x55a878a382b0 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87daa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a3d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87da000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87daa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a3d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36750/0 (socket says 172.21.15.116:36750) 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.786+0000 7f87daa00640 1 -- 172.21.15.116:0/1556419470 learned_addr learned my addr 172.21.15.116:0/1556419470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87da000640 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87daa00640 1 -- 172.21.15.116:0/1556419470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 msgr2=0x55a878a386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87daa00640 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a386d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87daa00640 1 -- 172.21.15.116:0/1556419470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 msgr2=0x55a878a402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87daa00640 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87daa00640 1 -- 172.21.15.116:0/1556419470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a87895be10 con 0x55a878a38ca0 2024-03-04T07:46:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87da000640 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87daa00640 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a3d7a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009ca0 tx=0x7f87cc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cab45c48e377003 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87cbe00640 1 -- 172.21.15.116:0/1556419470 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc004220 con 0x55a878a38ca0 2024-03-04T07:46:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87cbe00640 1 -- 172.21.15.116:0/1556419470 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87cc0043c0 con 0x55a878a38ca0 2024-03-04T07:46:39.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.787+0000 7f87cbe00640 1 -- 172.21.15.116:0/1556419470 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc019970 con 0x55a878a38ca0 2024-03-04T07:46:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e0a00640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f87c00a1d80 0x7f87c00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e0a00640 1 -- 172.21.15.116:0/3070617008 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f87d40ce360 con 0x7f87dc103060 2024-03-04T07:46:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 -- 172.21.15.116:0/1556419470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 msgr2=0x55a878a3d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a3d7a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009ca0 tx=0x7f87cc011810 comp rx=0 tx=0).stop 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87da000640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f87c00a1d80 0x7f87c00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 -- 172.21.15.116:0/1556419470 shutdown_connections 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a3d7a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/1556419470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a386d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 -- 172.21.15.116:0/1556419470 >> 172.21.15.116:0/1556419470 conn(0x55a878a33820 msgr2=0x55a878a33c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 -- 172.21.15.116:0/1556419470 shutdown_connections 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.788+0000 7f87e4ba5740 1 -- 172.21.15.116:0/1556419470 wait complete. 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.789+0000 7f87da000640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f87c00a1d80 0x7f87c00a4240 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f87c402c250 tx=0x7f87c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:39.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.789+0000 7f87e4ba5740 1 -- start start 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.789+0000 7f87e4ba5740 1 Processor -- start 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.789+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a42680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.789+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a42bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.789+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a49c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.789+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a878959f40 con 0x55a878a3de10 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a878999ac0 con 0x55a878a38ca0 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a878a40dc0 con 0x55a878a382b0 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87e4ba5740 10 client.0 _ll_register_callbacks cb 0x55a878998aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87da000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a42bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a49c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87daa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a42680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a49c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47514/0 (socket says 172.21.15.116:47514) 2024-03-04T07:46:39.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 -- 172.21.15.116:0/757280494 learned_addr learned my addr 172.21.15.116:0/757280494 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87da000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a42bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36760/0 (socket says 172.21.15.116:36760) 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 -- 172.21.15.116:0/757280494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 msgr2=0x55a878a42680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a42680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 -- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 msgr2=0x55a878a42bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a42bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87db400640 1 -- 172.21.15.116:0/757280494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a87877a7e0 con 0x55a878a3de10 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87daa00640 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a42680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.790+0000 7f87da000640 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a42bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87db400640 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a49c00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f87d4097a60 tx=0x7f87d409aab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87143028607cb9a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87c8c00640 1 -- 172.21.15.116:0/757280494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d40a35a0 con 0x55a878a3de10 2024-03-04T07:46:39.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87c8c00640 1 -- 172.21.15.116:0/757280494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f87d4005310 con 0x55a878a3de10 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f87cc0043c0 con 0x55a878a3de10 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87c8c00640 1 -- 172.21.15.116:0/757280494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87d409f1e0 con 0x55a878a3de10 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87e4ba5740 10 client.16689 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a8787daf70 con 0x55a878a3de10 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87c8c00640 1 -- 172.21.15.116:0/757280494 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87cc0043c0 con 0x55a878a3de10 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.791+0000 7f87c8c00640 1 -- 172.21.15.116:0/757280494 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f87d4099f50 con 0x55a878a3de10 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87e4ba5740 10 client.16689 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87e4ba5740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87e4ba5740 20 client.16689 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878a4e4a0 0x55a878a508f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87e4ba5740 4 client.16689 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8789aaf60 con 0x55a878a4e4a0 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87daa00640 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878a4e4a0 0x55a878a508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87daa00640 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878a4e4a0 0x55a878a508f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7daf427d03b0f84c server_cookie=f318261b207ea04b in_seq=0 out_seq=0 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.792+0000 7f87c8c00640 10 client.16689 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87c8c00640 1 -- 172.21.15.116:0/757280494 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a8789aaf60 con 0x55a878a4e4a0 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87c8c00640 10 client.16689 handle_command_reply: tid=0 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 client.16689 shutdown 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 msgr2=0x55a878a49c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a49c00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f87d4097a60 tx=0x7f87d409aab0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 shutdown_connections 2024-03-04T07:46:39.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878a4e4a0 0x55a878a508f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a3de10 0x55a878a49c00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a38ca0 0x55a878a42bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87c8c00640 0 client.16689 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/757280494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a42680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.793+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 >> 172.21.15.116:0/757280494 conn(0x55a878a33820 msgr2=0x55a878a32a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.794+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 shutdown_connections 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.794+0000 7f87e4ba5740 1 -- 172.21.15.116:0/757280494 wait complete. 2024-03-04T07:46:39.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.794+0000 7f87e4ba5740 20 client.16689 trim_cache size 0 max 16384 2024-03-04T07:46:39.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 Processor -- start 2024-03-04T07:46:39.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 -- start start 2024-03-04T07:46:39.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a382b0 0x55a878a321d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a38ca0 0x55a878a49bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a32710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1a89e0 con 0x55a878a382b0 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1aafc0 con 0x55a878a3de10 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.811+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1a8fe0 con 0x55a878a38ca0 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87daa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a382b0 0x55a878a321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a32710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a32710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36764/0 (socket says 172.21.15.116:36764) 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 -- 172.21.15.116:0/2372605883 learned_addr learned my addr 172.21.15.116:0/2372605883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87da000640 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a38ca0 0x55a878a49bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 -- 172.21.15.116:0/2372605883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a38ca0 msgr2=0x55a878a49bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a38ca0 0x55a878a49bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 -- 172.21.15.116:0/2372605883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a382b0 msgr2=0x55a878a321d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a382b0 0x55a878a321d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 -- 172.21.15.116:0/2372605883 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a878672170 con 0x55a878a3de10 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87db400640 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a32710 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f87d4005880 tx=0x7f87d40995a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79635355ea5cdd9e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.812+0000 7f87cb400640 1 -- 172.21.15.116:0/2372605883 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d40a3830 con 0x55a878a3de10 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87cb400640 1 -- 172.21.15.116:0/2372605883 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87d40b1070 con 0x55a878a3de10 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87cb400640 1 -- 172.21.15.116:0/2372605883 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4005570 con 0x55a878a3de10 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 -- 172.21.15.116:0/2372605883 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 msgr2=0x55a878a32710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a32710 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f87d4005880 tx=0x7f87d40995a0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 -- 172.21.15.116:0/2372605883 shutdown_connections 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a32710 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a38ca0 0x55a878a49bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/2372605883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a382b0 0x55a878a321d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 -- 172.21.15.116:0/2372605883 >> 172.21.15.116:0/2372605883 conn(0x55a878a33820 msgr2=0x55a878a3a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 -- 172.21.15.116:0/2372605883 shutdown_connections 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.813+0000 7f87e4ba5740 1 -- 172.21.15.116:0/2372605883 wait complete. 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 -- start start 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 Processor -- start 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a47500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a38ca0 0x55a878a47a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a48a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1a9640 con 0x55a878a38ca0 2024-03-04T07:46:39.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1aa960 con 0x55a878a3de10 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc1a9ca0 con 0x55a878a382b0 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.814+0000 7f87e4ba5740 10 client.0 _ll_register_callbacks cb 0x7f87d40057d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a48a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87da000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a38ca0 0x55a878a47a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87daa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a47500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a48a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36774/0 (socket says 172.21.15.116:36774) 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 -- 172.21.15.116:0/3433359152 learned_addr learned my addr 172.21.15.116:0/3433359152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 -- 172.21.15.116:0/3433359152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 msgr2=0x55a878a47500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a47500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 -- 172.21.15.116:0/3433359152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a38ca0 msgr2=0x55a878a47a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a38ca0 0x55a878a47a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 -- 172.21.15.116:0/3433359152 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a878a48ff0 con 0x55a878a3de10 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87db400640 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a48a00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f87d409b250 tx=0x7f87d4004e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a739b47d5e0cdbda server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87c8c00640 1 -- 172.21.15.116:0/3433359152 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d409a7e0 con 0x55a878a3de10 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87c8c00640 1 -- 172.21.15.116:0/3433359152 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f87d4005310 con 0x55a878a3de10 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.815+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f87d40b1070 con 0x55a878a3de10 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87c8c00640 1 -- 172.21.15.116:0/3433359152 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87d40a3830 con 0x55a878a3de10 2024-03-04T07:46:39.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 10 client.26822 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a878aa9fc0 con 0x55a878a3de10 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87c8c00640 1 -- 172.21.15.116:0/3433359152 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87d40b1070 con 0x55a878a3de10 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87c8c00640 1 -- 172.21.15.116:0/3433359152 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f87d40ab030 con 0x55a878a3de10 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 10 client.26822 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 10 client.26822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 20 client.26822 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878aaa580 0x55a878aac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 4 client.26822 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a8789a52b0 con 0x55a878aaa580 2024-03-04T07:46:39.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.816+0000 7f87daa00640 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878aaa580 0x55a878aac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:39.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.817+0000 7f87daa00640 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878aaa580 0x55a878aac9a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be33771850d4c6df server_cookie=720098edc26fbbfb in_seq=0 out_seq=0 2024-03-04T07:46:39.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.817+0000 7f87c8c00640 10 client.26822 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:39.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87c8c00640 1 -- 172.21.15.116:0/3433359152 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a8789a52b0 con 0x55a878aaa580 2024-03-04T07:46:39.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87c8c00640 10 client.26822 handle_command_reply: tid=0 2024-03-04T07:46:39.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 client.26822 shutdown 2024-03-04T07:46:39.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 msgr2=0x55a878a48a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a48a00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f87d409b250 tx=0x7f87d4004e90 comp rx=0 tx=0).stop 2024-03-04T07:46:39.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 shutdown_connections 2024-03-04T07:46:39.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a878aaa580 0x55a878aac9a0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a878a3de10 0x55a878a48a00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a878a38ca0 0x55a878a47a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 --2- 172.21.15.116:0/3433359152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a878a382b0 0x55a878a47500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87c8c00640 0 client.26822 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 >> 172.21.15.116:0/3433359152 conn(0x55a878a33820 msgr2=0x55a878a36a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 shutdown_connections 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 1 -- 172.21.15.116:0/3433359152 wait complete. 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.818+0000 7f87e4ba5740 20 client.26822 trim_cache size 0 max 16384 2024-03-04T07:46:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:39.828 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:46:39.828 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "67ef69a1-bb73-4f4c-ad3c-775745506056", 2024-03-04T07:46:39.828 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:46:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f87c00a1d80 msgr2=0x7f87c00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f87c00a1d80 0x7f87c00a4240 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f87c402c250 tx=0x7f87c4005fd0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc103060 msgr2=0x7f87dc1a0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc103060 0x7f87dc1a0f50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f87d400eb10 tx=0x7f87d400c2d0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 shutdown_connections 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87dc103060 0x7f87dc1a0f50 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f87c00a1d80 0x7f87c00a4240 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87dc1026a0 0x7f87dc199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 --2- 172.21.15.116:0/3070617008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87dc1014b0 0x7f87dc1999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 >> 172.21.15.116:0/3070617008 conn(0x7f87dc0fca40 msgr2=0x7f87dc073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:39.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 shutdown_connections 2024-03-04T07:46:39.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:39.819+0000 7f87dbe00640 1 -- 172.21.15.116:0/3070617008 wait complete. 2024-03-04T07:46:39.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:46:39.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:46:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 -- 172.21.15.116:0/1438676100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc41027c0 msgr2=0x7fcfc4104c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/1438676100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc41027c0 0x7fcfc4104c30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009900 tx=0x7fcfb002bc30 comp rx=0 tx=0).stop 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 -- 172.21.15.116:0/1438676100 shutdown_connections 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/1438676100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfc4105170 0x7fcfc41055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/1438676100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc41027c0 0x7fcfc4104c30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/1438676100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 0x7fcfc4102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 -- 172.21.15.116:0/1438676100 >> 172.21.15.116:0/1438676100 conn(0x7fcfc40fba70 msgr2=0x7fcfc40fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 -- 172.21.15.116:0/1438676100 shutdown_connections 2024-03-04T07:46:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.047+0000 7fcfc8a00640 1 -- 172.21.15.116:0/1438676100 wait complete. 2024-03-04T07:46:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.048+0000 7fcfc8a00640 1 Processor -- start 2024-03-04T07:46:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.048+0000 7fcfc8a00640 1 -- start start 2024-03-04T07:46:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 0x7fcfc419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc8a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfc41027c0 0x7fcfc419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc4105170 0x7fcfc41a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4113880 con 0x7fcfc4105170 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc8a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4113700 con 0x7fcfc40ffe50 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc8a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4113a00 con 0x7fcfc41027c0 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 0x7fcfc419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc4105170 0x7fcfc41a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 0x7fcfc419def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36796/0 (socket says 172.21.15.116:36796) 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3213321311 learned_addr learned my addr 172.21.15.116:0/3213321311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1400640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfc41027c0 0x7fcfc419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3213321311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfc41027c0 msgr2=0x7fcfc419e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfc41027c0 0x7fcfc419e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc4105170 msgr2=0x7fcfc41a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc4105170 0x7fcfc41a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3213321311 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc41a5b10 con 0x7fcfc40ffe50 2024-03-04T07:46:40.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.049+0000 7fcfc2800640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc4105170 0x7fcfc41a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.050+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 0x7fcfc419def0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400e9f0 tx=0x7fcfb400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.050+0000 7fcfbaa00640 1 -- 172.21.15.116:0/3213321311 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb400cd80 con 0x7fcfc40ffe50 2024-03-04T07:46:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.050+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc41a5e00 con 0x7fcfc40ffe50 2024-03-04T07:46:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.050+0000 7fcfbaa00640 1 -- 172.21.15.116:0/3213321311 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfb4010910 con 0x7fcfc40ffe50 2024-03-04T07:46:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.050+0000 7fcfbaa00640 1 -- 172.21.15.116:0/3213321311 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40188f0 con 0x7fcfc40ffe50 2024-03-04T07:46:40.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.050+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc41a6270 con 0x7fcfc40ffe50 2024-03-04T07:46:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.054+0000 7fcfcb7e0740 1 Processor -- start 2024-03-04T07:46:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfcb7e0740 1 -- start start 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfbaa00640 1 -- 172.21.15.116:0/3213321311 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fcfb4010ac0 con 0x7fcfc40ffe50 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190d26b0 0x55dc190d4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc18ffa090 con 0x55dc190ccb50 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc19031d30 con 0x55dc190cd540 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc18fdaa30 con 0x55dc190d26b0 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfc2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.055+0000 7fcfc1400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190d26b0 0x55dc190d4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190d26b0 0x55dc190d4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58170/0 (socket says 172.21.15.116:58170) 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 -- 172.21.15.116:0/2234617566 learned_addr learned my addr 172.21.15.116:0/2234617566 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47550/0 (socket says 172.21.15.116:47550) 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 -- 172.21.15.116:0/2234617566 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 msgr2=0x55dc190d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 -- 172.21.15.116:0/2234617566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 msgr2=0x55dc190ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 -- 172.21.15.116:0/2234617566 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc18e107d0 con 0x55dc190d26b0 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc2800640 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:40.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.056+0000 7fcfc1400640 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190d26b0 0x55dc190d4b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0000c00 tx=0x7fcfb00095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfdbef2ec0fba6b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.057+0000 7fcfaf000640 1 -- 172.21.15.116:0/2234617566 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0045070 con 0x55dc190d26b0 2024-03-04T07:46:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.057+0000 7fcfaf000640 1 -- 172.21.15.116:0/2234617566 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0004040 con 0x55dc190d26b0 2024-03-04T07:46:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.057+0000 7fcfaf000640 1 -- 172.21.15.116:0/2234617566 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0004eb0 con 0x55dc190d26b0 2024-03-04T07:46:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.057+0000 7fcfbaa00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfa80a1e00 0x7fcfa80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.057+0000 7fcfbaa00640 1 -- 172.21.15.116:0/3213321311 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcfb40ce190 con 0x7fcfc40ffe50 2024-03-04T07:46:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.057+0000 7fcfc2800640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfa80a1e00 0x7fcfa80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2234617566 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190d26b0 msgr2=0x55dc190d4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190d26b0 0x55dc190d4b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0000c00 tx=0x7fcfb00095e0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2234617566 shutdown_connections 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190d26b0 0x55dc190d4b40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2234617566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2234617566 >> 172.21.15.116:0/2234617566 conn(0x55dc190c80c0 msgr2=0x55dc190c84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2234617566 shutdown_connections 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfc2800640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfa80a1e00 0x7fcfa80a42c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55dc190cdda0 tx=0x7fcfbc0096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:40.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2234617566 wait complete. 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.058+0000 7fcfcb7e0740 1 -- start start 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 1 Processor -- start 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190ccb50 0x55dc190d6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190de3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc18fd00c0 con 0x55dc190d26b0 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc1902ddd0 con 0x55dc190cd540 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc190d5600 con 0x55dc190ccb50 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfcb7e0740 10 client.0 _ll_register_callbacks cb 0x55dc18fed860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:40.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfc2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190de3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfc1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190ccb50 0x55dc190d6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfc1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36808/0 (socket says 172.21.15.116:36808) 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfc2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190de3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47560/0 (socket says 172.21.15.116:47560) 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.059+0000 7fcfc1400640 1 -- 172.21.15.116:0/1529489417 learned_addr learned my addr 172.21.15.116:0/1529489417 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfc1e00640 1 -- 172.21.15.116:0/1529489417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 msgr2=0x55dc190d7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfc1e00640 1 -- 172.21.15.116:0/1529489417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 msgr2=0x55dc190de3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190de3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfc1e00640 1 -- 172.21.15.116:0/1529489417 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc18d463f0 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190ccb50 0x55dc190d6e50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55dc190cdfb0 tx=0x7fcfb409ad00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f7e6f528d846b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfa3400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfa3400640 1 -- 172.21.15.116:0/1529489417 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4018d90 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfa3400640 1 -- 172.21.15.116:0/1529489417 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcfb40d00b0 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfb0004040 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.060+0000 7fcfa3400640 1 -- 172.21.15.116:0/1529489417 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfb409af30 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 10 client.26734 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55dc18d060f0 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfa3400640 1 -- 172.21.15.116:0/1529489417 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0004040 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfa3400640 1 -- 172.21.15.116:0/1529489417 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcfb409a4a0 con 0x55dc190ccb50 2024-03-04T07:46:40.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 10 client.26734 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 10 client.26734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 20 client.26734 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc190e29a0 0x55dc190e4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 4 client.26734 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc1903f270 con 0x55dc190e29a0 2024-03-04T07:46:40.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.061+0000 7fcfc1400640 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc190e29a0 0x55dc190e4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.062+0000 7fcfc1400640 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc190e29a0 0x55dc190e4e10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=611c7f40a75c7600 server_cookie=a352c3590fe5b772 in_seq=0 out_seq=0 2024-03-04T07:46:40.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.062+0000 7fcfa3400640 10 client.26734 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:40.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfa3400640 1 -- 172.21.15.116:0/1529489417 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dc1903f270 con 0x55dc190e29a0 2024-03-04T07:46:40.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfa3400640 10 client.26734 handle_command_reply: tid=0 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 client.26734 shutdown 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190ccb50 msgr2=0x55dc190d6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190ccb50 0x55dc190d6e50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55dc190cdfb0 tx=0x7fcfb409ad00 comp rx=0 tx=0).stop 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 shutdown_connections 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc190e29a0 0x55dc190e4e10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190de3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190cd540 0x55dc190d7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfa3400640 0 client.26734 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/1529489417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190ccb50 0x55dc190d6e50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 >> 172.21.15.116:0/1529489417 conn(0x55dc190c80c0 msgr2=0x55dc190c7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 shutdown_connections 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/1529489417 wait complete. 2024-03-04T07:46:40.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.063+0000 7fcfcb7e0740 20 client.26734 trim_cache size 0 max 16384 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.080+0000 7fcfcb7e0740 1 Processor -- start 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.080+0000 7fcfcb7e0740 1 -- start start 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190d70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190d26b0 0x55dc190e96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41acf70 con 0x55dc190ccb50 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41af550 con 0x55dc190d26b0 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41ad570 con 0x55dc190cd540 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190d70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190d26b0 0x55dc190e96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190d70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47568/0 (socket says 172.21.15.116:47568) 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc1e00640 1 -- 172.21.15.116:0/2609123674 learned_addr learned my addr 172.21.15.116:0/2609123674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc1400640 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc2800640 1 -- 172.21.15.116:0/2609123674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 msgr2=0x55dc190de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc2800640 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc2800640 1 -- 172.21.15.116:0/2609123674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 msgr2=0x55dc190d70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc2800640 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190d70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc2800640 1 -- 172.21.15.116:0/2609123674 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc190cddd0 con 0x55dc190d26b0 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.081+0000 7fcfc1400640 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.082+0000 7fcfc2800640 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190d26b0 0x55dc190e96d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc011d50 tx=0x7fcfbc019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9f6322abd00cb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.082+0000 7fcfae600640 1 -- 172.21.15.116:0/2609123674 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc011d80 con 0x55dc190d26b0 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.082+0000 7fcfae600640 1 -- 172.21.15.116:0/2609123674 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfbc005020 con 0x55dc190d26b0 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.082+0000 7fcfae600640 1 -- 172.21.15.116:0/2609123674 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc028680 con 0x55dc190d26b0 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.082+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2609123674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190d26b0 msgr2=0x55dc190e96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.082+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190d26b0 0x55dc190e96d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc011d50 tx=0x7fcfbc019000 comp rx=0 tx=0).stop 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2609123674 shutdown_connections 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190d26b0 0x55dc190e96d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/2609123674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190ccb50 0x55dc190d70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2609123674 >> 172.21.15.116:0/2609123674 conn(0x55dc190c80c0 msgr2=0x55dc190cbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2609123674 shutdown_connections 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/2609123674 wait complete. 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 -- start start 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.083+0000 7fcfcb7e0740 1 Processor -- start 2024-03-04T07:46:40.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190ccb50 0x55dc190dcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190dcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfcb7e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190dd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41adbd0 con 0x55dc190d26b0 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41aeef0 con 0x55dc190ccb50 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfcb7e0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41ae230 con 0x55dc190cd540 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfcb7e0740 10 client.0 _ll_register_callbacks cb 0x7fcfb4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190ccb50 0x55dc190dcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190ccb50 0x55dc190dcab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36832/0 (socket says 172.21.15.116:36832) 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3877983752 learned_addr learned my addr 172.21.15.116:0/3877983752 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc2800640 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190dd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1400640 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190dcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3877983752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 msgr2=0x55dc190dcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190dcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3877983752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 msgr2=0x55dc190dd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190dd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.084+0000 7fcfc1e00640 1 -- 172.21.15.116:0/3877983752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc1913e2a0 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfc1e00640 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190ccb50 0x55dc190dcab0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c6a0 tx=0x7fcfb400c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e465233ac646dad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfa3400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfa3400640 1 -- 172.21.15.116:0/3877983752 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb409aa70 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfa3400640 1 -- 172.21.15.116:0/3877983752 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcfb4018d50 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfbc005020 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfa3400640 1 -- 172.21.15.116:0/3877983752 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfb409efd0 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 10 client.26840 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55dc1913f580 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfa3400640 1 -- 172.21.15.116:0/3877983752 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfbc005020 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfa3400640 1 -- 172.21.15.116:0/3877983752 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcfb4095aa0 con 0x55dc190ccb50 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 10 client.26840 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:40.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 10 client.26840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 20 client.26840 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc1913fc60 0x55dc19142080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 4 client.26840 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.085+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dc190395c0 con 0x55dc1913fc60 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfc1400640 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc1913fc60 0x55dc19142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfc1400640 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc1913fc60 0x55dc19142080 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b0160c162ea4ff8 server_cookie=c97f17d2c7f0f985 in_seq=0 out_seq=0 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfa3400640 10 client.26840 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfa3400640 1 -- 172.21.15.116:0/3877983752 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55dc190395c0 con 0x55dc1913fc60 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfa3400640 10 client.26840 handle_command_reply: tid=0 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfcb7e0740 1 client.26840 shutdown 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190ccb50 msgr2=0x55dc190dcab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190ccb50 0x55dc190dcab0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c6a0 tx=0x7fcfb400c840 comp rx=0 tx=0).stop 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 shutdown_connections 2024-03-04T07:46:40.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.086+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55dc1913fc60 0x55dc19142080 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc190d26b0 0x55dc190dd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dc190cd540 0x55dc190dcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfcb7e0740 1 --2- 172.21.15.116:0/3877983752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55dc190ccb50 0x55dc190dcab0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfa3400640 0 client.26840 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 >> 172.21.15.116:0/3877983752 conn(0x55dc190c80c0 msgr2=0x55dc190cbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 shutdown_connections 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfcb7e0740 1 -- 172.21.15.116:0/3877983752 wait complete. 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfcb7e0740 20 client.26840 trim_cache size 0 max 16384 2024-03-04T07:46:40.094 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: "67ef69a1-bb73-4f4c-ad3c-775745506056": { 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "67ef69a1-bb73-4f4c-ad3c-775745506056", 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:46:40.095 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfa80a1e00 msgr2=0x7fcfa80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfa80a1e00 0x7fcfa80a42c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55dc190cdda0 tx=0x7fcfbc0096a0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 msgr2=0x7fcfc419def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 0x7fcfc419def0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400e9f0 tx=0x7fcfb400eeb0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 shutdown_connections 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc4105170 0x7fcfc41a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcfa80a1e00 0x7fcfa80a42c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfc41027c0 0x7fcfc419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 --2- 172.21.15.116:0/3213321311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfc40ffe50 0x7fcfc419def0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:40.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.087+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 >> 172.21.15.116:0/3213321311 conn(0x7fcfc40fba70 msgr2=0x7fcfc41007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:40.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.088+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 shutdown_connections 2024-03-04T07:46:40.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:46:40.088+0000 7fcfc8a00640 1 -- 172.21.15.116:0/3213321311 wait complete. 2024-03-04T07:46:40.105 INFO:tasks.cephfs.filesystem:scrub status for tag:67ef69a1-bb73-4f4c-ad3c-775745506056 - {'path': '/', 'tag': '67ef69a1-bb73-4f4c-ad3c-775745506056', 'options': 'recursive,force'} 2024-03-04T07:46:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:40 smithi133 ceph-mon[36505]: pgmap v578: 97 pgs: 97 active+clean; 4.3 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:40 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:46:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:40 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:40 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:46:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:40 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:40 smithi116 ceph-mon[32150]: pgmap v578: 97 pgs: 97 active+clean; 4.3 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:40 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:46:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:40 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:40 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:46:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:40 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:40 smithi152 ceph-mon[37154]: pgmap v578: 97 pgs: 97 active+clean; 4.3 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:46:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:40 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:46:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:40 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:40 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:46:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:40 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:46:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 -- 172.21.15.116:0/131213505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd74101600 msgr2=0x7fbd74101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 --2- 172.21.15.116:0/131213505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd74101600 0x7fbd74101a20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009900 tx=0x7fbd6402bc40 comp rx=0 tx=0).stop 2024-03-04T07:46:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 -- 172.21.15.116:0/131213505 shutdown_connections 2024-03-04T07:46:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 --2- 172.21.15.116:0/131213505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd741031b0 0x7fbd74109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 --2- 172.21.15.116:0/131213505 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbd741027f0 0x7fbd74102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 --2- 172.21.15.116:0/131213505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd74101600 0x7fbd74101a20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 -- 172.21.15.116:0/131213505 >> 172.21.15.116:0/131213505 conn(0x7fbd740fcb90 msgr2=0x7fbd740fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 -- 172.21.15.116:0/131213505 shutdown_connections 2024-03-04T07:46:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.277+0000 7fbd7a200640 1 -- 172.21.15.116:0/131213505 wait complete. 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.278+0000 7fbd7a200640 1 Processor -- start 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.278+0000 7fbd7a200640 1 -- start start 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.278+0000 7fbd7a200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbd74101600 0x7fbd74075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.278+0000 7fbd7a200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd741027f0 0x7fbd740739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd7a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd741031b0 0x7fbd74073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd7a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7410d290 con 0x7fbd741031b0 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd7a200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7410d110 con 0x7fbd74101600 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd7a200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7410d410 con 0x7fbd741027f0 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd73400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbd74101600 0x7fbd74075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd72a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd741027f0 0x7fbd740739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd73400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbd74101600 0x7fbd74075380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36862/0 (socket says 172.21.15.116:36862) 2024-03-04T07:46:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd73400640 1 -- 172.21.15.116:0/1978727317 learned_addr learned my addr 172.21.15.116:0/1978727317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd741031b0 0x7fbd74073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd72a00640 1 -- 172.21.15.116:0/1978727317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbd74101600 msgr2=0x7fbd74075380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd72a00640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbd74101600 0x7fbd74075380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd72a00640 1 -- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd741031b0 msgr2=0x7fbd74073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd72a00640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd741031b0 0x7fbd74073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.279+0000 7fbd72a00640 1 -- 172.21.15.116:0/1978727317 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd74074790 con 0x7fbd741027f0 2024-03-04T07:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.280+0000 7fbd72a00640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd741027f0 0x7fbd740739c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000e9f0 tx=0x7fbd6000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.280+0000 7fbd6be00640 1 -- 172.21.15.116:0/1978727317 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6000ccc0 con 0x7fbd741027f0 2024-03-04T07:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.280+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd741a1990 con 0x7fbd741027f0 2024-03-04T07:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.280+0000 7fbd6be00640 1 -- 172.21.15.116:0/1978727317 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd60005020 con 0x7fbd741027f0 2024-03-04T07:46:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.280+0000 7fbd6be00640 1 -- 172.21.15.116:0/1978727317 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd600049e0 con 0x7fbd741027f0 2024-03-04T07:46:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.280+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd741a1e90 con 0x7fbd741027f0 2024-03-04T07:46:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.283+0000 7fbd6be00640 1 -- 172.21.15.116:0/1978727317 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fbd60005560 con 0x7fbd741027f0 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.283+0000 7fbd7ce73740 1 Processor -- start 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.283+0000 7fbd7ce73740 1 -- start start 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x56193353cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933541d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933544840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619334a2370 con 0x56193353d240 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193344b550 con 0x5619335423b0 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193349e410 con 0x56193353c850 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x56193353cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd72a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933544840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933541d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x56193353cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58214/0 (socket says 172.21.15.116:58214) 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 -- 172.21.15.116:0/4183314916 learned_addr learned my addr 172.21.15.116:0/4183314916 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 -- 172.21.15.116:0/4183314916 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 msgr2=0x561933544840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 --2- 172.21.15.116:0/4183314916 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933544840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 -- 172.21.15.116:0/4183314916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 msgr2=0x561933541d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 --2- 172.21.15.116:0/4183314916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933541d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.284+0000 7fbd73e00640 1 -- 172.21.15.116:0/4183314916 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56193345f3c0 con 0x56193353c850 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.285+0000 7fbd73e00640 1 --2- 172.21.15.116:0/4183314916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x56193353cc70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009fd0 tx=0x7fbd6c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4db2174e1768fcd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.285+0000 7fbd6be00640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbd580a1e20 0x7fbd580a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.285+0000 7fbd5fe00640 1 -- 172.21.15.116:0/4183314916 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c004d20 con 0x56193353c850 2024-03-04T07:46:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.285+0000 7fbd5fe00640 1 -- 172.21.15.116:0/4183314916 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6c004050 con 0x56193353c850 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.285+0000 7fbd6be00640 1 -- 172.21.15.116:0/1978727317 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fbd600cdd70 con 0x7fbd741027f0 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.285+0000 7fbd5fe00640 1 -- 172.21.15.116:0/4183314916 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c005260 con 0x56193353c850 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.285+0000 7fbd73400640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbd580a1e20 0x7fbd580a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 -- 172.21.15.116:0/4183314916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 msgr2=0x56193353cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/4183314916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x56193353cc70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009fd0 tx=0x7fbd6c0118d0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd73400640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbd580a1e20 0x7fbd580a42e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbd6402c8f0 tx=0x7fbd6402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 -- 172.21.15.116:0/4183314916 shutdown_connections 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/4183314916 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933544840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/4183314916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933541d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/4183314916 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x56193353cc70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 -- 172.21.15.116:0/4183314916 >> 172.21.15.116:0/4183314916 conn(0x561933537dc0 msgr2=0x5619335381c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 -- 172.21.15.116:0/4183314916 shutdown_connections 2024-03-04T07:46:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.286+0000 7fbd7ce73740 1 -- 172.21.15.116:0/4183314916 wait complete. 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 -- start start 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 Processor -- start 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933547160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x56193354e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561933545360 con 0x56193353d240 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193345e9b0 con 0x5619335423b0 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619335454e0 con 0x56193353c850 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd7ce73740 10 client.0 _ll_register_callbacks cb 0x56193349a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd73400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd73400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58230/0 (socket says 172.21.15.116:58230) 2024-03-04T07:46:41.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.287+0000 7fbd73400640 1 -- 172.21.15.116:0/1890224763 learned_addr learned my addr 172.21.15.116:0/1890224763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x56193354e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd72a00640 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933547160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd73e00640 1 -- 172.21.15.116:0/1890224763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 msgr2=0x561933546c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd73e00640 1 -- 172.21.15.116:0/1890224763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 msgr2=0x561933547160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933547160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd73e00640 1 -- 172.21.15.116:0/1890224763 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619331b7210 con 0x5619335423b0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x56193354e130 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0099b0 tx=0x7fbd6c004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7df74b9ff55b590 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.288+0000 7fbd5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1890224763 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c028020 con 0x5619335423b0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1890224763 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fbd6c004410 con 0x5619335423b0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd6c004050 con 0x5619335423b0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1890224763 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd6c004650 con 0x5619335423b0 2024-03-04T07:46:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 10 client.26852 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561933283620 con 0x5619335423b0 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1890224763 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6c004050 con 0x5619335423b0 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1890224763 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fbd6c00f500 con 0x5619335423b0 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 10 client.26852 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 10 client.26852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 20 client.26852 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335527a0 0x561933554b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 4 client.26852 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.289+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619334af8b0 con 0x5619335527a0 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.290+0000 7fbd73400640 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335527a0 0x561933554b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.290+0000 7fbd73400640 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335527a0 0x561933554b90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8b54d2daaeaaa8c server_cookie=662e701ecccdb55f in_seq=0 out_seq=0 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.290+0000 7fbd5cc00640 10 client.26852 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1890224763 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5619334af8b0 con 0x5619335527a0 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd5cc00640 10 client.26852 handle_command_reply: tid=0 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 client.26852 shutdown 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 msgr2=0x56193354e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x56193354e130 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0099b0 tx=0x7fbd6c004cf0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 shutdown_connections 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335527a0 0x561933554b90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x56193354e130 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x561933547160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd5cc00640 0 client.26852 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1890224763 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 >> 172.21.15.116:0/1890224763 conn(0x561933537dc0 msgr2=0x561933536fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 shutdown_connections 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.291+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1890224763 wait complete. 2024-03-04T07:46:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.292+0000 7fbd7ce73740 20 client.26852 trim_cache size 0 max 16384 2024-03-04T07:46:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.306+0000 7fbd7ce73740 1 Processor -- start 2024-03-04T07:46:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.306+0000 7fbd7ce73740 1 -- start start 2024-03-04T07:46:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x56193354e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933547160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741a8c00 con 0x56193353d240 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741ab1e0 con 0x5619335423b0 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741a9200 con 0x56193353c850 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933547160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933547160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36892/0 (socket says 172.21.15.116:36892) 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:58240/0 (socket says 172.21.15.116:58240) 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 -- 172.21.15.116:0/3294268680 learned_addr learned my addr 172.21.15.116:0/3294268680 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd72a00640 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x56193354e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 -- 172.21.15.116:0/3294268680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 msgr2=0x561933546c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 -- 172.21.15.116:0/3294268680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 msgr2=0x56193354e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x56193354e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73e00640 1 -- 172.21.15.116:0/3294268680 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561933538b30 con 0x5619335423b0 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd72a00640 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x56193354e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.307+0000 7fbd73400640 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.308+0000 7fbd73e00640 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933547160 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c031040 tx=0x7fbd6c005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cb3bab84481c885 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.308+0000 7fbd5f400640 1 -- 172.21.15.116:0/3294268680 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c028040 con 0x5619335423b0 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.308+0000 7fbd5f400640 1 -- 172.21.15.116:0/3294268680 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6c004410 con 0x5619335423b0 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.308+0000 7fbd5f400640 1 -- 172.21.15.116:0/3294268680 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c019890 con 0x5619335423b0 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 -- 172.21.15.116:0/3294268680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 msgr2=0x561933547160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933547160 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c031040 tx=0x7fbd6c005f10 comp rx=0 tx=0).stop 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 -- 172.21.15.116:0/3294268680 shutdown_connections 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5619335423b0 0x561933547160 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353d240 0x56193354e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/3294268680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193353c850 0x561933546c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 -- 172.21.15.116:0/3294268680 >> 172.21.15.116:0/3294268680 conn(0x561933537dc0 msgr2=0x56193353e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 -- 172.21.15.116:0/3294268680 shutdown_connections 2024-03-04T07:46:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 -- 172.21.15.116:0/3294268680 wait complete. 2024-03-04T07:46:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 -- start start 2024-03-04T07:46:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.309+0000 7fbd7ce73740 1 Processor -- start 2024-03-04T07:46:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353c850 0x56193354b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193353d240 0x56193354be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd7ce73740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5619335423b0 0x56193354ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741a9860 con 0x56193353c850 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741aab80 con 0x56193353d240 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd7ce73740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd741a9ec0 con 0x5619335423b0 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd7ce73740 10 client.0 _ll_register_callbacks cb 0x7fbd60005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd72a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193353d240 0x56193354be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd73400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353c850 0x56193354b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd73e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5619335423b0 0x56193354ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd72a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193353d240 0x56193354be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:36902/0 (socket says 172.21.15.116:36902) 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.310+0000 7fbd72a00640 1 -- 172.21.15.116:0/1653417032 learned_addr learned my addr 172.21.15.116:0/1653417032 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd73e00640 1 -- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193353d240 msgr2=0x56193354be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193353d240 0x56193354be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd73e00640 1 -- 172.21.15.116:0/1653417032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353c850 msgr2=0x56193354b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353c850 0x56193354b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd73e00640 1 -- 172.21.15.116:0/1653417032 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56193354d420 con 0x5619335423b0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd72a00640 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193353d240 0x56193354be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd73400640 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353c850 0x56193354b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd73e00640 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5619335423b0 0x56193354ce30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00f9a0 tx=0x7fbd6c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1417fa6c1413fd7b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1653417032 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c028040 con 0x5619335423b0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd6c004410 con 0x5619335423b0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1653417032 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fbd6c019610 con 0x5619335423b0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1653417032 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd6c019850 con 0x5619335423b0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd7ce73740 10 client.26758 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.311+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5619335ae0f0 con 0x5619335423b0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1653417032 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6c004410 con 0x5619335423b0 2024-03-04T07:46:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1653417032 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fbd6c036020 con 0x5619335423b0 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd7ce73740 10 client.26758 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd7ce73740 10 client.26758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd7ce73740 20 client.26758 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335aea80 0x5619335b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd7ce73740 4 client.26758 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5619334a9c00 con 0x5619335aea80 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.312+0000 7fbd73400640 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335aea80 0x5619335b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.313+0000 7fbd73400640 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335aea80 0x5619335b0e70 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de747cd8e9929666 server_cookie=657fac6f2c265571 in_seq=0 out_seq=0 2024-03-04T07:46:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.313+0000 7fbd5cc00640 10 client.26758 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:41.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.328+0000 7fbd5cc00640 1 -- 172.21.15.116:0/1653417032 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5619334a9c00 con 0x5619335aea80 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.328+0000 7fbd5cc00640 10 client.26758 handle_command_reply: tid=0 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.328+0000 7fbd7ce73740 1 client.26758 shutdown 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5619335423b0 msgr2=0x56193354ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5619335423b0 0x56193354ce30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00f9a0 tx=0x7fbd6c005e80 comp rx=0 tx=0).stop 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 shutdown_connections 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5619335aea80 0x5619335b0e70 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5619335423b0 0x56193354ce30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd5cc00640 0 client.26758 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193353d240 0x56193354be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 --2- 172.21.15.116:0/1653417032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193353c850 0x56193354b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 >> 172.21.15.116:0/1653417032 conn(0x561933537dc0 msgr2=0x561933546af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 shutdown_connections 2024-03-04T07:46:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 1 -- 172.21.15.116:0/1653417032 wait complete. 2024-03-04T07:46:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.329+0000 7fbd7ce73740 20 client.26758 trim_cache size 0 max 16384 2024-03-04T07:46:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:46:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:46:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:46:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbd580a1e20 msgr2=0x7fbd580a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbd580a1e20 0x7fbd580a42e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbd6402c8f0 tx=0x7fbd6402e040 comp rx=0 tx=0).stop 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd741027f0 msgr2=0x7fbd740739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd741027f0 0x7fbd740739c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbd6000e9f0 tx=0x7fbd6000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 shutdown_connections 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd741031b0 0x7fbd74073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbd580a1e20 0x7fbd580a42e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd741027f0 0x7fbd740739c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 --2- 172.21.15.116:0/1978727317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbd74101600 0x7fbd74075380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 >> 172.21.15.116:0/1978727317 conn(0x7fbd740fcb90 msgr2=0x7fbd74105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 shutdown_connections 2024-03-04T07:46:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:41.330+0000 7fbd7a200640 1 -- 172.21.15.116:0/1978727317 wait complete. 2024-03-04T07:46:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:46:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:41 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:46:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:41 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:46:42.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:42.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:41 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi133 to 174.1M 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: Unable to set osd_memory_target on smithi133 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: pgmap v579: 97 pgs: 97 active+clean; 4.3 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:46:43.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:46:43.088 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi133 to 174.1M 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: Unable to set osd_memory_target on smithi133 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: pgmap v579: 97 pgs: 97 active+clean; 4.3 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:46:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:46:43.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi133 to 174.1M 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: Unable to set osd_memory_target on smithi133 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: pgmap v579: 97 pgs: 97 active+clean; 4.3 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:46:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:46:43.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:46:43.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:46:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:43 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi116 to 176.7M 2024-03-04T07:46:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:43 smithi133 ceph-mon[36505]: Unable to set osd_memory_target on smithi116 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-04T07:46:44.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:43 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:46:44.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:43 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi116 to 176.7M 2024-03-04T07:46:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:43 smithi116 ceph-mon[32150]: Unable to set osd_memory_target on smithi116 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-04T07:46:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:43 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:46:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:43 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi116 to 176.7M 2024-03-04T07:46:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:43 smithi152 ceph-mon[37154]: Unable to set osd_memory_target on smithi116 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-04T07:46:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:43 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:46:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:44 smithi116 ceph-mon[32150]: pgmap v580: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-04T07:46:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:44 smithi152 ceph-mon[37154]: pgmap v580: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-04T07:46:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:44 smithi133 ceph-mon[36505]: pgmap v580: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-04T07:46:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:46 smithi116 ceph-mon[32150]: pgmap v581: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:46 smithi152 ceph-mon[37154]: pgmap v581: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:46 smithi133 ceph-mon[36505]: pgmap v581: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:48 smithi116 ceph-mon[32150]: pgmap v582: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:46:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:48 smithi152 ceph-mon[37154]: pgmap v582: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:46:49.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:48 smithi133 ceph-mon[36505]: pgmap v582: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:46:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:50 smithi116 ceph-mon[32150]: pgmap v583: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:50 smithi152 ceph-mon[37154]: pgmap v583: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:51.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:50 smithi133 ceph-mon[36505]: pgmap v583: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:52 smithi116 ceph-mon[32150]: pgmap v584: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:52 smithi152 ceph-mon[37154]: pgmap v584: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:52 smithi133 ceph-mon[36505]: pgmap v584: 97 pgs: 97 active+clean; 4.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:46:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 -- 172.21.15.116:0/3053629590 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea41097e0 msgr2=0x7f2ea4110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 --2- 172.21.15.116:0/3053629590 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea41097e0 0x7f2ea4110070 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009900 tx=0x7f2e9c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:46:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 -- 172.21.15.116:0/3053629590 shutdown_connections 2024-03-04T07:46:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 --2- 172.21.15.116:0/3053629590 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea41097e0 0x7f2ea4110070 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 --2- 172.21.15.116:0/3053629590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea4108e20 0x7f2ea41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 --2- 172.21.15.116:0/3053629590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ea4107c30 0x7f2ea4108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 -- 172.21.15.116:0/3053629590 >> 172.21.15.116:0/3053629590 conn(0x7f2ea4076be0 msgr2=0x7f2ea4079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 -- 172.21.15.116:0/3053629590 shutdown_connections 2024-03-04T07:46:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.473+0000 7f2eac000640 1 -- 172.21.15.116:0/3053629590 wait complete. 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 Processor -- start 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 -- start start 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea4107c30 0x7f2ea419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea4108e20 0x7f2ea419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ea41097e0 0x7f2ea41a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4113890 con 0x7f2ea4108e20 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4113710 con 0x7f2ea4107c30 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2eac000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4113a10 con 0x7f2ea41097e0 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2ea9600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea4107c30 0x7f2ea419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.474+0000 7f2ea8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea4108e20 0x7f2ea419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea4107c30 0x7f2ea419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55356/0 (socket says 172.21.15.116:55356) 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 -- 172.21.15.116:0/4090051562 learned_addr learned my addr 172.21.15.116:0/4090051562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:54.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2eaa000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ea41097e0 0x7f2ea41a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 -- 172.21.15.116:0/4090051562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ea41097e0 msgr2=0x7f2ea41a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ea41097e0 0x7f2ea41a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 -- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea4108e20 msgr2=0x7f2ea419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea4108e20 0x7f2ea419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 -- 172.21.15.116:0/4090051562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea41a5a60 con 0x7f2ea4107c30 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea9600640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea4107c30 0x7f2ea419de40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800ea80 tx=0x7f2e9800ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea2000640 1 -- 172.21.15.116:0/4090051562 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9800ce40 con 0x7f2ea4107c30 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea2000640 1 -- 172.21.15.116:0/4090051562 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e98011950 con 0x7f2ea4107c30 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2ea2000640 1 -- 172.21.15.116:0/4090051562 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9801ab80 con 0x7f2ea4107c30 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.475+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea41a5d50 con 0x7f2ea4107c30 2024-03-04T07:46:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.476+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea41a6210 con 0x7f2ea4107c30 2024-03-04T07:46:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 Processor -- start 2024-03-04T07:46:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2ea2000640 1 -- 172.21.15.116:0/4090051562 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2e98011b00 con 0x7f2ea4107c30 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 -- start start 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dc2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dc7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2f3d28370 con 0x55e2f3dc83b0 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2f3cd1550 con 0x55e2f3dc2850 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2f3d24410 con 0x55e2f3dc3240 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2ea9600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dc7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2ea8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaa000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dc2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaa000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dc2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55370/0 (socket says 172.21.15.116:55370) 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2ea9600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dc7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:48438/0 (socket says 172.21.15.116:48438) 2024-03-04T07:46:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.479+0000 7f2eaa000640 1 -- 172.21.15.116:0/166094802 learned_addr learned my addr 172.21.15.116:0/166094802 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2eaa000640 1 -- 172.21.15.116:0/166094802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 msgr2=0x55e2f3dc7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2eaa000640 1 --2- 172.21.15.116:0/166094802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dc7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2eaa000640 1 -- 172.21.15.116:0/166094802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 msgr2=0x55e2f3dca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2eaa000640 1 --2- 172.21.15.116:0/166094802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2eaa000640 1 -- 172.21.15.116:0/166094802 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2f3ce53c0 con 0x55e2f3dc2850 2024-03-04T07:46:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2eaa000640 1 --2- 172.21.15.116:0/166094802 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dc2c70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009230 tx=0x7f2e9c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8014bed8a3aa4301 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2ea2000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2e8c0a1ea0 0x7f2e8c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2e96600640 1 -- 172.21.15.116:0/166094802 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c005d50 con 0x55e2f3dc2850 2024-03-04T07:46:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2e96600640 1 -- 172.21.15.116:0/166094802 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9c02cb90 con 0x55e2f3dc2850 2024-03-04T07:46:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.480+0000 7f2e96600640 1 -- 172.21.15.116:0/166094802 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c005650 con 0x55e2f3dc2850 2024-03-04T07:46:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2ea2000640 1 -- 172.21.15.116:0/4090051562 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2e980ce3a0 con 0x7f2ea4107c30 2024-03-04T07:46:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2ea9600640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2e8c0a1ea0 0x7f2e8c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 -- 172.21.15.116:0/166094802 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 msgr2=0x55e2f3dc2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/166094802 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dc2c70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009230 tx=0x7f2e9c02c660 comp rx=0 tx=0).stop 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2ea9600640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2e8c0a1ea0 0x7f2e8c0a4360 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x55e2f3dc3640 tx=0x7f2e980d1de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 -- 172.21.15.116:0/166094802 shutdown_connections 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/166094802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/166094802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dc7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/166094802 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dc2c70 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 -- 172.21.15.116:0/166094802 >> 172.21.15.116:0/166094802 conn(0x55e2f3dbddc0 msgr2=0x55e2f3dbe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 -- 172.21.15.116:0/166094802 shutdown_connections 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.481+0000 7f2eaedfc740 1 -- 172.21.15.116:0/166094802 wait complete. 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.482+0000 7f2eaedfc740 1 -- start start 2024-03-04T07:46:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.482+0000 7f2eaedfc740 1 Processor -- start 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc2850 0x55e2f3dccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dcd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2f3dcb300 con 0x55e2f3dc2850 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2f3ce49b0 con 0x55e2f3dc83b0 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2f3dcb480 con 0x55e2f3dc3240 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaedfc740 10 client.0 _ll_register_callbacks cb 0x55e2f3d20340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc2850 0x55e2f3dccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc2850 0x55e2f3dccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42582/0 (socket says 172.21.15.116:42582) 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 -- 172.21.15.116:0/3392342017 learned_addr learned my addr 172.21.15.116:0/3392342017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dcd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 -- 172.21.15.116:0/3392342017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 msgr2=0x55e2f3dcd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dcd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 -- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc83b0 msgr2=0x55e2f3dd4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.483+0000 7f2eaa000640 1 -- 172.21.15.116:0/3392342017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e2f3a3d210 con 0x55e2f3dc2850 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2eaa000640 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc2850 0x55e2f3dccb80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c02c7c0 tx=0x7f2e9c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4d9e7bba8a070d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2e8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2e8aa00640 1 -- 172.21.15.116:0/3392342017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c003800 con 0x55e2f3dc2850 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2e8aa00640 1 -- 172.21.15.116:0/3392342017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2e9c0039a0 con 0x55e2f3dc2850 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e9c02cb90 con 0x55e2f3dc2850 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2e8aa00640 1 -- 172.21.15.116:0/3392342017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e9c05a050 con 0x55e2f3dc2850 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2eaedfc740 10 client.16743 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e2f3b09620 con 0x55e2f3dc2850 2024-03-04T07:46:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.484+0000 7f2e8aa00640 1 -- 172.21.15.116:0/3392342017 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9c02cb90 con 0x55e2f3dc2850 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2e8aa00640 1 -- 172.21.15.116:0/3392342017 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2e9c004d60 con 0x55e2f3dc2850 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2eaedfc740 10 client.16743 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2eaedfc740 10 client.16743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2eaedfc740 20 client.16743 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3dd85f0 0x55e2f3dda9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2eaedfc740 4 client.16743 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e2f3d358b0 con 0x55e2f3dd85f0 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.485+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3dd85f0 0x55e2f3dda9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.486+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3dd85f0 0x55e2f3dda9e0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2e77084a57d1880 server_cookie=32e955221e03b959 in_seq=0 out_seq=0 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.486+0000 7f2e8aa00640 10 client.16743 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.486+0000 7f2e8aa00640 1 -- 172.21.15.116:0/3392342017 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e2f3d358b0 con 0x55e2f3dd85f0 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.486+0000 7f2e8aa00640 10 client.16743 handle_command_reply: tid=0 2024-03-04T07:46:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.486+0000 7f2eaedfc740 1 client.16743 shutdown 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc2850 msgr2=0x55e2f3dccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc2850 0x55e2f3dccb80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c02c7c0 tx=0x7f2e9c009260 comp rx=0 tx=0).stop 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 shutdown_connections 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3dd85f0 0x55e2f3dda9e0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc3240 0x55e2f3dcd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3392342017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc2850 0x55e2f3dccb80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2e8aa00640 0 client.16743 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 >> 172.21.15.116:0/3392342017 conn(0x55e2f3dbddc0 msgr2=0x55e2f3dbcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 shutdown_connections 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3392342017 wait complete. 2024-03-04T07:46:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.487+0000 7f2eaedfc740 20 client.16743 trim_cache size 0 max 16384 2024-03-04T07:46:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 Processor -- start 2024-03-04T07:46:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 -- start start 2024-03-04T07:46:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dcc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dcccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea41ad060 con 0x55e2f3dc3240 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea41af640 con 0x55e2f3dc2850 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea41ad660 con 0x55e2f3dc83b0 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2ea8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2ea8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42598/0 (socket says 172.21.15.116:42598) 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2ea8c00640 1 -- 172.21.15.116:0/3712167298 learned_addr learned my addr 172.21.15.116:0/3712167298 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.502+0000 7f2eaa000640 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dcc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2eaa000640 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dcccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2ea8c00640 1 -- 172.21.15.116:0/3712167298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc83b0 msgr2=0x55e2f3dcccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dcccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2ea8c00640 1 -- 172.21.15.116:0/3712167298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 msgr2=0x55e2f3dcc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dcc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2ea8c00640 1 -- 172.21.15.116:0/3712167298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2f3db8e40 con 0x55e2f3dc3240 2024-03-04T07:46:54.504 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2eaa000640 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dcccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:46:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd4090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009e50 tx=0x7f2e90005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2324939160168242 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2e95c00640 1 -- 172.21.15.116:0/3712167298 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e900202f0 con 0x55e2f3dc3240 2024-03-04T07:46:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2e95c00640 1 -- 172.21.15.116:0/3712167298 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e90020c50 con 0x55e2f3dc3240 2024-03-04T07:46:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.503+0000 7f2e95c00640 1 -- 172.21.15.116:0/3712167298 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9001edf0 con 0x55e2f3dc3240 2024-03-04T07:46:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3712167298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc3240 msgr2=0x55e2f3dd4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd4090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009e50 tx=0x7f2e90005f90 comp rx=0 tx=0).stop 2024-03-04T07:46:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3712167298 shutdown_connections 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dcccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd4090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/3712167298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc2850 0x55e2f3dcc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3712167298 >> 172.21.15.116:0/3712167298 conn(0x55e2f3dbddc0 msgr2=0x55e2f3dbe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3712167298 shutdown_connections 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 -- 172.21.15.116:0/3712167298 wait complete. 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.504+0000 7f2eaedfc740 1 -- start start 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 1 Processor -- start 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc2850 0x55e2f3dd3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea41adcc0 con 0x55e2f3dc83b0 2024-03-04T07:46:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea41aefe0 con 0x55e2f3dc3240 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea41ae320 con 0x55e2f3dc2850 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaedfc740 10 client.0 _ll_register_callbacks cb 0x7f2e90009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2ea8c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2ea8c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55396/0 (socket says 172.21.15.116:55396) 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaa000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc2850 0x55e2f3dd3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2ea8c00640 1 -- 172.21.15.116:0/1960815985 learned_addr learned my addr 172.21.15.116:0/1960815985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.505+0000 7f2eaa000640 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2ea8c00640 1 -- 172.21.15.116:0/1960815985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc2850 msgr2=0x55e2f3dd3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc2850 0x55e2f3dd3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2ea8c00640 1 -- 172.21.15.116:0/1960815985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 msgr2=0x55e2f3dd22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2ea8c00640 1 -- 172.21.15.116:0/1960815985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e2f3dd2b20 con 0x55e2f3dc3240 2024-03-04T07:46:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2ea8c00640 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd40b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009540 tx=0x7f2e90020170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f5b801d43037907 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2e8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2e8aa00640 1 -- 172.21.15.116:0/1960815985 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9002d070 con 0x55e2f3dc3240 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e90020c50 con 0x55e2f3dc3240 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.506+0000 7f2e8aa00640 1 -- 172.21.15.116:0/1960815985 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2e9001ed80 con 0x55e2f3dc3240 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2e8aa00640 1 -- 172.21.15.116:0/1960815985 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e90039050 con 0x55e2f3dc3240 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 10 client.26876 fetch_fsmap learned FSMap version 19 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e2f3e34930 con 0x55e2f3dc3240 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2e8aa00640 1 -- 172.21.15.116:0/1960815985 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e90020c50 con 0x55e2f3dc3240 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2e8aa00640 1 -- 172.21.15.116:0/1960815985 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2e90004320 con 0x55e2f3dc3240 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 10 client.26876 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 10 client.26876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 20 client.26876 populate_metadata read hostname 'smithi116' 2024-03-04T07:46:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3e35210 0x55e2f3e37630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:46:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 4 client.26876 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:46:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e2f3d2fc00 con 0x55e2f3e35210 2024-03-04T07:46:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.507+0000 7f2eaa000640 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3e35210 0x55e2f3e37630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:46:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.508+0000 7f2eaa000640 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3e35210 0x55e2f3e37630 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79fe42cedfb10237 server_cookie=cf79bcd644d5bed0 in_seq=0 out_seq=0 2024-03-04T07:46:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.508+0000 7f2e8aa00640 10 client.26876 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2e8aa00640 1 -- 172.21.15.116:0/1960815985 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e2f3d2fc00 con 0x55e2f3e35210 2024-03-04T07:46:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2e8aa00640 10 client.26876 handle_command_reply: tid=0 2024-03-04T07:46:54.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 client.26876 shutdown 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc3240 msgr2=0x55e2f3dd40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd40b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009540 tx=0x7f2e90020170 comp rx=0 tx=0).stop 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 shutdown_connections 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2f3e35210 0x55e2f3e37630 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2f3dc83b0 0x55e2f3dd22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e2f3dc3240 0x55e2f3dd40b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 --2- 172.21.15.116:0/1960815985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e2f3dc2850 0x55e2f3dd3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 >> 172.21.15.116:0/1960815985 conn(0x55e2f3dbddc0 msgr2=0x55e2f3dcc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.524+0000 7f2e8aa00640 0 client.26876 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.525+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 shutdown_connections 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.525+0000 7f2eaedfc740 1 -- 172.21.15.116:0/1960815985 wait complete. 2024-03-04T07:46:54.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.525+0000 7f2eaedfc740 20 client.26876 trim_cache size 0 max 16384 2024-03-04T07:46:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:46:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:46:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:46:54.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2e8c0a1ea0 msgr2=0x7f2e8c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2e8c0a1ea0 0x7f2e8c0a4360 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x55e2f3dc3640 tx=0x7f2e980d1de0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea4107c30 msgr2=0x7f2ea419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea4107c30 0x7f2ea419de40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800ea80 tx=0x7f2e9800ef40 comp rx=0 tx=0).stop 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 shutdown_connections 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2e8c0a1ea0 0x7f2e8c0a4360 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ea41097e0 0x7f2ea41a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ea4108e20 0x7f2ea419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 --2- 172.21.15.116:0/4090051562 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea4107c30 0x7f2ea419de40 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:46:54.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 >> 172.21.15.116:0/4090051562 conn(0x7f2ea4076be0 msgr2=0x7f2ea410c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:46:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 shutdown_connections 2024-03-04T07:46:54.528 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:46:54.526+0000 7f2eac000640 1 -- 172.21.15.116:0/4090051562 wait complete. 2024-03-04T07:46:55.139 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:54 smithi133 ceph-mon[36505]: pgmap v585: 97 pgs: 97 active+clean; 4.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:46:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:54 smithi116 ceph-mon[32150]: pgmap v585: 97 pgs: 97 active+clean; 4.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:46:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:54 smithi152 ceph-mon[37154]: pgmap v585: 97 pgs: 97 active+clean; 4.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:46:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:56 smithi116 ceph-mon[32150]: pgmap v586: 97 pgs: 97 active+clean; 4.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:46:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:56 smithi152 ceph-mon[37154]: pgmap v586: 97 pgs: 97 active+clean; 4.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:46:57.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:56 smithi133 ceph-mon[36505]: pgmap v586: 97 pgs: 97 active+clean; 4.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:46:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:46:58 smithi116 ceph-mon[32150]: pgmap v587: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:46:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:46:58 smithi152 ceph-mon[37154]: pgmap v587: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:46:59.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:46:58 smithi133 ceph-mon[36505]: pgmap v587: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:47:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:00 smithi116 ceph-mon[32150]: pgmap v588: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:00 smithi152 ceph-mon[37154]: pgmap v588: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:00 smithi133 ceph-mon[36505]: pgmap v588: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:02 smithi116 ceph-mon[32150]: pgmap v589: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:02 smithi152 ceph-mon[37154]: pgmap v589: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:02 smithi133 ceph-mon[36505]: pgmap v589: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:04 smithi116 ceph-mon[32150]: pgmap v590: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:04 smithi152 ceph-mon[37154]: pgmap v590: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:05.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:04 smithi133 ceph-mon[36505]: pgmap v590: 97 pgs: 97 active+clean; 4.6 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:07.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:06 smithi116 ceph-mon[32150]: pgmap v591: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:06 smithi152 ceph-mon[37154]: pgmap v591: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:07.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:06 smithi133 ceph-mon[36505]: pgmap v591: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 KiB/s wr, 12 op/s 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.666+0000 7f1975e00640 1 -- 172.21.15.116:0/3710876665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970107c30 msgr2=0x7f1970108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.666+0000 7f1975e00640 1 --2- 172.21.15.116:0/3710876665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970107c30 0x7f1970108050 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1960009900 tx=0x7f196002bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.666+0000 7f1975e00640 1 -- 172.21.15.116:0/3710876665 shutdown_connections 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.666+0000 7f1975e00640 1 --2- 172.21.15.116:0/3710876665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19701097e0 0x7f1970110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.666+0000 7f1975e00640 1 --2- 172.21.15.116:0/3710876665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1970108e20 0x7f19701092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.666+0000 7f1975e00640 1 --2- 172.21.15.116:0/3710876665 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970107c30 0x7f1970108050 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.666+0000 7f1975e00640 1 -- 172.21.15.116:0/3710876665 >> 172.21.15.116:0/3710876665 conn(0x7f1970076be0 msgr2=0x7f1970079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.667+0000 7f1975e00640 1 -- 172.21.15.116:0/3710876665 shutdown_connections 2024-03-04T07:47:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.667+0000 7f1975e00640 1 -- 172.21.15.116:0/3710876665 wait complete. 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 Processor -- start 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 -- start start 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1970107c30 0x7f197019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970108e20 0x7f197019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19701097e0 0x7f19701a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701138d0 con 0x7f19701097e0 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1970113750 con 0x7f1970108e20 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.668+0000 7f1975e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1970113a50 con 0x7f1970107c30 2024-03-04T07:47:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970108e20 0x7f197019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f196f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19701097e0 0x7f19701a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970108e20 0x7f197019e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45768/0 (socket says 172.21.15.116:45768) 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 -- 172.21.15.116:0/3852839352 learned_addr learned my addr 172.21.15.116:0/3852839352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f196ea00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1970107c30 0x7f197019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 -- 172.21.15.116:0/3852839352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1970107c30 msgr2=0x7f197019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1970107c30 0x7f197019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 -- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19701097e0 msgr2=0x7f19701a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19701097e0 0x7f19701a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 -- 172.21.15.116:0/3852839352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19701a5aa0 con 0x7f1970108e20 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f196f400640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19701097e0 0x7f19701a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1967e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970108e20 0x7f197019e3c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f195800d950 tx=0x7f195800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.669+0000 7f1974a00640 1 -- 172.21.15.116:0/3852839352 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958017070 con 0x7f1970108e20 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.670+0000 7f1974a00640 1 -- 172.21.15.116:0/3852839352 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1958009c30 con 0x7f1970108e20 2024-03-04T07:47:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.670+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19701a5d90 con 0x7f1970108e20 2024-03-04T07:47:07.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.670+0000 7f1974a00640 1 -- 172.21.15.116:0/3852839352 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195800f6b0 con 0x7f1970108e20 2024-03-04T07:47:07.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.670+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19701a6280 con 0x7f1970108e20 2024-03-04T07:47:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 Processor -- start 2024-03-04T07:47:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1974a00640 1 -- 172.21.15.116:0/3852839352 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f19580109d0 con 0x7f1970108e20 2024-03-04T07:47:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 -- start start 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf86850 0x55633bf86c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf8bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf8c3b0 0x55633bf8e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633beec370 con 0x55633bf8c3b0 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633be95550 con 0x55633bf87240 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f1978c08740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633bee8410 con 0x55633bf86850 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.673+0000 7f196ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf8bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf8bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45784/0 (socket says 172.21.15.116:45784) 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 -- 172.21.15.116:0/4151416651 learned_addr learned my addr 172.21.15.116:0/4151416651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196f400640 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf86850 0x55633bf86c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f1967e00640 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf8c3b0 0x55633bf8e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 -- 172.21.15.116:0/4151416651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf86850 msgr2=0x55633bf86c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf86850 0x55633bf86c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 -- 172.21.15.116:0/4151416651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf8c3b0 msgr2=0x55633bf8e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf8c3b0 0x55633bf8e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 -- 172.21.15.116:0/4151416651 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55633bea93c0 con 0x55633bf87240 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196f400640 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf86850 0x55633bf86c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f196ea00640 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf8bd40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1960000c00 tx=0x7f19600095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da2803bc097db89b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f1974a00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19540a1e20 0x7f19540a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.674+0000 7f195fe00640 1 -- 172.21.15.116:0/4151416651 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196002c980 con 0x55633bf87240 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f195fe00640 1 -- 172.21.15.116:0/4151416651 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f196002cb20 con 0x55633bf87240 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f195fe00640 1 -- 172.21.15.116:0/4151416651 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960004920 con 0x55633bf87240 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1974a00640 1 -- 172.21.15.116:0/3852839352 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f19580cdc80 con 0x7f1970108e20 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f196f400640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19540a1e20 0x7f19540a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1978c08740 1 -- 172.21.15.116:0/4151416651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 msgr2=0x55633bf8bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1978c08740 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf8bd40 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1960000c00 tx=0x7f19600095e0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f196f400640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19540a1e20 0x7f19540a42e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55633bf87aa0 tx=0x7f1968009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1978c08740 1 -- 172.21.15.116:0/4151416651 shutdown_connections 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1978c08740 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf8c3b0 0x55633bf8e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1978c08740 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf8bd40 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1978c08740 1 --2- 172.21.15.116:0/4151416651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf86850 0x55633bf86c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.675+0000 7f1978c08740 1 -- 172.21.15.116:0/4151416651 >> 172.21.15.116:0/4151416651 conn(0x55633bf81dc0 msgr2=0x55633bf821c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.676+0000 7f1978c08740 1 -- 172.21.15.116:0/4151416651 shutdown_connections 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.676+0000 7f1978c08740 1 -- 172.21.15.116:0/4151416651 wait complete. 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.676+0000 7f1978c08740 1 -- start start 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.676+0000 7f1978c08740 1 Processor -- start 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1978c08740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf90ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1978c08740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf91010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1978c08740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf98050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1978c08740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633bf8f300 con 0x55633bf86850 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1978c08740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633bea89b0 con 0x55633bf87240 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1978c08740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633bf8f480 con 0x55633bf8c3b0 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1978c08740 10 client.0 _ll_register_callbacks cb 0x55633bee4340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f196ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf90ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf91010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf91010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45786/0 (socket says 172.21.15.116:45786) 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f196ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf90ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50520/0 (socket says 172.21.15.116:50520) 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 -- 172.21.15.116:0/3936385352 learned_addr learned my addr 172.21.15.116:0/3936385352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f196f400640 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf98050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 -- 172.21.15.116:0/3936385352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 msgr2=0x55633bf98050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf98050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.683 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 -- 172.21.15.116:0/3936385352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 msgr2=0x55633bf90ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf90ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f1967e00640 1 -- 172.21.15.116:0/3936385352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55633bc01210 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.677+0000 7f196ea00640 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf90ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f1967e00640 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf91010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1958096ec0 tx=0x7f195809a5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7980815a61587889 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f195cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f195cc00640 1 -- 172.21.15.116:0/3936385352 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958020070 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f195cc00640 1 -- 172.21.15.116:0/3936385352 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f19580a2cc0 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f196002cb20 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f195cc00640 1 -- 172.21.15.116:0/3936385352 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f195809aa60 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f1978c08740 10 client.26888 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55633bccd620 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f195cc00640 1 -- 172.21.15.116:0/3936385352 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f196002cb20 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.678+0000 7f195cc00640 1 -- 172.21.15.116:0/3936385352 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f195809bce0 con 0x55633bf87240 2024-03-04T07:47:07.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f1978c08740 10 client.26888 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f1978c08740 10 client.26888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f1978c08740 20 client.26888 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f1978c08740 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bf9c860 0x55633bf9ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f1978c08740 4 client.26888 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55633bef98b0 con 0x55633bf9c860 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f196ea00640 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bf9c860 0x55633bf9ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f196ea00640 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bf9c860 0x55633bf9ec90 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64bfb858bac242ab server_cookie=56c215e9e552733b in_seq=0 out_seq=0 2024-03-04T07:47:07.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.679+0000 7f195cc00640 10 client.26888 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f195cc00640 1 -- 172.21.15.116:0/3936385352 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55633bef98b0 con 0x55633bf9c860 2024-03-04T07:47:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f195cc00640 10 client.26888 handle_command_reply: tid=0 2024-03-04T07:47:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 client.26888 shutdown 2024-03-04T07:47:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 msgr2=0x55633bf91010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf91010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1958096ec0 tx=0x7f195809a5a0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 shutdown_connections 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bf9c860 0x55633bf9ec90 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf98050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf91010 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f195cc00640 0 client.26888 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 --2- 172.21.15.116:0/3936385352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf90ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.680+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 >> 172.21.15.116:0/3936385352 conn(0x55633bf81dc0 msgr2=0x55633bf80fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.681+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 shutdown_connections 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.681+0000 7f1978c08740 1 -- 172.21.15.116:0/3936385352 wait complete. 2024-03-04T07:47:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.681+0000 7f1978c08740 20 client.26888 trim_cache size 0 max 16384 2024-03-04T07:47:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.695+0000 7f1978c08740 1 Processor -- start 2024-03-04T07:47:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.695+0000 7f1978c08740 1 -- start start 2024-03-04T07:47:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1978c08740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf86850 0x55633bf80750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1978c08740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf87240 0x55633bf98010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1978c08740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf80c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1978c08740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701acee0 con 0x55633bf87240 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1978c08740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701af4c0 con 0x55633bf86850 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1978c08740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701ad4e0 con 0x55633bf8c3b0 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf86850 0x55633bf80750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf86850 0x55633bf80750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45794/0 (socket says 172.21.15.116:45794) 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 -- 172.21.15.116:0/3443049315 learned_addr learned my addr 172.21.15.116:0/3443049315 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1967e00640 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf87240 0x55633bf98010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196f400640 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf80c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 -- 172.21.15.116:0/3443049315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 msgr2=0x55633bf80c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf80c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 -- 172.21.15.116:0/3443049315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf87240 msgr2=0x55633bf98010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf87240 0x55633bf98010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f196ea00640 1 -- 172.21.15.116:0/3443049315 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55633bf81710 con 0x55633bf86850 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.696+0000 7f1967e00640 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf87240 0x55633bf98010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.697+0000 7f196ea00640 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf86850 0x55633bf80750 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f196002c710 tx=0x7f196002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94f6da97e1395c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.697+0000 7f195f400640 1 -- 172.21.15.116:0/3443049315 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960004350 con 0x55633bf86850 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.697+0000 7f195f400640 1 -- 172.21.15.116:0/3443049315 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1960005520 con 0x55633bf86850 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.697+0000 7f195f400640 1 -- 172.21.15.116:0/3443049315 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960004e00 con 0x55633bf86850 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.697+0000 7f1978c08740 1 -- 172.21.15.116:0/3443049315 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf86850 msgr2=0x55633bf80750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.697+0000 7f1978c08740 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf86850 0x55633bf80750 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f196002c710 tx=0x7f196002bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.697+0000 7f1978c08740 1 -- 172.21.15.116:0/3443049315 shutdown_connections 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf80c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf87240 0x55633bf98010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 --2- 172.21.15.116:0/3443049315 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf86850 0x55633bf80750 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 -- 172.21.15.116:0/3443049315 >> 172.21.15.116:0/3443049315 conn(0x55633bf81dc0 msgr2=0x55633bf821a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 -- 172.21.15.116:0/3443049315 shutdown_connections 2024-03-04T07:47:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 -- 172.21.15.116:0/3443049315 wait complete. 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 -- start start 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.698+0000 7f1978c08740 1 Processor -- start 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1978c08740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf95de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1978c08740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf96320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1978c08740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf972e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1978c08740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701adb40 con 0x55633bf86850 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1978c08740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701aee60 con 0x55633bf87240 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1978c08740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19701ae1a0 con 0x55633bf8c3b0 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1978c08740 10 client.0 _ll_register_callbacks cb 0x7f1958096d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f196ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf95de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1967e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf96320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f196ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf95de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50534/0 (socket says 172.21.15.116:50534) 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f196ea00640 1 -- 172.21.15.116:0/3612722285 learned_addr learned my addr 172.21.15.116:0/3612722285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f196f400640 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1967e00640 1 -- 172.21.15.116:0/3612722285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 msgr2=0x55633bf972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1967e00640 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1967e00640 1 -- 172.21.15.116:0/3612722285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 msgr2=0x55633bf95de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1967e00640 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf95de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.699+0000 7f1967e00640 1 -- 172.21.15.116:0/3612722285 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55633bf978d0 con 0x55633bf87240 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1967e00640 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf96320 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f195809a9f0 tx=0x7f195809a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2956e0a5e8128fb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f195cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f195cc00640 1 -- 172.21.15.116:0/3612722285 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958020040 con 0x55633bf87240 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f195cc00640 1 -- 172.21.15.116:0/3612722285 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f19580967b0 con 0x55633bf87240 2024-03-04T07:47:07.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1960005520 con 0x55633bf87240 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f195cc00640 1 -- 172.21.15.116:0/3612722285 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f195809eb00 con 0x55633bf87240 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 10 client.26894 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55633bff8820 con 0x55633bf87240 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f195cc00640 1 -- 172.21.15.116:0/3612722285 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1960005520 con 0x55633bf87240 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f195cc00640 1 -- 172.21.15.116:0/3612722285 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f195809bce0 con 0x55633bf87240 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 10 client.26894 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 10 client.26894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 20 client.26894 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bff9030 0x55633bffb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 4 client.26894 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.700+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55633bef3c00 con 0x55633bff9030 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.701+0000 7f196ea00640 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bff9030 0x55633bffb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.701+0000 7f196ea00640 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bff9030 0x55633bffb450 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69d645eb3db92bbc server_cookie=1e81341fef05d231 in_seq=0 out_seq=0 2024-03-04T07:47:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.701+0000 7f195cc00640 10 client.26894 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:07.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f195cc00640 1 -- 172.21.15.116:0/3612722285 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55633bef3c00 con 0x55633bff9030 2024-03-04T07:47:07.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f195cc00640 10 client.26894 handle_command_reply: tid=0 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 client.26894 shutdown 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 msgr2=0x55633bf96320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf96320 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f195809a9f0 tx=0x7f195809a6d0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 shutdown_connections 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55633bff9030 0x55633bffb450 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55633bf8c3b0 0x55633bf972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55633bf87240 0x55633bf96320 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 --2- 172.21.15.116:0/3612722285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55633bf86850 0x55633bf95de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 >> 172.21.15.116:0/3612722285 conn(0x55633bf81dc0 msgr2=0x55633bf80750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f195cc00640 0 client.26894 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 shutdown_connections 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.716+0000 7f1978c08740 1 -- 172.21.15.116:0/3612722285 wait complete. 2024-03-04T07:47:07.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.717+0000 7f1978c08740 20 client.26894 trim_cache size 0 max 16384 2024-03-04T07:47:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:47:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:47:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:47:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:47:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.717+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19540a1e20 msgr2=0x7f19540a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.717+0000 7f1975e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19540a1e20 0x7f19540a42e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55633bf87aa0 tx=0x7f1968009500 comp rx=0 tx=0).stop 2024-03-04T07:47:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.717+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970108e20 msgr2=0x7f197019e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.717+0000 7f1975e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970108e20 0x7f197019e3c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f195800d950 tx=0x7f195800de10 comp rx=0 tx=0).stop 2024-03-04T07:47:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 shutdown_connections 2024-03-04T07:47:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19701097e0 0x7f19701a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1970108e20 0x7f197019e3c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f19540a1e20 0x7f19540a42e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 --2- 172.21.15.116:0/3852839352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1970107c30 0x7f197019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 >> 172.21.15.116:0/3852839352 conn(0x7f1970076be0 msgr2=0x7f197010c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 shutdown_connections 2024-03-04T07:47:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:07.718+0000 7f1975e00640 1 -- 172.21.15.116:0/3852839352 wait complete. 2024-03-04T07:47:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:08 smithi116 ceph-mon[32150]: pgmap v592: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-04T07:47:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:08 smithi152 ceph-mon[37154]: pgmap v592: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-04T07:47:09.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:08 smithi133 ceph-mon[36505]: pgmap v592: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-04T07:47:10.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:47:10.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:47:10.157 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:251/600: open fd = 3 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:252/600: open fd = 3 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:253/600: open fd = 3 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.158 INFO:tasks.workunit.client.0.smithi116.stdout:254/600: open fd = 3 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:255/600: open fd = 3 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:256/600: open fd = 3 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:257/600: open fd = 3 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.159 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:258/600: open fd = 3 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:259/600: open fd = 3 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:260/600: open fd = 3 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.160 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:261/600: open fd = 3 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:262/600: open fd = 3 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:263/600: open fd = 3 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:264/600: open fd = 3 2024-03-04T07:47:10.161 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:265/600: open fd = 3 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:266/600: open fd = 3 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:267/600: open fd = 3 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.162 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:268/600: open fd = 3 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:269/600: open fd = 3 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:270/600: open fd = 3 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:271/600: open fd = 3 2024-03-04T07:47:10.163 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:272/600: open fd = 3 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:273/600: open fd = 3 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:274/600: open fd = 3 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.164 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:275/600: open fd = 3 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:276/600: open fd = 3 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:277/600: open fd = 3 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:278/600: open fd = 3 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.165 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:279/600: open fd = 3 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:280/600: open fd = 3 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:281/600: open fd = 3 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:282/600: open fd = 3 2024-03-04T07:47:10.166 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:283/600: open fd = 3 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:284/600: open fd = 3 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:285/600: open fd = 3 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.167 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:286/600: open fd = 3 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:287/600: open fd = 3 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:288/600: open fd = 3 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:289/600: open fd = 3 2024-03-04T07:47:10.168 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:290/600: open fd = 3 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:291/600: open fd = 3 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:292/600: open fd = 3 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.169 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:293/600: open fd = 3 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:294/600: open fd = 3 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.170 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:295/600: open fd = 3 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:296/600: open fd = 3 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:297/600: open fd = 3 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.171 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:298/600: open fd = 3 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:299/600: open fd = 3 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:300/600: open fd = 3 2024-03-04T07:47:10.172 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:301/600: open fd = 3 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:302/600: open fd = 3 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:303/600: open fd = 3 2024-03-04T07:47:10.173 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:304/600: open fd = 3 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:305/600: open fd = 3 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:306/600: open fd = 3 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.174 INFO:tasks.workunit.client.0.smithi116.stdout:307/600: open fd = 3 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:308/600: open fd = 3 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:309/600: open fd = 3 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.175 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:310/600: open fd = 3 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:311/600: open fd = 3 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:312/600: open fd = 3 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:47:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 -- 172.21.15.116:0/1160444955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 msgr2=0x7faba8109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 --2- 172.21.15.116:0/1160444955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba8109260 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fab90009900 tx=0x7fab9002bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 -- 172.21.15.116:0/1160444955 shutdown_connections 2024-03-04T07:47:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 --2- 172.21.15.116:0/1160444955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faba81097a0 0x7faba8110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 --2- 172.21.15.116:0/1160444955 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba8109260 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 --2- 172.21.15.116:0/1160444955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faba8107bf0 0x7faba8108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 -- 172.21.15.116:0/1160444955 >> 172.21.15.116:0/1160444955 conn(0x7faba8076b90 msgr2=0x7faba8078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 -- 172.21.15.116:0/1160444955 shutdown_connections 2024-03-04T07:47:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.332+0000 7fabae200640 1 -- 172.21.15.116:0/1160444955 wait complete. 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 Processor -- start 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 -- start start 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faba8107bf0 0x7faba819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faba81097a0 0x7faba81a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba8113870 con 0x7faba81097a0 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81136f0 con 0x7faba8108de0 2024-03-04T07:47:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.333+0000 7fabae200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81139f0 con 0x7faba8107bf0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7faba7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faba81097a0 0x7faba81a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45824/0 (socket says 172.21.15.116:45824) 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 -- 172.21.15.116:0/2769194446 learned_addr learned my addr 172.21.15.116:0/2769194446 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7faba7400640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faba8107bf0 0x7faba819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 -- 172.21.15.116:0/2769194446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faba8107bf0 msgr2=0x7faba819dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faba8107bf0 0x7faba819dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 -- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faba81097a0 msgr2=0x7faba81a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faba81097a0 0x7faba81a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 -- 172.21.15.116:0/2769194446 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba81a5b00 con 0x7faba8108de0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.334+0000 7fab9fe00640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba819e420 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fab90009230 tx=0x7fab9002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.335+0000 7faba4c00640 1 -- 172.21.15.116:0/2769194446 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90041070 con 0x7faba8108de0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.335+0000 7faba4c00640 1 -- 172.21.15.116:0/2769194446 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab90004450 con 0x7faba8108de0 2024-03-04T07:47:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.335+0000 7faba4c00640 1 -- 172.21.15.116:0/2769194446 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90003e20 con 0x7faba8108de0 2024-03-04T07:47:10.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.335+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba81a5d90 con 0x7faba8108de0 2024-03-04T07:47:10.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.335+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba81a61d0 con 0x7faba8108de0 2024-03-04T07:47:10.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7faba4c00640 1 -- 172.21.15.116:0/2769194446 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fab90044070 con 0x7faba8108de0 2024-03-04T07:47:10.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 Processor -- start 2024-03-04T07:47:10.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 -- start start 2024-03-04T07:47:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01af540 0x5572c01b4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01b46b0 0x5572c01b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c00dc090 con 0x5572c01af540 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c0113d30 con 0x5572c01aeb50 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7fabb0fba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c00bca30 con 0x5572c01b46b0 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.338+0000 7faba7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45830/0 (socket says 172.21.15.116:45830) 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01af540 0x5572c01b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 -- 172.21.15.116:0/3404046021 learned_addr learned my addr 172.21.15.116:0/3404046021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7fab9fe00640 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01b46b0 0x5572c01b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 -- 172.21.15.116:0/3404046021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01b46b0 msgr2=0x5572c01b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01b46b0 0x5572c01b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 -- 172.21.15.116:0/3404046021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01af540 msgr2=0x5572c01b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01af540 0x5572c01b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 -- 172.21.15.116:0/3404046021 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572bfef27d0 con 0x5572c01aeb50 2024-03-04T07:47:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba7e00640 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01aef70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7faba0009fd0 tx=0x7faba00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21483dd196878ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7faba4c00640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab8c0a1e00 0x7fab8c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7fab97e00640 1 -- 172.21.15.116:0/3404046021 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba0004d20 con 0x5572c01aeb50 2024-03-04T07:47:10.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7fab97e00640 1 -- 172.21.15.116:0/3404046021 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faba0004050 con 0x5572c01aeb50 2024-03-04T07:47:10.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.339+0000 7fab97e00640 1 -- 172.21.15.116:0/3404046021 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba0005260 con 0x5572c01aeb50 2024-03-04T07:47:10.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7faba4c00640 1 -- 172.21.15.116:0/2769194446 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fab900ef990 con 0x7faba8108de0 2024-03-04T07:47:10.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7faba7400640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab8c0a1e00 0x7fab8c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 -- 172.21.15.116:0/3404046021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 msgr2=0x5572c01aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01aef70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7faba0009fd0 tx=0x7faba00118d0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7faba7400640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab8c0a1e00 0x7fab8c0a42c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x5572c01af940 tx=0x7fab98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 -- 172.21.15.116:0/3404046021 shutdown_connections 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01b46b0 0x5572c01b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01af540 0x5572c01b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3404046021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01aef70 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 -- 172.21.15.116:0/3404046021 >> 172.21.15.116:0/3404046021 conn(0x5572c01aa0c0 msgr2=0x5572c01aa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 -- 172.21.15.116:0/3404046021 shutdown_connections 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.340+0000 7fabb0fba740 1 -- 172.21.15.116:0/3404046021 wait complete. 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.341+0000 7fabb0fba740 1 -- start start 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.341+0000 7fabb0fba740 1 Processor -- start 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.341+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01b9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01c04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7fabb0fba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c00b20c0 con 0x5572c01b46b0 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7fabb0fba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c010fdd0 con 0x5572c01aeb50 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7fabb0fba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c01b7660 con 0x5572c01af540 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7fabb0fba740 10 client.0 _ll_register_callbacks cb 0x5572c00cf860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01b8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45844/0 (socket says 172.21.15.116:45844) 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 -- 172.21.15.116:0/279915927 learned_addr learned my addr 172.21.15.116:0/279915927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7fab9fe00640 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01b9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7e00640 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01c04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 -- 172.21.15.116:0/279915927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 msgr2=0x5572c01b9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01b9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 -- 172.21.15.116:0/279915927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 msgr2=0x5572c01c04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01c04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.342+0000 7faba7400640 1 -- 172.21.15.116:0/279915927 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572bfe283f0 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7faba7400640 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01b8f50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fab98013010 tx=0x7fab98004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a859da02a491f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fab94c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fab94c00640 1 -- 172.21.15.116:0/279915927 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98003d00 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fab94c00640 1 -- 172.21.15.116:0/279915927 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fab9802a420 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faba0004050 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fab94c00640 1 -- 172.21.15.116:0/279915927 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab9802a620 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fabb0fba740 10 client.26906 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5572bfde80f0 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fab94c00640 1 -- 172.21.15.116:0/279915927 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faba0004050 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.343+0000 7fab94c00640 1 -- 172.21.15.116:0/279915927 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fab980207b0 con 0x5572c01aeb50 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fabb0fba740 10 client.26906 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:10.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fabb0fba740 10 client.26906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fabb0fba740 20 client.26906 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fabb0fba740 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c01c4a70 0x5572c01c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fabb0fba740 4 client.26906 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572c0121270 con 0x5572c01c4a70 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fab9fe00640 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c01c4a70 0x5572c01c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fab9fe00640 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c01c4a70 0x5572c01c6e80 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=885b1c6309df3507 server_cookie=b4d3328957ebbeeb in_seq=0 out_seq=0 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.344+0000 7fab94c00640 10 client.26906 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fab94c00640 1 -- 172.21.15.116:0/279915927 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5572c0121270 con 0x5572c01c4a70 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fab94c00640 10 client.26906 handle_command_reply: tid=0 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 client.26906 shutdown 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 msgr2=0x5572c01b8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01b8f50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fab98013010 tx=0x7fab98004730 comp rx=0 tx=0).stop 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 shutdown_connections 2024-03-04T07:47:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c01c4a70 0x5572c01c6e80 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01c04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01b9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 --2- 172.21.15.116:0/279915927 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01b8f50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fab94c00640 0 client.26906 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 >> 172.21.15.116:0/279915927 conn(0x5572c01aa0c0 msgr2=0x5572c01a9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.345+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 shutdown_connections 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.346+0000 7fabb0fba740 1 -- 172.21.15.116:0/279915927 wait complete. 2024-03-04T07:47:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.346+0000 7fabb0fba740 20 client.26906 trim_cache size 0 max 16384 2024-03-04T07:47:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 Processor -- start 2024-03-04T07:47:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 -- start start 2024-03-04T07:47:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01aeb50 0x5572c01a8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01af540 0x5572c01c0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01a9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81ad050 con 0x5572c01b46b0 2024-03-04T07:47:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81af630 con 0x5572c01af540 2024-03-04T07:47:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.363+0000 7fabb0fba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81ad650 con 0x5572c01aeb50 2024-03-04T07:47:10.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01af540 0x5572c01c0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7faba7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01af540 0x5572c01c0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45858/0 (socket says 172.21.15.116:45858) 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 -- 172.21.15.116:0/3644592769 learned_addr learned my addr 172.21.15.116:0/3644592769 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7faba7400640 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01aeb50 0x5572c01a8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 -- 172.21.15.116:0/3644592769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01aeb50 msgr2=0x5572c01a8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01aeb50 0x5572c01a8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 -- 172.21.15.116:0/3644592769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 msgr2=0x5572c01a9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01a9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 -- 172.21.15.116:0/3644592769 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572c01c4810 con 0x5572c01af540 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7faba7400640 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01aeb50 0x5572c01a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab9fe00640 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01af540 0x5572c01c0490 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fab900bc380 tx=0x7fab900dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71a77ceb0f5819c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab97400640 1 -- 172.21.15.116:0/3644592769 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9004b040 con 0x5572c01af540 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab97400640 1 -- 172.21.15.116:0/3644592769 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab90004c60 con 0x5572c01af540 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.364+0000 7fab97400640 1 -- 172.21.15.116:0/3644592769 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab900ed560 con 0x5572c01af540 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 -- 172.21.15.116:0/3644592769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01af540 msgr2=0x5572c01c0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01af540 0x5572c01c0490 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fab900bc380 tx=0x7fab900dc040 comp rx=0 tx=0).stop 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 -- 172.21.15.116:0/3644592769 shutdown_connections 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01a9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01af540 0x5572c01c0490 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 --2- 172.21.15.116:0/3644592769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01aeb50 0x5572c01a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 -- 172.21.15.116:0/3644592769 >> 172.21.15.116:0/3644592769 conn(0x5572c01aa0c0 msgr2=0x5572c01b8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 -- 172.21.15.116:0/3644592769 shutdown_connections 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.365+0000 7fabb0fba740 1 -- 172.21.15.116:0/3644592769 wait complete. 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 -- start start 2024-03-04T07:47:10.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 Processor -- start 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01bef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01bf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01bf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81adcb0 con 0x5572c01b46b0 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81aefd0 con 0x5572c01aeb50 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba81ae310 con 0x5572c01af540 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.366+0000 7fabb0fba740 10 client.0 _ll_register_callbacks cb 0x7fab980044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01bef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01bef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45874/0 (socket says 172.21.15.116:45874) 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 -- 172.21.15.116:0/382842148 learned_addr learned my addr 172.21.15.116:0/382842148 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7fab9fe00640 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01bf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 -- 172.21.15.116:0/382842148 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 msgr2=0x5572c01bf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01bf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 -- 172.21.15.116:0/382842148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 msgr2=0x5572c01bf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01bf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 -- 172.21.15.116:0/382842148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572c02205f0 con 0x5572c01aeb50 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7faba7400640 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01bef10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fab98020c40 tx=0x7fab980209c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97b3c12fd5d3cf07 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7fab94c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7fab94c00640 1 -- 172.21.15.116:0/382842148 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9802a420 con 0x5572c01aeb50 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7fab94c00640 1 -- 172.21.15.116:0/382842148 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fab98003d00 con 0x5572c01aeb50 2024-03-04T07:47:10.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.367+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab90004c60 con 0x5572c01aeb50 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fab94c00640 1 -- 172.21.15.116:0/382842148 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab98045050 con 0x5572c01aeb50 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 10 client.26912 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5572c0221850 con 0x5572c01aeb50 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fab94c00640 1 -- 172.21.15.116:0/382842148 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab90004c60 con 0x5572c01aeb50 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fab94c00640 1 -- 172.21.15.116:0/382842148 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fab980214d0 con 0x5572c01aeb50 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 10 client.26912 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 10 client.26912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 20 client.26912 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c02221b0 0x5572c02245d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 4 client.26912 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.368+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5572c011b5c0 con 0x5572c02221b0 2024-03-04T07:47:10.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fab9fe00640 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c02221b0 0x5572c02245d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fab9fe00640 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c02221b0 0x5572c02245d0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=128fabf1e284dea5 server_cookie=28c54bfc3eea97f6 in_seq=0 out_seq=0 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fab94c00640 10 client.26912 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fab94c00640 1 -- 172.21.15.116:0/382842148 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5572c011b5c0 con 0x5572c02221b0 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fab94c00640 10 client.26912 handle_command_reply: tid=0 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 client.26912 shutdown 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 msgr2=0x5572c01bef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01bef10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fab98020c40 tx=0x7fab980209c0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 shutdown_connections 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5572c02221b0 0x5572c02245d0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572c01b46b0 0x5572c01bf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fab94c00640 0 client.26912 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572c01af540 0x5572c01bf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 --2- 172.21.15.116:0/382842148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5572c01aeb50 0x5572c01bef10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.369+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 >> 172.21.15.116:0/382842148 conn(0x5572c01aa0c0 msgr2=0x5572c01ad320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 shutdown_connections 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabb0fba740 1 -- 172.21.15.116:0/382842148 wait complete. 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabb0fba740 20 client.26912 trim_cache size 0 max 16384 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:47:10.375 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:47:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab8c0a1e00 msgr2=0x7fab8c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab8c0a1e00 0x7fab8c0a42c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x5572c01af940 tx=0x7fab98005f90 comp rx=0 tx=0).stop 2024-03-04T07:47:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 msgr2=0x7faba819e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba819e420 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fab90009230 tx=0x7fab9002c650 comp rx=0 tx=0).stop 2024-03-04T07:47:10.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 shutdown_connections 2024-03-04T07:47:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faba81097a0 0x7faba81a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7faba8108de0 0x7faba819e420 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fab8c0a1e00 0x7fab8c0a42c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 --2- 172.21.15.116:0/2769194446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faba8107bf0 0x7faba819dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 >> 172.21.15.116:0/2769194446 conn(0x7faba8076b90 msgr2=0x7faba810c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 shutdown_connections 2024-03-04T07:47:10.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.370+0000 7fabae200640 1 -- 172.21.15.116:0/2769194446 wait complete. 2024-03-04T07:47:10.387 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:47:10.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 -- 172.21.15.116:0/2670772437 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e8103620 msgr2=0x7fa3e8103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 --2- 172.21.15.116:0/2670772437 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e8103620 0x7fa3e8103a40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009900 tx=0x7fa3dc02bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 -- 172.21.15.116:0/2670772437 shutdown_connections 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 --2- 172.21.15.116:0/2670772437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e81051d0 0x7fa3e810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 --2- 172.21.15.116:0/2670772437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8104810 0x7fa3e8104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 --2- 172.21.15.116:0/2670772437 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e8103620 0x7fa3e8103a40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 -- 172.21.15.116:0/2670772437 >> 172.21.15.116:0/2670772437 conn(0x7fa3e80febb0 msgr2=0x7fa3e8100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 -- 172.21.15.116:0/2670772437 shutdown_connections 2024-03-04T07:47:10.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.599+0000 7fa3ef000640 1 -- 172.21.15.116:0/2670772437 wait complete. 2024-03-04T07:47:10.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 Processor -- start 2024-03-04T07:47:10.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 -- start start 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8103620 0x7fa3e8199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e8104810 0x7fa3e8199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e81051d0 0x7fa3e81a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e810f2f0 con 0x7fa3e8104810 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e810f170 con 0x7fa3e81051d0 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ef000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e810f470 con 0x7fa3e8103620 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ed200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e8104810 0x7fa3e8199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3edc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8103620 0x7fa3e8199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.600+0000 7fa3ee600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e81051d0 0x7fa3e81a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e81051d0 0x7fa3e81a0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45894/0 (socket says 172.21.15.116:45894) 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3edc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8103620 0x7fa3e8199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:51210/0 (socket says 172.21.15.116:51210) 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 -- 172.21.15.116:0/409148406 learned_addr learned my addr 172.21.15.116:0/409148406 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 -- 172.21.15.116:0/409148406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8103620 msgr2=0x7fa3e8199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8103620 0x7fa3e8199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 -- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e8104810 msgr2=0x7fa3e8199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e8104810 0x7fa3e8199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 -- 172.21.15.116:0/409148406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e81a15b0 con 0x7fa3e81051d0 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ed200640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e8104810 0x7fa3e8199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3ee600640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e81051d0 0x7fa3e81a0f40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000efd0 tx=0x7fa3e000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3e6000640 1 -- 172.21.15.116:0/409148406 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0003b70 con 0x7fa3e81051d0 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3e6000640 1 -- 172.21.15.116:0/409148406 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3e0003d10 con 0x7fa3e81051d0 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.601+0000 7fa3e6000640 1 -- 172.21.15.116:0/409148406 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e00109f0 con 0x7fa3e81051d0 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.602+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e81a18a0 con 0x7fa3e81051d0 2024-03-04T07:47:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.602+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e81a1df0 con 0x7fa3e81051d0 2024-03-04T07:47:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.603+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e810a990 con 0x7fa3e81051d0 2024-03-04T07:47:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.605+0000 7fa3e6000640 1 -- 172.21.15.116:0/409148406 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa3e0004020 con 0x7fa3e81051d0 2024-03-04T07:47:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.606+0000 7fa3e6000640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3d00a1e00 0x7fa3d00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.606+0000 7fa3e6000640 1 -- 172.21.15.116:0/409148406 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa3e00ce1a0 con 0x7fa3e81051d0 2024-03-04T07:47:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.606+0000 7fa3edc00640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3d00a1e00 0x7fa3d00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.607+0000 7fa3edc00640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3d00a1e00 0x7fa3d00a42c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc02c170 tx=0x7fa3dc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:10.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.610+0000 7fa3e6000640 1 -- 172.21.15.116:0/409148406 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa3e009b590 con 0x7fa3e81051d0 2024-03-04T07:47:10.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.848+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3e80621d0 con 0x7fa3e81051d0 2024-03-04T07:47:10.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.849+0000 7fa3e6000640 1 -- 172.21.15.116:0/409148406 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fa3e009ae90 con 0x7fa3e81051d0 2024-03-04T07:47:10.849 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:47:10.849 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:47:10.849 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:47:10.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.851+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3d00a1e00 msgr2=0x7fa3d00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.851+0000 7fa3ef000640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3d00a1e00 0x7fa3d00a42c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc02c170 tx=0x7fa3dc02df90 comp rx=0 tx=0).stop 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.851+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e81051d0 msgr2=0x7fa3e81a0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.851+0000 7fa3ef000640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e81051d0 0x7fa3e81a0f40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000efd0 tx=0x7fa3e000c6a0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.851+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 shutdown_connections 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.852+0000 7fa3ef000640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa3e81051d0 0x7fa3e81a0f40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.852+0000 7fa3ef000640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3e8104810 0x7fa3e8199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.852+0000 7fa3ef000640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa3d00a1e00 0x7fa3d00a42c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.852+0000 7fa3ef000640 1 --2- 172.21.15.116:0/409148406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8103620 0x7fa3e8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.852+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 >> 172.21.15.116:0/409148406 conn(0x7fa3e80febb0 msgr2=0x7fa3e8107b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.852+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 shutdown_connections 2024-03-04T07:47:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:10.852+0000 7fa3ef000640 1 -- 172.21.15.116:0/409148406 wait complete. 2024-03-04T07:47:10.871 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:47:10.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:47:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 -- 172.21.15.116:0/1462009326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 msgr2=0x7f21dc102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 --2- 172.21.15.116:0/1462009326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc102b20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009900 tx=0x7f21c802bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 -- 172.21.15.116:0/1462009326 shutdown_connections 2024-03-04T07:47:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 --2- 172.21.15.116:0/1462009326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21dc103060 0x7f21dc1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 --2- 172.21.15.116:0/1462009326 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc102b20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 --2- 172.21.15.116:0/1462009326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc1014b0 0x7f21dc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 -- 172.21.15.116:0/1462009326 >> 172.21.15.116:0/1462009326 conn(0x7f21dc0fca40 msgr2=0x7f21dc0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 -- 172.21.15.116:0/1462009326 shutdown_connections 2024-03-04T07:47:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.089+0000 7f21e1e00640 1 -- 172.21.15.116:0/1462009326 wait complete. 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.090+0000 7f21e1e00640 1 Processor -- start 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.090+0000 7f21e1e00640 1 -- start start 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.090+0000 7f21e1e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21dc1014b0 0x7f21dc1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc103060 0x7f21dc1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e1e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc10d190 con 0x7f21dc103060 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e1e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc10d010 con 0x7f21dc1026a0 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e1e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc10d310 con 0x7f21dc1014b0 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc103060 0x7f21dc1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e0a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21dc1014b0 0x7f21dc1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45904/0 (socket says 172.21.15.116:45904) 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 -- 172.21.15.116:0/576213134 learned_addr learned my addr 172.21.15.116:0/576213134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 -- 172.21.15.116:0/576213134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21dc1014b0 msgr2=0x7f21dc1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21dc1014b0 0x7f21dc1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 -- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc103060 msgr2=0x7f21dc1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc103060 0x7f21dc1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21dbe00640 1 -- 172.21.15.116:0/576213134 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc1a1600 con 0x7f21dc1026a0 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.091+0000 7f21e1400640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc103060 0x7f21dc1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.092+0000 7f21dbe00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc199f20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f21c802c8e0 tx=0x7f21c802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.092+0000 7f21d9600640 1 -- 172.21.15.116:0/576213134 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8043020 con 0x7f21dc1026a0 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.092+0000 7f21d9600640 1 -- 172.21.15.116:0/576213134 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21c80054e0 con 0x7f21dc1026a0 2024-03-04T07:47:11.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.092+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc1a1890 con 0x7f21dc1026a0 2024-03-04T07:47:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.092+0000 7f21d9600640 1 -- 172.21.15.116:0/576213134 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8003c70 con 0x7f21dc1026a0 2024-03-04T07:47:11.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.092+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc1a1e30 con 0x7f21dc1026a0 2024-03-04T07:47:11.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.095+0000 7f21d9600640 1 -- 172.21.15.116:0/576213134 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f21c8046020 con 0x7f21dc1026a0 2024-03-04T07:47:11.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.095+0000 7f21e61e4740 1 Processor -- start 2024-03-04T07:47:11.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.095+0000 7f21e61e4740 1 -- start start 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d68290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d68860 0x55d682d6d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d6d9d0 0x55d682d6fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e61e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d682cccd60 con 0x55d682d6d9d0 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e61e4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d682c75cf0 con 0x55d682d68860 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e61e4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d682cc8e00 con 0x55d682d67e70 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d68860 0x55d682d6d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d68860 0x55d682d6d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45910/0 (socket says 172.21.15.116:45910) 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 -- 172.21.15.116:0/3733080661 learned_addr learned my addr 172.21.15.116:0/3733080661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21dbe00640 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d6d9d0 0x55d682d6fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e1400640 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d68290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 -- 172.21.15.116:0/3733080661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 msgr2=0x55d682d68290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d68290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 -- 172.21.15.116:0/3733080661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d6d9d0 msgr2=0x55d682d6fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d6d9d0 0x55d682d6fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21e0a00640 1 -- 172.21.15.116:0/3733080661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6829e13f0 con 0x55d682d68860 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21dbe00640 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d6d9d0 0x55d682d6fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.096+0000 7f21d9600640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f21c40a1e00 0x7f21c40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.097+0000 7f21e0a00640 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d68860 0x55d682d6d360 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009ca0 tx=0x7f21d0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88e3d43f2779ef2b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.097+0000 7f21d9600640 1 -- 172.21.15.116:0/576213134 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f21c80efdc0 con 0x7f21dc1026a0 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.097+0000 7f21e1400640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f21c40a1e00 0x7f21c40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.097+0000 7f21cdc00640 1 -- 172.21.15.116:0/3733080661 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0004220 con 0x55d682d68860 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.097+0000 7f21cdc00640 1 -- 172.21.15.116:0/3733080661 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21d00043c0 con 0x55d682d68860 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.097+0000 7f21cdc00640 1 -- 172.21.15.116:0/3733080661 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0019930 con 0x55d682d68860 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 -- 172.21.15.116:0/3733080661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d68860 msgr2=0x55d682d6d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d68860 0x55d682d6d360 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009ca0 tx=0x7f21d0011810 comp rx=0 tx=0).stop 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e1400640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f21c40a1e00 0x7f21c40a42c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x55d682d690c0 tx=0x7f21d4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 -- 172.21.15.116:0/3733080661 shutdown_connections 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d6d9d0 0x55d682d6fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d68860 0x55d682d6d360 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 --2- 172.21.15.116:0/3733080661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d68290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 -- 172.21.15.116:0/3733080661 >> 172.21.15.116:0/3733080661 conn(0x55d682d633e0 msgr2=0x55d682d637e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 -- 172.21.15.116:0/3733080661 shutdown_connections 2024-03-04T07:47:11.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 -- 172.21.15.116:0/3733080661 wait complete. 2024-03-04T07:47:11.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.098+0000 7f21e61e4740 1 -- start start 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 1 Processor -- start 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d67e70 0x55d682d723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d72930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d6d9d0 0x55d682d79970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d682d70980 con 0x55d682d68860 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d682c6b220 con 0x55d682d67e70 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d682d70b00 con 0x55d682d6d9d0 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e61e4740 10 client.0 _ll_register_callbacks cb 0x55d6829e0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d67e70 0x55d682d723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d67e70 0x55d682d723f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45912/0 (socket says 172.21.15.116:45912) 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e0a00640 1 -- 172.21.15.116:0/742418074 learned_addr learned my addr 172.21.15.116:0/742418074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21dbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d72930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.099+0000 7f21e1400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d6d9d0 0x55d682d79970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21e0a00640 1 -- 172.21.15.116:0/742418074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d6d9d0 msgr2=0x55d682d79970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21e0a00640 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d6d9d0 0x55d682d79970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21e0a00640 1 -- 172.21.15.116:0/742418074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 msgr2=0x55d682d72930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21e0a00640 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d72930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21e0a00640 1 -- 172.21.15.116:0/742418074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d682aab7d0 con 0x55d682d67e70 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21e0a00640 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d67e70 0x55d682d723f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f21d0011d10 tx=0x7f21d0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7ed755c8db4fbae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21c2000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21c2000640 1 -- 172.21.15.116:0/742418074 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0028070 con 0x55d682d67e70 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21c2000640 1 -- 172.21.15.116:0/742418074 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f21d0005040 con 0x55d682d67e70 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.100+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21d00043c0 con 0x55d682d67e70 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21c2000640 1 -- 172.21.15.116:0/742418074 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21d0034050 con 0x55d682d67e70 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 10 client.26930 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:11.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d682d7cd30 con 0x55d682d67e70 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21c2000640 1 -- 172.21.15.116:0/742418074 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21d00043c0 con 0x55d682d67e70 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21c2000640 1 -- 172.21.15.116:0/742418074 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f21d00048c0 con 0x55d682d67e70 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 10 client.26930 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 10 client.26930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 20 client.26930 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682d7e150 0x55d682d80580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 4 client.26930 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.101+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d682cda2a0 con 0x55d682d7e150 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.102+0000 7f21dbe00640 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682d7e150 0x55d682d80580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.102+0000 7f21dbe00640 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682d7e150 0x55d682d80580 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e8372d895c835d server_cookie=eef4df8c61314b5c in_seq=0 out_seq=0 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.102+0000 7f21c2000640 10 client.26930 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21c2000640 1 -- 172.21.15.116:0/742418074 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d682cda2a0 con 0x55d682d7e150 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21c2000640 10 client.26930 handle_command_reply: tid=0 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 client.26930 shutdown 2024-03-04T07:47:11.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d67e70 msgr2=0x55d682d723f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d67e70 0x55d682d723f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f21d0011d10 tx=0x7f21d0003820 comp rx=0 tx=0).stop 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 shutdown_connections 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682d7e150 0x55d682d80580 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d6d9d0 0x55d682d79970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d72930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 --2- 172.21.15.116:0/742418074 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d67e70 0x55d682d723f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 >> 172.21.15.116:0/742418074 conn(0x55d682d633e0 msgr2=0x55d682d62700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21c2000640 0 client.26930 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 shutdown_connections 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 1 -- 172.21.15.116:0/742418074 wait complete. 2024-03-04T07:47:11.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.103+0000 7f21e61e4740 20 client.26930 trim_cache size 0 max 16384 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 Processor -- start 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 -- start start 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d67e70 0x55d682d61e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d68860 0x55d682d79930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1a8a90 con 0x55d682d67e70 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1ab070 con 0x55d682d6d9d0 2024-03-04T07:47:11.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e61e4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1a9090 con 0x55d682d68860 2024-03-04T07:47:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d67e70 0x55d682d61e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21dbe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d68860 0x55d682d79930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21dbe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d68860 0x55d682d79930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:51244/0 (socket says 172.21.15.116:51244) 2024-03-04T07:47:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21dbe00640 1 -- 172.21.15.116:0/2629183443 learned_addr learned my addr 172.21.15.116:0/2629183443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.117+0000 7f21e0a00640 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d67e70 0x55d682d61e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50668/0 (socket says 172.21.15.116:50668) 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21dbe00640 1 -- 172.21.15.116:0/2629183443 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 msgr2=0x55d682d623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21dbe00640 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21dbe00640 1 -- 172.21.15.116:0/2629183443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d67e70 msgr2=0x55d682d61e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21dbe00640 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d67e70 0x55d682d61e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21dbe00640 1 -- 172.21.15.116:0/2629183443 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d682d70860 con 0x55d682d68860 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21e1400640 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21e0a00640 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d67e70 0x55d682d61e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21dbe00640 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d68860 0x55d682d79930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f21c80b8f10 tx=0x7f21c8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30c764fc2250da2d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21cd200640 1 -- 172.21.15.116:0/2629183443 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c80bd1b0 con 0x55d682d68860 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21cd200640 1 -- 172.21.15.116:0/2629183443 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21c80d5070 con 0x55d682d68860 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.118+0000 7f21cd200640 1 -- 172.21.15.116:0/2629183443 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c80badb0 con 0x55d682d68860 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 -- 172.21.15.116:0/2629183443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d68860 msgr2=0x55d682d79930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d68860 0x55d682d79930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f21c80b8f10 tx=0x7f21c8009260 comp rx=0 tx=0).stop 2024-03-04T07:47:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 -- 172.21.15.116:0/2629183443 shutdown_connections 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d68860 0x55d682d79930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 --2- 172.21.15.116:0/2629183443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d67e70 0x55d682d61e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 -- 172.21.15.116:0/2629183443 >> 172.21.15.116:0/2629183443 conn(0x55d682d633e0 msgr2=0x55d682d637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 -- 172.21.15.116:0/2629183443 shutdown_connections 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 -- 172.21.15.116:0/2629183443 wait complete. 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.119+0000 7f21e61e4740 1 -- start start 2024-03-04T07:47:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 1 Processor -- start 2024-03-04T07:47:11.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d76df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d77330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d78340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1a96f0 con 0x55d682d68860 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1aaa10 con 0x55d682d6d9d0 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1a9d50 con 0x55d682d67e70 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e61e4740 10 client.0 _ll_register_callbacks cb 0x7f21d4009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e0a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d76df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21dbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d77330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21e1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d78340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21dbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d77330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50682/0 (socket says 172.21.15.116:50682) 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.120+0000 7f21dbe00640 1 -- 172.21.15.116:0/1538641695 learned_addr learned my addr 172.21.15.116:0/1538641695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21dbe00640 1 -- 172.21.15.116:0/1538641695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 msgr2=0x55d682d76df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21dbe00640 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d76df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21dbe00640 1 -- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 msgr2=0x55d682d78340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:11.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21dbe00640 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d78340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21dbe00640 1 -- 172.21.15.116:0/1538641695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d682d78930 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21e0a00640 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d76df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21e1400640 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d78340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21dbe00640 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d77330 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21c80bc7b0 tx=0x7f21c802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6292706571bb0a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21c2000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21c2000640 1 -- 172.21.15.116:0/1538641695 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c80bcab0 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21c2000640 1 -- 172.21.15.116:0/1538641695 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f21c80ed990 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21c80d5070 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21c2000640 1 -- 172.21.15.116:0/1538641695 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21c80c0c00 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21e61e4740 10 client.16809 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.121+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55d682d79ca0 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21c2000640 1 -- 172.21.15.116:0/1538641695 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21c80d5070 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21c2000640 1 -- 172.21.15.116:0/1538641695 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f21c80c4dc0 con 0x55d682d68860 2024-03-04T07:47:11.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21e61e4740 10 client.16809 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21e61e4740 10 client.16809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21e61e4740 20 client.16809 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21e61e4740 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682dda090 0x55d682ddc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21e61e4740 4 client.16809 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d682cd45f0 con 0x55d682dda090 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.122+0000 7f21e0a00640 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682dda090 0x55d682ddc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e0a00640 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682dda090 0x55d682ddc4b0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f042c5da1e8f8799 server_cookie=59b427339e2453a5 in_seq=0 out_seq=0 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21c2000640 10 client.16809 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21c2000640 1 -- 172.21.15.116:0/1538641695 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d682cd45f0 con 0x55d682dda090 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21c2000640 10 client.16809 handle_command_reply: tid=0 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 client.16809 shutdown 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 msgr2=0x55d682d77330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d77330 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f21c80bc7b0 tx=0x7f21c802bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 shutdown_connections 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55d682dda090 0x55d682ddc4b0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55d682d6d9d0 0x55d682d78340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d682d68860 0x55d682d77330 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21c2000640 0 client.16809 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 --2- 172.21.15.116:0/1538641695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d682d67e70 0x55d682d76df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 >> 172.21.15.116:0/1538641695 conn(0x55d682d633e0 msgr2=0x55d682d66ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 shutdown_connections 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 1 -- 172.21.15.116:0/1538641695 wait complete. 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.123+0000 7f21e61e4740 20 client.16809 trim_cache size 0 max 16384 2024-03-04T07:47:11.128 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:47:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f21c40a1e00 msgr2=0x7f21c40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f21c40a1e00 0x7f21c40a42c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x55d682d690c0 tx=0x7f21d4009500 comp rx=0 tx=0).stop 2024-03-04T07:47:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 msgr2=0x7f21dc199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc199f20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f21c802c8e0 tx=0x7f21c802c650 comp rx=0 tx=0).stop 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 shutdown_connections 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21dc103060 0x7f21dc1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f21dc1026a0 0x7f21dc199f20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f21c40a1e00 0x7f21c40a42c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 --2- 172.21.15.116:0/576213134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21dc1014b0 0x7f21dc1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 >> 172.21.15.116:0/576213134 conn(0x7f21dc0fca40 msgr2=0x7f21dc073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 shutdown_connections 2024-03-04T07:47:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:11.124+0000 7f21e1e00640 1 -- 172.21.15.116:0/576213134 wait complete. 2024-03-04T07:47:11.138 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:10 smithi116 ceph-mon[32150]: pgmap v593: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-03-04T07:47:11.139 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:10 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/409148406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:47:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:10 smithi152 ceph-mon[37154]: pgmap v593: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-03-04T07:47:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:10 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/409148406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:47:11.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:10 smithi133 ceph-mon[36505]: pgmap v593: 97 pgs: 97 active+clean; 4.7 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 7 op/s 2024-03-04T07:47:11.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:10 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/409148406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:47:12.142 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:47:12.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:47:12.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.368+0000 7f5650800640 1 -- 172.21.15.116:0/2517895345 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481049e0 msgr2=0x7f5648104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.368+0000 7f5650800640 1 --2- 172.21.15.116:0/2517895345 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481049e0 0x7f5648104e60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5634009900 tx=0x7f563402bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.369+0000 7f5650800640 1 -- 172.21.15.116:0/2517895345 shutdown_connections 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.369+0000 7f5650800640 1 --2- 172.21.15.116:0/2517895345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56481053a0 0x7f564810bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.369+0000 7f5650800640 1 --2- 172.21.15.116:0/2517895345 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481049e0 0x7f5648104e60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.369+0000 7f5650800640 1 --2- 172.21.15.116:0/2517895345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56481037f0 0x7f5648103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.369+0000 7f5650800640 1 -- 172.21.15.116:0/2517895345 >> 172.21.15.116:0/2517895345 conn(0x7f56480fed80 msgr2=0x7f56481011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.369+0000 7f5650800640 1 -- 172.21.15.116:0/2517895345 shutdown_connections 2024-03-04T07:47:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.369+0000 7f5650800640 1 -- 172.21.15.116:0/2517895345 wait complete. 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 Processor -- start 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 -- start start 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481037f0 0x7f5648199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56481049e0 0x7f564819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56481053a0 0x7f56481a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564810f4c0 con 0x7f56481053a0 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564810f340 con 0x7f56481037f0 2024-03-04T07:47:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.370+0000 7f5650800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564810f640 con 0x7f56481049e0 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481037f0 0x7f5648199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481037f0 0x7f5648199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45942/0 (socket says 172.21.15.116:45942) 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 -- 172.21.15.116:0/3114514906 learned_addr learned my addr 172.21.15.116:0/3114514906 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564d400640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56481049e0 0x7f564819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564e800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56481053a0 0x7f56481a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 -- 172.21.15.116:0/3114514906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56481049e0 msgr2=0x7f564819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56481049e0 0x7f564819a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 -- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56481053a0 msgr2=0x7f56481a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56481053a0 0x7f56481a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 -- 172.21.15.116:0/3114514906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56481a1700 con 0x7f56481037f0 2024-03-04T07:47:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.371+0000 7f564de00640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481037f0 0x7f5648199ae0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f563c00e9c0 tx=0x7f563c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.372+0000 7f5642a00640 1 -- 172.21.15.116:0/3114514906 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c00cd80 con 0x7f56481037f0 2024-03-04T07:47:12.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.372+0000 7f5642a00640 1 -- 172.21.15.116:0/3114514906 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f563c0035f0 con 0x7f56481037f0 2024-03-04T07:47:12.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.372+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56481a19f0 con 0x7f56481037f0 2024-03-04T07:47:12.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.372+0000 7f5642a00640 1 -- 172.21.15.116:0/3114514906 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c010690 con 0x7f56481037f0 2024-03-04T07:47:12.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.372+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56481a1f10 con 0x7f56481037f0 2024-03-04T07:47:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.375+0000 7f5653578740 1 Processor -- start 2024-03-04T07:47:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.375+0000 7f5653578740 1 -- start start 2024-03-04T07:47:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f5653578740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed3f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f5653578740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed445c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f5653578740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed44c30 0x55ec5ed470c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f5653578740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5ec52e20 con 0x55ec5ed44c30 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f5653578740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5eca4a20 con 0x55ec5ed3fac0 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f5653578740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5ec4db00 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed3f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed3f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:51268/0 (socket says 172.21.15.116:51268) 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 -- 172.21.15.116:0/2305520815 learned_addr learned my addr 172.21.15.116:0/2305520815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564d400640 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed44c30 0x55ec5ed470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 -- 172.21.15.116:0/2305520815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 msgr2=0x55ec5ed445c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed445c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 -- 172.21.15.116:0/2305520815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed44c30 msgr2=0x55ec5ed470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed44c30 0x55ec5ed470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f5642a00640 1 -- 172.21.15.116:0/3114514906 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f563c0037a0 con 0x7f56481037f0 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564e800640 1 -- 172.21.15.116:0/2305520815 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec5ec62e10 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.376+0000 7f564d400640 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed44c30 0x55ec5ed470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.377+0000 7f564e800640 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed3f4f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5644009fd0 tx=0x7f56440118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed990965fe0faf56 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.377+0000 7f563b000640 1 -- 172.21.15.116:0/2305520815 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644004d20 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.377+0000 7f563b000640 1 -- 172.21.15.116:0/2305520815 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5644004050 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.377+0000 7f563b000640 1 -- 172.21.15.116:0/2305520815 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644005260 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.377+0000 7f5653578740 1 -- 172.21.15.116:0/2305520815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 msgr2=0x55ec5ed3f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.377+0000 7f5653578740 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed3f4f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5644009fd0 tx=0x7f56440118d0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5653578740 1 -- 172.21.15.116:0/2305520815 shutdown_connections 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5642a00640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56300a1d80 0x7f56300a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5653578740 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed44c30 0x55ec5ed470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5653578740 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5653578740 1 --2- 172.21.15.116:0/2305520815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed3f4f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5653578740 1 -- 172.21.15.116:0/2305520815 >> 172.21.15.116:0/2305520815 conn(0x55ec5ed3a640 msgr2=0x55ec5ed3aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5642a00640 1 -- 172.21.15.116:0/3114514906 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f563c0ce130 con 0x7f56481037f0 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5653578740 1 -- 172.21.15.116:0/2305520815 shutdown_connections 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f564e800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56300a1d80 0x7f56300a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.378+0000 7f5653578740 1 -- 172.21.15.116:0/2305520815 wait complete. 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 -- start start 2024-03-04T07:47:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f564e800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56300a1d80 0x7f56300a4240 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5644005fd0 tx=0x7f5644009d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 Processor -- start 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed44c30 0x55ec5ed50a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5ec60f40 con 0x55ec5ed3fac0 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5eca0ac0 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5ed47be0 con 0x55ec5ed44c30 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.379+0000 7f5653578740 10 client.0 _ll_register_callbacks cb 0x55ec5ec9faa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45944/0 (socket says 172.21.15.116:45944) 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 -- 172.21.15.116:0/3676059644 learned_addr learned my addr 172.21.15.116:0/3676059644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564e800640 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed44c30 0x55ec5ed50a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 -- 172.21.15.116:0/3676059644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed44c30 msgr2=0x55ec5ed50a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed44c30 0x55ec5ed50a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 -- 172.21.15.116:0/3676059644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3fac0 msgr2=0x55ec5ed499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 -- 172.21.15.116:0/3676059644 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec5ea817e0 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f564de00640 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed494a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f563c0a30d0 tx=0x7f563c09b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52ec0696bf9e1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f562f400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.380+0000 7f562f400640 1 -- 172.21.15.116:0/3676059644 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c0b2070 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f562f400640 1 -- 172.21.15.116:0/3676059644 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f563c0b91f0 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5644004050 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f562f400640 1 -- 172.21.15.116:0/3676059644 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f563c0a3600 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 10 client.26945 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ec5eae1f70 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f562f400640 1 -- 172.21.15.116:0/3676059644 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5644004050 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f562f400640 1 -- 172.21.15.116:0/3676059644 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f563c099db0 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 10 client.26945 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 10 client.26945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 20 client.26945 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed55260 0x55ec5ed57690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 4 client.26945 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.381+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec5ecb1f60 con 0x55ec5ed55260 2024-03-04T07:47:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.382+0000 7f564d400640 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed55260 0x55ec5ed57690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.382+0000 7f564d400640 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed55260 0x55ec5ed57690 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a893fc807517d56 server_cookie=1ea40fd1cf8569c9 in_seq=0 out_seq=0 2024-03-04T07:47:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.382+0000 7f562f400640 10 client.26945 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f562f400640 1 -- 172.21.15.116:0/3676059644 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ec5ecb1f60 con 0x55ec5ed55260 2024-03-04T07:47:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f562f400640 10 client.26945 handle_command_reply: tid=0 2024-03-04T07:47:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 client.26945 shutdown 2024-03-04T07:47:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3f0d0 msgr2=0x55ec5ed494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed494a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f563c0a30d0 tx=0x7f563c09b560 comp rx=0 tx=0).stop 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 shutdown_connections 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed55260 0x55ec5ed57690 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed44c30 0x55ec5ed50a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 --2- 172.21.15.116:0/3676059644 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed494a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 >> 172.21.15.116:0/3676059644 conn(0x55ec5ed3a640 msgr2=0x55ec5ed3b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f562f400640 0 client.26945 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 shutdown_connections 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 1 -- 172.21.15.116:0/3676059644 wait complete. 2024-03-04T07:47:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.383+0000 7f5653578740 20 client.26945 trim_cache size 0 max 16384 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 Processor -- start 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 -- start start 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed398e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed509e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed5bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481a8c90 con 0x55ec5ed3a640 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481ab270 con 0x55ec5ed3fac0 2024-03-04T07:47:12.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f5653578740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481a9290 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f564e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed5bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f564e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed5bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45960/0 (socket says 172.21.15.116:45960) 2024-03-04T07:47:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f564e800640 1 -- 172.21.15.116:0/506382358 learned_addr learned my addr 172.21.15.116:0/506382358 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.401+0000 7f564de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f564d400640 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f564e800640 1 -- 172.21.15.116:0/506382358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 msgr2=0x55ec5ed509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f564e800640 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f564e800640 1 -- 172.21.15.116:0/506382358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 msgr2=0x55ec5ed398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f564e800640 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f564e800640 1 -- 172.21.15.116:0/506382358 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec5e979170 con 0x55ec5ed3fac0 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f564e800640 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed5bf50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5644009f70 tx=0x7f5644023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0201a4d36c0b94a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f563a600640 1 -- 172.21.15.116:0/506382358 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644035030 con 0x55ec5ed3fac0 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f563a600640 1 -- 172.21.15.116:0/506382358 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f564402a040 con 0x55ec5ed3fac0 2024-03-04T07:47:12.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.402+0000 7f563a600640 1 -- 172.21.15.116:0/506382358 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56440059a0 con 0x55ec5ed3fac0 2024-03-04T07:47:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 -- 172.21.15.116:0/506382358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 msgr2=0x55ec5ed5bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed5bf50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5644009f70 tx=0x7f5644023000 comp rx=0 tx=0).stop 2024-03-04T07:47:12.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 -- 172.21.15.116:0/506382358 shutdown_connections 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed5bf50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 --2- 172.21.15.116:0/506382358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 -- 172.21.15.116:0/506382358 >> 172.21.15.116:0/506382358 conn(0x55ec5ed408a0 msgr2=0x55ec5ed391d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 -- 172.21.15.116:0/506382358 shutdown_connections 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 -- 172.21.15.116:0/506382358 wait complete. 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.403+0000 7f5653578740 1 -- start start 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 1 Processor -- start 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed4f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed4f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed4fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481a98f0 con 0x55ec5ed3a640 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481aac10 con 0x55ec5ed3fac0 2024-03-04T07:47:12.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481a9f50 con 0x55ec5ed3f0d0 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f5653578740 10 client.0 _ll_register_callbacks cb 0x7f563c003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f564de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed4f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f564e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed4fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f564e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed4fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:45970/0 (socket says 172.21.15.116:45970) 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.404+0000 7f564e800640 1 -- 172.21.15.116:0/4182442321 learned_addr learned my addr 172.21.15.116:0/4182442321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564d400640 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed4f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564e800640 1 -- 172.21.15.116:0/4182442321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 msgr2=0x55ec5ed4f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564e800640 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed4f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564e800640 1 -- 172.21.15.116:0/4182442321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 msgr2=0x55ec5ed4f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564e800640 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed4f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564e800640 1 -- 172.21.15.116:0/4182442321 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec5edb0b50 con 0x55ec5ed3fac0 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564de00640 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed4f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:12.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f564e800640 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed4fcf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5644009540 tx=0x7f5644005950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c480d7c63bac0b49 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f562f400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f562f400640 1 -- 172.21.15.116:0/4182442321 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644035040 con 0x55ec5ed3fac0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f562f400640 1 -- 172.21.15.116:0/4182442321 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f56440045a0 con 0x55ec5ed3fac0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.405+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f564402a040 con 0x55ec5ed3fac0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f562f400640 1 -- 172.21.15.116:0/4182442321 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f564400f4e0 con 0x55ec5ed3fac0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 10 client.26951 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55ec5edb1e90 con 0x55ec5ed3fac0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f562f400640 1 -- 172.21.15.116:0/4182442321 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f564402a040 con 0x55ec5ed3fac0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f562f400640 1 -- 172.21.15.116:0/4182442321 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f5644019430 con 0x55ec5ed3fac0 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 10 client.26951 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 10 client.26951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 20 client.26951 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed44c30 0x55ec5edb4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 4 client.26951 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:47:12.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.406+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ec5ecac2b0 con 0x55ec5ed44c30 2024-03-04T07:47:12.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.407+0000 7f564de00640 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed44c30 0x55ec5edb4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.407+0000 7f562f400640 10 client.26951 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.407+0000 7f564de00640 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed44c30 0x55ec5edb4640 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34b313aeded6f6f1 server_cookie=ca535b3523c7dc58 in_seq=0 out_seq=0 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f562f400640 1 -- 172.21.15.116:0/4182442321 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ec5ecac2b0 con 0x55ec5ed44c30 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f562f400640 10 client.26951 handle_command_reply: tid=0 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 client.26951 shutdown 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 msgr2=0x55ec5ed4fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed4fcf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5644009540 tx=0x7f5644005950 comp rx=0 tx=0).stop 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 shutdown_connections 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55ec5ed44c30 0x55ec5edb4640 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55ec5ed3fac0 0x55ec5ed4fcf0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec5ed3f0d0 0x55ec5ed4f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f562f400640 0 client.26951 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 --2- 172.21.15.116:0/4182442321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec5ed3a640 0x55ec5ed4f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 >> 172.21.15.116:0/4182442321 conn(0x55ec5ed408a0 msgr2=0x55ec5ed39a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 shutdown_connections 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.408+0000 7f5653578740 1 -- 172.21.15.116:0/4182442321 wait complete. 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5653578740 20 client.26951 trim_cache size 0 max 16384 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "95ac34b6-e8ee-46d7-83ef-be7c043c8b9e", 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:47:12.412 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:47:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56300a1d80 msgr2=0x7f56300a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56300a1d80 0x7f56300a4240 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5644005fd0 tx=0x7f5644009d40 comp rx=0 tx=0).stop 2024-03-04T07:47:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481037f0 msgr2=0x7f5648199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481037f0 0x7f5648199ae0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f563c00e9c0 tx=0x7f563c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:47:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 shutdown_connections 2024-03-04T07:47:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56481053a0 0x7f56481a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56300a1d80 0x7f56300a4240 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56481049e0 0x7f564819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 --2- 172.21.15.116:0/3114514906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56481037f0 0x7f5648199ae0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.409+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 >> 172.21.15.116:0/3114514906 conn(0x7f56480fed80 msgr2=0x7f5648107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.410+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 shutdown_connections 2024-03-04T07:47:12.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.410+0000 7f5650800640 1 -- 172.21.15.116:0/3114514906 wait complete. 2024-03-04T07:47:12.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:47:12.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 -- 172.21.15.116:0/1697492113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41049e0 msgr2=0x7fa7f4104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 --2- 172.21.15.116:0/1697492113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41049e0 0x7fa7f4104e60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009900 tx=0x7fa7e402bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 -- 172.21.15.116:0/1697492113 shutdown_connections 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 --2- 172.21.15.116:0/1697492113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7f41053a0 0x7fa7f410bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 --2- 172.21.15.116:0/1697492113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41049e0 0x7fa7f4104e60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 --2- 172.21.15.116:0/1697492113 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7f41037f0 0x7fa7f4103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 -- 172.21.15.116:0/1697492113 >> 172.21.15.116:0/1697492113 conn(0x7fa7f40fed80 msgr2=0x7fa7f41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.634+0000 7fa7fc400640 1 -- 172.21.15.116:0/1697492113 shutdown_connections 2024-03-04T07:47:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.635+0000 7fa7fc400640 1 -- 172.21.15.116:0/1697492113 wait complete. 2024-03-04T07:47:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.635+0000 7fa7fc400640 1 Processor -- start 2024-03-04T07:47:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fc400640 1 -- start start 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fc400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7f41037f0 0x7fa7f4199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fc400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41053a0 0x7fa7f419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fc400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7f419f210 0x7fa7f41a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fc400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f410f450 con 0x7fa7f41053a0 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fc400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f410f2d0 con 0x7fa7f419f210 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fc400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f410f5d0 con 0x7fa7f41037f0 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7f9000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41053a0 0x7fa7f419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7f9a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7f41037f0 0x7fa7f4199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7f9000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41053a0 0x7fa7f419a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50752/0 (socket says 172.21.15.116:50752) 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7f9000640 1 -- 172.21.15.116:0/806063061 learned_addr learned my addr 172.21.15.116:0/806063061 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.636+0000 7fa7fa400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7f419f210 0x7fa7f41a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f9a00640 1 -- 172.21.15.116:0/806063061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7f419f210 msgr2=0x7fa7f41a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f9a00640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7f419f210 0x7fa7f41a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f9a00640 1 -- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41053a0 msgr2=0x7fa7f419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f9a00640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41053a0 0x7fa7f419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f9a00640 1 -- 172.21.15.116:0/806063061 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f41a1ca0 con 0x7fa7f41037f0 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7fa400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7f419f210 0x7fa7f41a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f9a00640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7f41037f0 0x7fa7f4199c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800c940 tx=0x7fa7e800ce00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f2000640 1 -- 172.21.15.116:0/806063061 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8007bb0 con 0x7fa7f41037f0 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f2000640 1 -- 172.21.15.116:0/806063061 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7e8007d50 con 0x7fa7f41037f0 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f41a1f90 con 0x7fa7f41037f0 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7f2000640 1 -- 172.21.15.116:0/806063061 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8018790 con 0x7fa7f41037f0 2024-03-04T07:47:12.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.637+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f41a23a0 con 0x7fa7f41037f0 2024-03-04T07:47:12.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.640+0000 7fa7ff1e6740 1 Processor -- start 2024-03-04T07:47:12.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7ff1e6740 1 -- start start 2024-03-04T07:47:12.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7f2000640 1 -- 172.21.15.116:0/806063061 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa7e800f450 con 0x7fa7f41037f0 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a6b50 0x5640956a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956ac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956ac6b0 0x5640956aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640955d4090 con 0x5640956a6b50 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56409560bd30 con 0x5640956ac6b0 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640955b4a30 con 0x5640956a7540 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7fa400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a6b50 0x5640956a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7fa400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a6b50 0x5640956a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50764/0 (socket says 172.21.15.116:50764) 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7fa400640 1 -- 172.21.15.116:0/2794014374 learned_addr learned my addr 172.21.15.116:0/2794014374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7f9000640 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956ac6b0 0x5640956aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.641+0000 7fa7f9a00640 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7fa400640 1 -- 172.21.15.116:0/2794014374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 msgr2=0x5640956ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7fa400640 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956ac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7fa400640 1 -- 172.21.15.116:0/2794014374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956ac6b0 msgr2=0x5640956aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7fa400640 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956ac6b0 0x5640956aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7fa400640 1 -- 172.21.15.116:0/2794014374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640953ea7d0 con 0x5640956a6b50 2024-03-04T07:47:12.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7fa400640 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a6b50 0x5640956a6f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0117e0 tx=0x7fa7ec011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eba8b6c57463aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7e2a00640 1 -- 172.21.15.116:0/2794014374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec011310 con 0x5640956a6b50 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7e2a00640 1 -- 172.21.15.116:0/2794014374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7ec003e50 con 0x5640956a6b50 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7f2000640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa7dc0a1d80 0x7fa7dc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.642+0000 7fa7e2a00640 1 -- 172.21.15.116:0/2794014374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec0045c0 con 0x5640956a6b50 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7f2000640 1 -- 172.21.15.116:0/806063061 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa7e80cdd30 con 0x7fa7f41037f0 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7f9a00640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa7dc0a1d80 0x7fa7dc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2794014374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a6b50 msgr2=0x5640956a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a6b50 0x5640956a6f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0117e0 tx=0x7fa7ec011ca0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2794014374 shutdown_connections 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956ac6b0 0x5640956aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2794014374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a6b50 0x5640956a6f70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2794014374 >> 172.21.15.116:0/2794014374 conn(0x5640956a20c0 msgr2=0x5640956a24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2794014374 shutdown_connections 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7f9a00640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa7dc0a1d80 0x7fa7dc0a4240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80a2cd0 tx=0x7fa7e80d17b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.643+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2794014374 wait complete. 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.644+0000 7fa7ff1e6740 1 -- start start 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.644+0000 7fa7ff1e6740 1 Processor -- start 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.644+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.644+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a7540 0x5640956b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.644+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956ac6b0 0x5640956b8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.644+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640955aa0c0 con 0x5640956a7540 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564095607dd0 con 0x5640956a6b50 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640956af660 con 0x5640956ac6b0 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7ff1e6740 10 client.0 _ll_register_callbacks cb 0x5640955c7860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7fa400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7f9000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a7540 0x5640956b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7fa400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956b0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46002/0 (socket says 172.21.15.116:46002) 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7fa400640 1 -- 172.21.15.116:0/2613312503 learned_addr learned my addr 172.21.15.116:0/2613312503 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7fa400640 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956ac6b0 0x5640956b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7f9000640 1 -- 172.21.15.116:0/2613312503 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956ac6b0 msgr2=0x5640956b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7f9000640 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956ac6b0 0x5640956b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7f9000640 1 -- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 msgr2=0x5640956b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7f9000640 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7f9000640 1 -- 172.21.15.116:0/2613312503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640953203f0 con 0x5640956a7540 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7fa400640 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7f9000640 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a7540 0x5640956b14b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402c8e0 tx=0x7fa7e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828bf14ad8ef307c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.645+0000 7fa7d7400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.646+0000 7fa7d7400640 1 -- 172.21.15.116:0/2613312503 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e40046c0 con 0x5640956a7540 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.646+0000 7fa7d7400640 1 -- 172.21.15.116:0/2613312503 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa7e4004860 con 0x5640956a7540 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.646+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7ec003e50 con 0x5640956a7540 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.646+0000 7fa7d7400640 1 -- 172.21.15.116:0/2613312503 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7e4004a60 con 0x5640956a7540 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.646+0000 7fa7ff1e6740 10 client.16839 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.646+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5640952e00f0 con 0x5640956a7540 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.646+0000 7fa7d7400640 1 -- 172.21.15.116:0/2613312503 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7ec003e50 con 0x5640956a7540 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7d7400640 1 -- 172.21.15.116:0/2613312503 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa7e4003710 con 0x5640956a7540 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7ff1e6740 10 client.16839 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7ff1e6740 10 client.16839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7ff1e6740 20 client.16839 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5640956bc9b0 0x5640956bee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7ff1e6740 4 client.16839 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564095619270 con 0x5640956bc9b0 2024-03-04T07:47:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7fa400640 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5640956bc9b0 0x5640956bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7fa400640 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5640956bc9b0 0x5640956bee10 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311e7a0dcb4527bb server_cookie=514a24f7705ab006 in_seq=0 out_seq=0 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.647+0000 7fa7d7400640 10 client.16839 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7d7400640 1 -- 172.21.15.116:0/2613312503 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564095619270 con 0x5640956bc9b0 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7d7400640 10 client.16839 handle_command_reply: tid=0 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 client.16839 shutdown 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a7540 msgr2=0x5640956b14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a7540 0x5640956b14b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402c8e0 tx=0x7fa7e4004270 comp rx=0 tx=0).stop 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 shutdown_connections 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5640956bc9b0 0x5640956bee10 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956ac6b0 0x5640956b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956a7540 0x5640956b14b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7d7400640 0 client.16839 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/2613312503 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 >> 172.21.15.116:0/2613312503 conn(0x5640956a20c0 msgr2=0x5640956a1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.648+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 shutdown_connections 2024-03-04T07:47:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.649+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/2613312503 wait complete. 2024-03-04T07:47:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.649+0000 7fa7ff1e6740 20 client.16839 trim_cache size 0 max 16384 2024-03-04T07:47:12.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 Processor -- start 2024-03-04T07:47:12.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 -- start start 2024-03-04T07:47:12.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956b8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41a9180 con 0x5640956ac6b0 2024-03-04T07:47:12.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41ab760 con 0x5640956a6b50 2024-03-04T07:47:12.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41a9780 con 0x5640956a7540 2024-03-04T07:47:12.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7fa400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7fa400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956a5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46008/0 (socket says 172.21.15.116:46008) 2024-03-04T07:47:12.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7fa400640 1 -- 172.21.15.116:0/1103731861 learned_addr learned my addr 172.21.15.116:0/1103731861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7f9000640 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.666+0000 7fa7fa400640 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7f9000640 1 -- 172.21.15.116:0/1103731861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 msgr2=0x5640956a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7f9000640 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7f9000640 1 -- 172.21.15.116:0/1103731861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 msgr2=0x5640956c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7f9000640 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7f9000640 1 -- 172.21.15.116:0/1103731861 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640956a2a40 con 0x5640956a7540 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7fa400640 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7f9000640 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956b8440 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402c170 tx=0x7fa7e402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=353a8b69584402e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7e2000640 1 -- 172.21.15.116:0/1103731861 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4057020 con 0x5640956a7540 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7e2000640 1 -- 172.21.15.116:0/1103731861 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7e4003c50 con 0x5640956a7540 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.667+0000 7fa7e2000640 1 -- 172.21.15.116:0/1103731861 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4003850 con 0x5640956a7540 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/1103731861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 msgr2=0x5640956b8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956b8440 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402c170 tx=0x7fa7e402bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/1103731861 shutdown_connections 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a7540 0x5640956b8440 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/1103731861 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a6b50 0x5640956a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/1103731861 >> 172.21.15.116:0/1103731861 conn(0x5640956a20c0 msgr2=0x5640956a5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/1103731861 shutdown_connections 2024-03-04T07:47:12.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/1103731861 wait complete. 2024-03-04T07:47:12.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.668+0000 7fa7ff1e6740 1 -- start start 2024-03-04T07:47:12.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 1 Processor -- start 2024-03-04T07:47:12.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a6b50 0x5640956b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a7540 0x5640956b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41a9de0 con 0x5640956ac6b0 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41ab100 con 0x5640956a7540 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41aa440 con 0x5640956a6b50 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7ff1e6740 10 client.0 _ll_register_callbacks cb 0x7fa7e4005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7f9000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a7540 0x5640956b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7f9000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a7540 0x5640956b65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46016/0 (socket says 172.21.15.116:46016) 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7f9000640 1 -- 172.21.15.116:0/4014407406 learned_addr learned my addr 172.21.15.116:0/4014407406 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.669+0000 7fa7fa400640 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a6b50 0x5640956b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7fa400640 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7f9000640 1 -- 172.21.15.116:0/4014407406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a6b50 msgr2=0x5640956b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7f9000640 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a6b50 0x5640956b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7f9000640 1 -- 172.21.15.116:0/4014407406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 msgr2=0x5640956b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:12.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7f9000640 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7f9000640 1 -- 172.21.15.116:0/4014407406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640956b7030 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7f9000640 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a7540 0x5640956b65b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4003b80 tx=0x7fa7e4004e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=806426b7dcc566c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7d7400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7d7400640 1 -- 172.21.15.116:0/4014407406 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4057070 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7d7400640 1 -- 172.21.15.116:0/4014407406 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa7e40050f0 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7e4003c50 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.670+0000 7fa7d7400640 1 -- 172.21.15.116:0/4014407406 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7e4063050 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 10 client.26966 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564095718f30 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7d7400640 1 -- 172.21.15.116:0/4014407406 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7e4003c50 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7d7400640 1 -- 172.21.15.116:0/4014407406 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa7e40489d0 con 0x5640956a7540 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 10 client.26966 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 10 client.26966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 20 client.26966 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564095719820 0x56409571bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 4 client.26966 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5640956135c0 con 0x564095719820 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.671+0000 7fa7fa400640 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564095719820 0x56409571bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7d7400640 10 client.26966 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7fa400640 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564095719820 0x56409571bc40 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dedb6dc6b49e059c server_cookie=b477baf9ef4cc092 in_seq=0 out_seq=0 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7d7400640 1 -- 172.21.15.116:0/4014407406 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5640956135c0 con 0x564095719820 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7d7400640 10 client.26966 handle_command_reply: tid=0 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 client.26966 shutdown 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a7540 msgr2=0x5640956b65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a7540 0x5640956b65b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4003b80 tx=0x7fa7e4004e30 comp rx=0 tx=0).stop 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 shutdown_connections 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564095719820 0x56409571bc40 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640956ac6b0 0x5640956b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5640956a7540 0x5640956b65b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 --2- 172.21.15.116:0/4014407406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640956a6b50 0x5640956b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 >> 172.21.15.116:0/4014407406 conn(0x5640956a20c0 msgr2=0x5640956a5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7d7400640 0 client.26966 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 shutdown_connections 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 1 -- 172.21.15.116:0/4014407406 wait complete. 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.672+0000 7fa7ff1e6740 20 client.26966 trim_cache size 0 max 16384 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:47:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:47:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "95ac34b6-e8ee-46d7-83ef-be7c043c8b9e": { 2024-03-04T07:47:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:47:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "95ac34b6-e8ee-46d7-83ef-be7c043c8b9e", 2024-03-04T07:47:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:47:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:47:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:47:12.678 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa7dc0a1d80 msgr2=0x7fa7dc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa7dc0a1d80 0x7fa7dc0a4240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80a2cd0 tx=0x7fa7e80d17b0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7f41037f0 msgr2=0x7fa7f4199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7f41037f0 0x7fa7f4199c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800c940 tx=0x7fa7e800ce00 comp rx=0 tx=0).stop 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 shutdown_connections 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa7f419f210 0x7fa7f41a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7f41053a0 0x7fa7f419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa7dc0a1d80 0x7fa7dc0a4240 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 --2- 172.21.15.116:0/806063061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7f41037f0 0x7fa7f4199c80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 >> 172.21.15.116:0/806063061 conn(0x7fa7f40fed80 msgr2=0x7fa7f4107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 shutdown_connections 2024-03-04T07:47:12.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:12.673+0000 7fa7fc400640 1 -- 172.21.15.116:0/806063061 wait complete. 2024-03-04T07:47:12.690 INFO:tasks.cephfs.filesystem:scrub status for tag:95ac34b6-e8ee-46d7-83ef-be7c043c8b9e - {'path': '/', 'tag': '95ac34b6-e8ee-46d7-83ef-be7c043c8b9e', 'options': 'recursive,force'} 2024-03-04T07:47:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:12 smithi116 ceph-mon[32150]: pgmap v594: 97 pgs: 97 active+clean; 4.7 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:47:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:12 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:47:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:12 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:12 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:47:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:12 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:13.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:12 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:47:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:12 smithi152 ceph-mon[37154]: pgmap v594: 97 pgs: 97 active+clean; 4.7 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:47:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:12 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:47:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:12 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:12 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:47:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:12 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:12 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:47:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:12 smithi133 ceph-mon[36505]: pgmap v594: 97 pgs: 97 active+clean; 4.7 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:47:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:12 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:47:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:12 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:12 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:47:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:12 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:12 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:47:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:13 smithi116 ceph-mon[32150]: pgmap v595: 97 pgs: 97 active+clean; 768 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:14.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:13 smithi152 ceph-mon[37154]: pgmap v595: 97 pgs: 97 active+clean; 768 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:13 smithi133 ceph-mon[36505]: pgmap v595: 97 pgs: 97 active+clean; 768 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:17.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:16 smithi133 ceph-mon[36505]: pgmap v596: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:47:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:16 smithi116 ceph-mon[32150]: pgmap v596: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:47:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:16 smithi152 ceph-mon[37154]: pgmap v596: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:47:19.074 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:18 smithi152 ceph-mon[37154]: pgmap v597: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:19.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:18 smithi133 ceph-mon[36505]: pgmap v597: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:18 smithi116 ceph-mon[32150]: pgmap v597: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.859+0000 7fb145a00640 1 -- 172.21.15.116:0/577355400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 msgr2=0x7fb140105430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.859+0000 7fb145a00640 1 --2- 172.21.15.116:0/577355400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb140105430 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb130009900 tx=0x7fb13002bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.859+0000 7fb145a00640 1 -- 172.21.15.116:0/577355400 shutdown_connections 2024-03-04T07:47:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.859+0000 7fb145a00640 1 --2- 172.21.15.116:0/577355400 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb140105430 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.859+0000 7fb145a00640 1 --2- 172.21.15.116:0/577355400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140102600 0x7fb140104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.859+0000 7fb145a00640 1 --2- 172.21.15.116:0/577355400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1400ffc90 0x7fb1401020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.859+0000 7fb145a00640 1 -- 172.21.15.116:0/577355400 >> 172.21.15.116:0/577355400 conn(0x7fb1400fb8b0 msgr2=0x7fb1400fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.860+0000 7fb145a00640 1 -- 172.21.15.116:0/577355400 shutdown_connections 2024-03-04T07:47:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.860+0000 7fb145a00640 1 -- 172.21.15.116:0/577355400 wait complete. 2024-03-04T07:47:20.864 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 Processor -- start 2024-03-04T07:47:20.864 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 -- start start 2024-03-04T07:47:20.864 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1400ffc90 0x7fb14019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140102600 0x7fb14019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb1401a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1401136e0 con 0x7fb140102600 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb140113560 con 0x7fb140104fb0 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb140113860 con 0x7fb1400ffc90 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb13f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140102600 0x7fb14019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb1401a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb1401a52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38104/0 (socket says 172.21.15.116:38104) 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb13f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140102600 0x7fb14019e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49258/0 (socket says 172.21.15.116:49258) 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb145000640 1 -- 172.21.15.116:0/3230696639 learned_addr learned my addr 172.21.15.116:0/3230696639 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.861+0000 7fb13fe00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1400ffc90 0x7fb14019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb145000640 1 -- 172.21.15.116:0/3230696639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1400ffc90 msgr2=0x7fb14019dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb145000640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1400ffc90 0x7fb14019dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb145000640 1 -- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140102600 msgr2=0x7fb14019e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb145000640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140102600 0x7fb14019e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb145000640 1 -- 172.21.15.116:0/3230696639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1401a5920 con 0x7fb140104fb0 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb145000640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb1401a52b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb13002c8e0 tx=0x7fb13002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb13cc00640 1 -- 172.21.15.116:0/3230696639 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130043020 con 0x7fb140104fb0 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb13cc00640 1 -- 172.21.15.116:0/3230696639 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1300054e0 con 0x7fb140104fb0 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1401a5bb0 con 0x7fb140104fb0 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.862+0000 7fb13cc00640 1 -- 172.21.15.116:0/3230696639 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130003d00 con 0x7fb140104fb0 2024-03-04T07:47:20.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.863+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1401a6150 con 0x7fb140104fb0 2024-03-04T07:47:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 Processor -- start 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb13cc00640 1 -- 172.21.15.116:0/3230696639 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb130046020 con 0x7fb140104fb0 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 -- start start 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a67a10 0x558a76a67e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a6cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a6fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a769cd370 con 0x558a76a68400 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a76976550 con 0x558a76a6d570 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb149c35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a769c9410 con 0x558a76a67a10 2024-03-04T07:47:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb13fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a6cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb13fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a6cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49266/0 (socket says 172.21.15.116:49266) 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb13fe00640 1 -- 172.21.15.116:0/1164523922 learned_addr learned my addr 172.21.15.116:0/1164523922 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb13f400640 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a67a10 0x558a76a67e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.866+0000 7fb13f400640 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a6fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb13fe00640 1 -- 172.21.15.116:0/1164523922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a67a10 msgr2=0x558a76a67e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb13fe00640 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a67a10 0x558a76a67e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb13fe00640 1 -- 172.21.15.116:0/1164523922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 msgr2=0x558a76a6fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb13fe00640 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a6fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb13fe00640 1 -- 172.21.15.116:0/1164523922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a7698a3c0 con 0x558a76a68400 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb13fe00640 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a6cf00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb138009d20 tx=0x7fb138011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950a66a6e78b85b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb135600640 1 -- 172.21.15.116:0/1164523922 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13801e070 con 0x558a76a68400 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb135600640 1 -- 172.21.15.116:0/1164523922 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb138019b10 con 0x558a76a68400 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.867+0000 7fb135600640 1 -- 172.21.15.116:0/1164523922 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13800eb80 con 0x558a76a68400 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb13cc00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1240a1df0 0x7fb1240a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb13cc00640 1 -- 172.21.15.116:0/3230696639 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb1300efdf0 con 0x7fb140104fb0 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 -- 172.21.15.116:0/1164523922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 msgr2=0x558a76a6cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a6cf00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb138009d20 tx=0x7fb138011810 comp rx=0 tx=0).stop 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb13f400640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1240a1df0 0x7fb1240a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 -- 172.21.15.116:0/1164523922 shutdown_connections 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a6fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a6cf00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 --2- 172.21.15.116:0/1164523922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a67a10 0x558a76a67e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 -- 172.21.15.116:0/1164523922 >> 172.21.15.116:0/1164523922 conn(0x558a76a62f80 msgr2=0x558a76a63380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 -- 172.21.15.116:0/1164523922 shutdown_connections 2024-03-04T07:47:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.868+0000 7fb149c35740 1 -- 172.21.15.116:0/1164523922 wait complete. 2024-03-04T07:47:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb13f400640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1240a1df0 0x7fb1240a42b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x558a76a68c60 tx=0x7fb128009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 -- start start 2024-03-04T07:47:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 Processor -- start 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a67a10 0x558a76a71d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a72270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d570 0x558a76a792b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a76a70520 con 0x558a76a68400 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a769899b0 con 0x558a76a67a10 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a76a706a0 con 0x558a76a6d570 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.869+0000 7fb149c35740 10 client.0 _ll_register_callbacks cb 0x558a769c5340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a67a10 0x558a76a71d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a67a10 0x558a76a71d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38124/0 (socket says 172.21.15.116:38124) 2024-03-04T07:47:20.877 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 -- 172.21.15.116:0/344645889 learned_addr learned my addr 172.21.15.116:0/344645889 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13f400640 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a72270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb145000640 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d570 0x558a76a792b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 -- 172.21.15.116:0/344645889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d570 msgr2=0x558a76a792b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d570 0x558a76a792b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 -- 172.21.15.116:0/344645889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 msgr2=0x558a76a72270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a72270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 -- 172.21.15.116:0/344645889 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a766e2210 con 0x558a76a67a10 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb13fe00640 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a67a10 0x558a76a71d30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb138009cf0 tx=0x7fb138019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40a008e212cf8037 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.870+0000 7fb12e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb12e000640 1 -- 172.21.15.116:0/344645889 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138027020 con 0x558a76a67a10 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb12e000640 1 -- 172.21.15.116:0/344645889 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb138005250 con 0x558a76a67a10 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb138019b10 con 0x558a76a67a10 2024-03-04T07:47:20.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb12e000640 1 -- 172.21.15.116:0/344645889 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb13801e070 con 0x558a76a67a10 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb149c35740 10 client.26978 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558a767ae620 con 0x558a76a67a10 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb12e000640 1 -- 172.21.15.116:0/344645889 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb138019b10 con 0x558a76a67a10 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb12e000640 1 -- 172.21.15.116:0/344645889 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb1380046c0 con 0x558a76a67a10 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb149c35740 10 client.26978 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb149c35740 10 client.26978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.871+0000 7fb149c35740 20 client.26978 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.872+0000 7fb149c35740 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76a7d850 0x558a76a7fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.872+0000 7fb149c35740 4 client.26978 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.872+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a769da8b0 con 0x558a76a7d850 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.872+0000 7fb13f400640 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76a7d850 0x558a76a7fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.872+0000 7fb13f400640 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76a7d850 0x558a76a7fca0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6292526dde7b8981 server_cookie=5f0eec6a9d4c8009 in_seq=0 out_seq=0 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.872+0000 7fb12e000640 10 client.26978 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:20.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.873+0000 7fb12e000640 1 -- 172.21.15.116:0/344645889 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558a769da8b0 con 0x558a76a7d850 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.873+0000 7fb12e000640 10 client.26978 handle_command_reply: tid=0 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.873+0000 7fb149c35740 1 client.26978 shutdown 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.873+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a67a10 msgr2=0x558a76a71d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.873+0000 7fb149c35740 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a67a10 0x558a76a71d30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb138009cf0 tx=0x7fb138019660 comp rx=0 tx=0).stop 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.873+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 shutdown_connections 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76a7d850 0x558a76a7fca0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d570 0x558a76a792b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a72270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 1 --2- 172.21.15.116:0/344645889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a67a10 0x558a76a71d30 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb12e000640 0 client.26978 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 >> 172.21.15.116:0/344645889 conn(0x558a76a62f80 msgr2=0x558a76a62200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:20.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 shutdown_connections 2024-03-04T07:47:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 1 -- 172.21.15.116:0/344645889 wait complete. 2024-03-04T07:47:20.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.874+0000 7fb149c35740 20 client.26978 trim_cache size 0 max 16384 2024-03-04T07:47:20.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 Processor -- start 2024-03-04T07:47:20.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 -- start start 2024-03-04T07:47:20.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a845a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d180 0x558a76a84ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1401acea0 con 0x558a76a68400 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1401af480 con 0x558a76a6d570 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb149c35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1401ad4a0 con 0x558a76a6d180 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb13f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb13fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a845a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb13f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38134/0 (socket says 172.21.15.116:38134) 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb13f400640 1 -- 172.21.15.116:0/2578249034 learned_addr learned my addr 172.21.15.116:0/2578249034 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.889+0000 7fb145000640 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d180 0x558a76a84ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb13f400640 1 -- 172.21.15.116:0/2578249034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d180 msgr2=0x558a76a84ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb13f400640 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d180 0x558a76a84ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb13f400640 1 -- 172.21.15.116:0/2578249034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 msgr2=0x558a76a845a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb13f400640 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a845a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb13f400640 1 -- 172.21.15.116:0/2578249034 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a76a80320 con 0x558a76a6d570 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb13fe00640 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a845a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb13f400640 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a791d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb14019f210 tx=0x7fb128004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0b910daa92160b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb134c00640 1 -- 172.21.15.116:0/2578249034 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1280228d0 con 0x558a76a6d570 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb134c00640 1 -- 172.21.15.116:0/2578249034 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb128033070 con 0x558a76a6d570 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.890+0000 7fb134c00640 1 -- 172.21.15.116:0/2578249034 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1280213f0 con 0x558a76a6d570 2024-03-04T07:47:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 -- 172.21.15.116:0/2578249034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 msgr2=0x558a76a791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a791d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb14019f210 tx=0x7fb128004ab0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 -- 172.21.15.116:0/2578249034 shutdown_connections 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a6d180 0x558a76a84ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a791d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 --2- 172.21.15.116:0/2578249034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a68400 0x558a76a845a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 -- 172.21.15.116:0/2578249034 >> 172.21.15.116:0/2578249034 conn(0x558a76a62f80 msgr2=0x558a76a66a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 -- 172.21.15.116:0/2578249034 shutdown_connections 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 -- 172.21.15.116:0/2578249034 wait complete. 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 -- start start 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.891+0000 7fb149c35740 1 Processor -- start 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb149c35740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a68400 0x558a76a78db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb149c35740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a77410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb149c35740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a77950 0x558a76a77dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb149c35740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1401adb00 con 0x558a76a77950 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb149c35740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1401aee20 con 0x558a76a6d570 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb149c35740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1401ae160 con 0x558a76a68400 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb149c35740 10 client.0 _ll_register_callbacks cb 0x7fb13801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb145000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a77950 0x558a76a77dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a77410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb145000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a77950 0x558a76a77dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49290/0 (socket says 172.21.15.116:49290) 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a77410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38138/0 (socket says 172.21.15.116:38138) 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb145000640 1 -- 172.21.15.116:0/979528863 learned_addr learned my addr 172.21.15.116:0/979528863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13fe00640 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a68400 0x558a76a78db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13f400640 1 -- 172.21.15.116:0/979528863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a68400 msgr2=0x558a76a78db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13f400640 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a68400 0x558a76a78db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13f400640 1 -- 172.21.15.116:0/979528863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a77950 msgr2=0x558a76a77dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13f400640 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a77950 0x558a76a77dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb13f400640 1 -- 172.21.15.116:0/979528863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a76a79540 con 0x558a76a6d570 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.892+0000 7fb145000640 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a77950 0x558a76a77dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb13f400640 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a77410 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb1280049d0 tx=0x7fb128004260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475abde58f68a4a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb12e000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb12e000640 1 -- 172.21.15.116:0/979528863 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12803b030 con 0x558a76a6d570 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb12e000640 1 -- 172.21.15.116:0/979528863 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb12801a040 con 0x558a76a6d570 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb128033070 con 0x558a76a6d570 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb12e000640 1 -- 172.21.15.116:0/979528863 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb12802e740 con 0x558a76a6d570 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb149c35740 10 client.26984 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558a76ada290 con 0x558a76a6d570 2024-03-04T07:47:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.893+0000 7fb12e000640 1 -- 172.21.15.116:0/979528863 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb128033070 con 0x558a76a6d570 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb12e000640 1 -- 172.21.15.116:0/979528863 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb12801d070 con 0x558a76a6d570 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb149c35740 10 client.26984 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb149c35740 10 client.26984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb149c35740 20 client.26984 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb149c35740 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76adac10 0x558a76add030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb149c35740 4 client.26984 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558a769d4c00 con 0x558a76adac10 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb13fe00640 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76adac10 0x558a76add030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb13fe00640 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76adac10 0x558a76add030 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4ad534ecbbdf64c server_cookie=843e3d66136b44ad in_seq=0 out_seq=0 2024-03-04T07:47:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.894+0000 7fb12e000640 10 client.26984 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.907+0000 7fb12e000640 1 -- 172.21.15.116:0/979528863 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558a769d4c00 con 0x558a76adac10 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.907+0000 7fb12e000640 10 client.26984 handle_command_reply: tid=0 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.907+0000 7fb149c35740 1 client.26984 shutdown 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.907+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 msgr2=0x558a76a77410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.907+0000 7fb149c35740 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a77410 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb1280049d0 tx=0x7fb128004260 comp rx=0 tx=0).stop 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 shutdown_connections 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a76adac10 0x558a76add030 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a76a77950 0x558a76a77dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a76a6d570 0x558a76a77410 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 --2- 172.21.15.116:0/979528863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a76a68400 0x558a76a78db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb12e000640 0 client.26984 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 >> 172.21.15.116:0/979528863 conn(0x558a76a62f80 msgr2=0x558a76a61970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 shutdown_connections 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 1 -- 172.21.15.116:0/979528863 wait complete. 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.908+0000 7fb149c35740 20 client.26984 trim_cache size 0 max 16384 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:47:20.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:47:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:47:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1240a1df0 msgr2=0x7fb1240a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1240a1df0 0x7fb1240a42b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x558a76a68c60 tx=0x7fb128009500 comp rx=0 tx=0).stop 2024-03-04T07:47:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 msgr2=0x7fb1401a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:20.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb1401a52b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb13002c8e0 tx=0x7fb13002c640 comp rx=0 tx=0).stop 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 shutdown_connections 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb140104fb0 0x7fb1401a52b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb140102600 0x7fb14019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb1240a1df0 0x7fb1240a42b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 --2- 172.21.15.116:0/3230696639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1400ffc90 0x7fb14019dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 >> 172.21.15.116:0/3230696639 conn(0x7fb1400fb8b0 msgr2=0x7fb140100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 shutdown_connections 2024-03-04T07:47:20.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:20.909+0000 7fb145a00640 1 -- 172.21.15.116:0/3230696639 wait complete. 2024-03-04T07:47:21.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:20 smithi133 ceph-mon[36505]: pgmap v598: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:47:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:20 smithi116 ceph-mon[32150]: pgmap v598: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:47:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:20 smithi152 ceph-mon[37154]: pgmap v598: 97 pgs: 97 active+clean; 830 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:47:23.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:22 smithi133 ceph-mon[36505]: pgmap v599: 97 pgs: 97 active+clean; 888 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:22 smithi116 ceph-mon[32150]: pgmap v599: 97 pgs: 97 active+clean; 888 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:22 smithi152 ceph-mon[37154]: pgmap v599: 97 pgs: 97 active+clean; 888 KiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:25.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:24 smithi133 ceph-mon[36505]: pgmap v600: 97 pgs: 97 active+clean; 888 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:47:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:24 smithi116 ceph-mon[32150]: pgmap v600: 97 pgs: 97 active+clean; 888 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:47:25.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:24 smithi152 ceph-mon[37154]: pgmap v600: 97 pgs: 97 active+clean; 888 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:47:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:26 smithi133 ceph-mon[36505]: pgmap v601: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:26 smithi116 ceph-mon[32150]: pgmap v601: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:27.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:26 smithi152 ceph-mon[37154]: pgmap v601: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:28 smithi133 ceph-mon[36505]: pgmap v602: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:47:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:28 smithi116 ceph-mon[32150]: pgmap v602: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:47:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:28 smithi152 ceph-mon[37154]: pgmap v602: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:47:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:30 smithi133 ceph-mon[36505]: pgmap v603: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:30 smithi116 ceph-mon[32150]: pgmap v603: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:30 smithi152 ceph-mon[37154]: pgmap v603: 97 pgs: 97 active+clean; 960 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:33.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:32 smithi133 ceph-mon[36505]: pgmap v604: 97 pgs: 97 active+clean; 1.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:32 smithi116 ceph-mon[32150]: pgmap v604: 97 pgs: 97 active+clean; 1.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:32 smithi152 ceph-mon[37154]: pgmap v604: 97 pgs: 97 active+clean; 1.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.051+0000 7fefae800640 1 -- 172.21.15.116:0/588405483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 msgr2=0x7fefa8103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.051+0000 7fefae800640 1 --2- 172.21.15.116:0/588405483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8103c50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fef98009900 tx=0x7fef9802bc50 comp rx=0 tx=0).stop 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.051+0000 7fefae800640 1 -- 172.21.15.116:0/588405483 shutdown_connections 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.051+0000 7fefae800640 1 --2- 172.21.15.116:0/588405483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fefa81053e0 0x7fefa810bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.051+0000 7fefae800640 1 --2- 172.21.15.116:0/588405483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefa8104a20 0x7fefa8104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.051+0000 7fefae800640 1 --2- 172.21.15.116:0/588405483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8103c50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.051+0000 7fefae800640 1 -- 172.21.15.116:0/588405483 >> 172.21.15.116:0/588405483 conn(0x7fefa80fedc0 msgr2=0x7fefa81011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.052+0000 7fefae800640 1 -- 172.21.15.116:0/588405483 shutdown_connections 2024-03-04T07:47:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.052+0000 7fefae800640 1 -- 172.21.15.116:0/588405483 wait complete. 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.052+0000 7fefae800640 1 Processor -- start 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefae800640 1 -- start start 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefae800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefae800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fefa8104a20 0x7fefa819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefae800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefa81053e0 0x7fefa81a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefae800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa810f4f0 con 0x7fefa8103830 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefae800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa810f370 con 0x7fefa8104a20 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefae800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa810f670 con 0x7fefa81053e0 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefa6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fefa8104a20 0x7fefa819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50070/0 (socket says 172.21.15.116:50070) 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefa7400640 1 -- 172.21.15.116:0/423788244 learned_addr learned my addr 172.21.15.116:0/423788244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.053+0000 7fefa7e00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefa81053e0 0x7fefa81a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa6a00640 1 -- 172.21.15.116:0/423788244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefa81053e0 msgr2=0x7fefa81a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa6a00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefa81053e0 0x7fefa81a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa6a00640 1 -- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 msgr2=0x7fefa8199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa6a00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa6a00640 1 -- 172.21.15.116:0/423788244 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa81a1730 con 0x7fefa8104a20 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa7e00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefa81053e0 0x7fefa81a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa7400640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefa6a00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fefa8104a20 0x7fefa819a050 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fef9400e9f0 tx=0x7fef9400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fef9fe00640 1 -- 172.21.15.116:0/423788244 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400ccc0 con 0x7fefa8104a20 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fef9fe00640 1 -- 172.21.15.116:0/423788244 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef94005020 con 0x7fefa8104a20 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa81a1a20 con 0x7fefa8104a20 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fef9fe00640 1 -- 172.21.15.116:0/423788244 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef940049e0 con 0x7fefa8104a20 2024-03-04T07:47:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.054+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa81a1f70 con 0x7fefa8104a20 2024-03-04T07:47:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.057+0000 7fefb15c7740 1 Processor -- start 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefb15c7740 1 -- start start 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fef9fe00640 1 -- 172.21.15.116:0/423788244 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fef94005560 con 0x7fefa8104a20 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfafc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfb4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfb7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefb15c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56026bf15370 con 0x56026bfaf850 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefb15c7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56026bebe550 con 0x56026bfb53b0 2024-03-04T07:47:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefb15c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56026bf11410 con 0x56026bfb0240 2024-03-04T07:47:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefa7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfafc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefa6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfb7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefa7400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfb4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefa7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfafc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50074/0 (socket says 172.21.15.116:50074) 2024-03-04T07:47:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.058+0000 7fefa7e00640 1 -- 172.21.15.116:0/334437998 learned_addr learned my addr 172.21.15.116:0/334437998 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fefa7400640 1 -- 172.21.15.116:0/334437998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 msgr2=0x56026bfb7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fefa7400640 1 --2- 172.21.15.116:0/334437998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfb7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fefa7400640 1 -- 172.21.15.116:0/334437998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 msgr2=0x56026bfafc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fefa7400640 1 --2- 172.21.15.116:0/334437998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfafc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fefa7400640 1 -- 172.21.15.116:0/334437998 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56026bed23c0 con 0x56026bfb0240 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fefa7400640 1 --2- 172.21.15.116:0/334437998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfb4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fef98006010 tx=0x7fef98009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c916681097e0005 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fef9cc00640 1 -- 172.21.15.116:0/334437998 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98042070 con 0x56026bfb0240 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fef9cc00640 1 -- 172.21.15.116:0/334437998 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef9802cd50 con 0x56026bfb0240 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fef9cc00640 1 -- 172.21.15.116:0/334437998 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef980052b0 con 0x56026bfb0240 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.059+0000 7fef9fe00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fef8c0a1ea0 0x7fef8c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fef9fe00640 1 -- 172.21.15.116:0/423788244 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fef940cdd50 con 0x7fefa8104a20 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefa7e00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fef8c0a1ea0 0x7fef8c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 -- 172.21.15.116:0/334437998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 msgr2=0x56026bfb4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 --2- 172.21.15.116:0/334437998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfb4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fef98006010 tx=0x7fef98009560 comp rx=0 tx=0).stop 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 -- 172.21.15.116:0/334437998 shutdown_connections 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 --2- 172.21.15.116:0/334437998 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfb7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 --2- 172.21.15.116:0/334437998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfb4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 --2- 172.21.15.116:0/334437998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfafc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 -- 172.21.15.116:0/334437998 >> 172.21.15.116:0/334437998 conn(0x56026bfaadc0 msgr2=0x56026bfab1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 -- 172.21.15.116:0/334437998 shutdown_connections 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.060+0000 7fefb15c7740 1 -- 172.21.15.116:0/334437998 wait complete. 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.061+0000 7fefa7e00640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fef8c0a1ea0 0x7fef8c0a4360 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x56026bfb0aa0 tx=0x7fefa0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.061+0000 7fefb15c7740 1 -- start start 2024-03-04T07:47:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.061+0000 7fefb15c7740 1 Processor -- start 2024-03-04T07:47:34.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfb9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfba060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfc10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefb15c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56026bfb8300 con 0x56026bfaf850 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefb15c7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56026bed19b0 con 0x56026bfb53b0 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefb15c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56026bfb8480 con 0x56026bfb0240 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefb15c7740 10 client.0 _ll_register_callbacks cb 0x56026bf0d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfb9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfb9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50084/0 (socket says 172.21.15.116:50084) 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7400640 1 -- 172.21.15.116:0/1405694193 learned_addr learned my addr 172.21.15.116:0/1405694193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7e00640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfc10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa6a00640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfba060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7e00640 1 -- 172.21.15.116:0/1405694193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 msgr2=0x56026bfba060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7e00640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfba060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7e00640 1 -- 172.21.15.116:0/1405694193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 msgr2=0x56026bfb9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7e00640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfb9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7e00640 1 -- 172.21.15.116:0/1405694193 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56026bc2a210 con 0x56026bfb53b0 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.062+0000 7fefa7400640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfb9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fefa7e00640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfc10a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fefa0012010 tx=0x7fefa0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30237af1922bd2fc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fef91600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:34.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fef91600640 1 -- 172.21.15.116:0/1405694193 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa002c060 con 0x56026bfb53b0 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fef91600640 1 -- 172.21.15.116:0/1405694193 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fefa0017030 con 0x56026bfb53b0 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef9802cd50 con 0x56026bfb53b0 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fef91600640 1 -- 172.21.15.116:0/1405694193 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefa001fc50 con 0x56026bfb53b0 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fefb15c7740 10 client.26996 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56026bcf6620 con 0x56026bfb53b0 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.063+0000 7fef91600640 1 -- 172.21.15.116:0/1405694193 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef9802cd50 con 0x56026bfb53b0 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fef91600640 1 -- 172.21.15.116:0/1405694193 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fefa0031070 con 0x56026bfb53b0 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefb15c7740 10 client.26996 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefb15c7740 10 client.26996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefb15c7740 20 client.26996 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026bfc5930 0x56026bfc7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefb15c7740 4 client.26996 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56026bf228b0 con 0x56026bfc5930 2024-03-04T07:47:34.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefa7400640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026bfc5930 0x56026bfc7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fefa7400640 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026bfc5930 0x56026bfc7da0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6c741fe2c61cc47 server_cookie=445be20ac73c0340 in_seq=0 out_seq=0 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.064+0000 7fef91600640 10 client.26996 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fef91600640 1 -- 172.21.15.116:0/1405694193 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56026bf228b0 con 0x56026bfc5930 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fef91600640 10 client.26996 handle_command_reply: tid=0 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 client.26996 shutdown 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 msgr2=0x56026bfc10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfc10a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fefa0012010 tx=0x7fefa0004ab0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 shutdown_connections 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026bfc5930 0x56026bfc7da0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb53b0 0x56026bfc10a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fef91600640 0 client.26996 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb0240 0x56026bfba060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1405694193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfb9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 >> 172.21.15.116:0/1405694193 conn(0x56026bfaadc0 msgr2=0x56026bfa9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 shutdown_connections 2024-03-04T07:47:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 1 -- 172.21.15.116:0/1405694193 wait complete. 2024-03-04T07:47:34.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.066+0000 7fefb15c7740 20 client.26996 trim_cache size 0 max 16384 2024-03-04T07:47:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.081+0000 7fefb15c7740 1 Processor -- start 2024-03-04T07:47:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.081+0000 7fefb15c7740 1 -- start start 2024-03-04T07:47:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfaf850 0x56026bfa9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfc1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfb53b0 0x56026bfa9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefb15c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa81a8c10 con 0x56026bfb53b0 2024-03-04T07:47:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefb15c7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa81ab1f0 con 0x56026bfb0240 2024-03-04T07:47:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefb15c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa81a9210 con 0x56026bfaf850 2024-03-04T07:47:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfc1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfc1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42604/0 (socket says 172.21.15.116:42604) 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 -- 172.21.15.116:0/2830751501 learned_addr learned my addr 172.21.15.116:0/2830751501 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa7e00640 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfb53b0 0x56026bfa9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa7400640 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfaf850 0x56026bfa9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 -- 172.21.15.116:0/2830751501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfaf850 msgr2=0x56026bfa9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfaf850 0x56026bfa9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 -- 172.21.15.116:0/2830751501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfb53b0 msgr2=0x56026bfa9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfb53b0 0x56026bfa9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 -- 172.21.15.116:0/2830751501 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56026bfc8420 con 0x56026bfb0240 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa7e00640 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfb53b0 0x56026bfa9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.082+0000 7fefa6a00640 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfc1060 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fef94009720 tx=0x7fef940d1790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57dc24130c198736 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fef93e00640 1 -- 172.21.15.116:0/2830751501 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9401c040 con 0x56026bfb0240 2024-03-04T07:47:34.088 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fef93e00640 1 -- 172.21.15.116:0/2830751501 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef940051d0 con 0x56026bfb0240 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fef93e00640 1 -- 172.21.15.116:0/2830751501 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef940970b0 con 0x56026bfb0240 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 -- 172.21.15.116:0/2830751501 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 msgr2=0x56026bfc1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfc1060 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fef94009720 tx=0x7fef940d1790 comp rx=0 tx=0).stop 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 -- 172.21.15.116:0/2830751501 shutdown_connections 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfb53b0 0x56026bfa9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfc1060 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 --2- 172.21.15.116:0/2830751501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfaf850 0x56026bfa9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 -- 172.21.15.116:0/2830751501 >> 172.21.15.116:0/2830751501 conn(0x56026bfaadc0 msgr2=0x56026bfab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 -- 172.21.15.116:0/2830751501 shutdown_connections 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.083+0000 7fefb15c7740 1 -- 172.21.15.116:0/2830751501 wait complete. 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 -- start start 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 Processor -- start 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfbf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfbf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb53b0 0x56026bfc0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa81a9870 con 0x56026bfaf850 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa81aab90 con 0x56026bfb0240 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa81a9ed0 con 0x56026bfb53b0 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.084+0000 7fefb15c7740 10 client.0 _ll_register_callbacks cb 0x7fefa0010ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfbf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfbf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfbf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42606/0 (socket says 172.21.15.116:42606) 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 -- 172.21.15.116:0/1407719296 learned_addr learned my addr 172.21.15.116:0/1407719296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa7e00640 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb53b0 0x56026bfc0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 -- 172.21.15.116:0/1407719296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb53b0 msgr2=0x56026bfc0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb53b0 0x56026bfc0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 -- 172.21.15.116:0/1407719296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 msgr2=0x56026bfbf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfbf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 -- 172.21.15.116:0/1407719296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56026bfc0b40 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefa6a00640 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfbf540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fef94083fd0 tx=0x7fef940d15b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d270eec41ed6b901 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fef91600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fef91600640 1 -- 172.21.15.116:0/1407719296 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef940b7020 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fef91600640 1 -- 172.21.15.116:0/1407719296 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fef940a2f50 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.085+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef940051d0 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fef91600640 1 -- 172.21.15.116:0/1407719296 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef9409aa40 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 10 client.27002 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56026c021b50 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fef91600640 1 -- 172.21.15.116:0/1407719296 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef940051d0 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fef91600640 1 -- 172.21.15.116:0/1407719296 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fef94099900 con 0x56026bfb0240 2024-03-04T07:47:34.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 10 client.27002 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 10 client.27002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 20 client.27002 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026c022480 0x56026c0248a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 4 client.27002 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56026bf1cc00 con 0x56026c022480 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.086+0000 7fefa7400640 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026c022480 0x56026c0248a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.087+0000 7fefa7400640 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026c022480 0x56026c0248a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da7f7ff1675700ec server_cookie=e85983c77852eea3 in_seq=0 out_seq=0 2024-03-04T07:47:34.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.087+0000 7fef91600640 10 client.27002 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.101+0000 7fef91600640 1 -- 172.21.15.116:0/1407719296 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56026bf1cc00 con 0x56026c022480 2024-03-04T07:47:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.101+0000 7fef91600640 10 client.27002 handle_command_reply: tid=0 2024-03-04T07:47:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.101+0000 7fefb15c7740 1 client.27002 shutdown 2024-03-04T07:47:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 msgr2=0x56026bfbf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfbf540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fef94083fd0 tx=0x7fef940d15b0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 shutdown_connections 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56026c022480 0x56026c0248a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56026bfb53b0 0x56026bfc0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56026bfb0240 0x56026bfbf540 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 --2- 172.21.15.116:0/1407719296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56026bfaf850 0x56026bfbf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 >> 172.21.15.116:0/1407719296 conn(0x56026bfaadc0 msgr2=0x56026bfab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fef91600640 0 client.27002 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 shutdown_connections 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 1 -- 172.21.15.116:0/1407719296 wait complete. 2024-03-04T07:47:34.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.102+0000 7fefb15c7740 20 client.27002 trim_cache size 0 max 16384 2024-03-04T07:47:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:47:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:47:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:47:34.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fef8c0a1ea0 msgr2=0x7fef8c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fef8c0a1ea0 0x7fef8c0a4360 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x56026bfb0aa0 tx=0x7fefa0009500 comp rx=0 tx=0).stop 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fefa8104a20 msgr2=0x7fefa819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fefa8104a20 0x7fefa819a050 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fef9400e9f0 tx=0x7fef9400eeb0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 shutdown_connections 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fef8c0a1ea0 0x7fef8c0a4360 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fefa81053e0 0x7fefa81a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fefa8104a20 0x7fefa819a050 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 --2- 172.21.15.116:0/423788244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa8103830 0x7fefa8199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:34.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 >> 172.21.15.116:0/423788244 conn(0x7fefa80fedc0 msgr2=0x7fefa8107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:34.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 shutdown_connections 2024-03-04T07:47:34.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:34.103+0000 7fefae800640 1 -- 172.21.15.116:0/423788244 wait complete. 2024-03-04T07:47:35.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:34 smithi133 ceph-mon[36505]: pgmap v605: 97 pgs: 97 active+clean; 1.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:34 smithi116 ceph-mon[32150]: pgmap v605: 97 pgs: 97 active+clean; 1.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:34 smithi152 ceph-mon[37154]: pgmap v605: 97 pgs: 97 active+clean; 1.0 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:36 smithi133 ceph-mon[36505]: pgmap v606: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:47:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:36 smithi116 ceph-mon[32150]: pgmap v606: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:47:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:36 smithi152 ceph-mon[37154]: pgmap v606: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:47:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:38 smithi133 ceph-mon[36505]: pgmap v607: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:38 smithi116 ceph-mon[32150]: pgmap v607: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:38 smithi152 ceph-mon[37154]: pgmap v607: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:40 smithi133 ceph-mon[36505]: pgmap v608: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:40 smithi116 ceph-mon[32150]: pgmap v608: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:40 smithi152 ceph-mon[37154]: pgmap v608: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:47:42.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:47:42.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:47:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 -- 172.21.15.116:0/3112526642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c075f90 msgr2=0x7fd97c0763b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 --2- 172.21.15.116:0/3112526642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c075f90 0x7fd97c0763b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd964009900 tx=0x7fd96402bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 -- 172.21.15.116:0/3112526642 shutdown_connections 2024-03-04T07:47:42.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 --2- 172.21.15.116:0/3112526642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c077b70 0x7fd97c078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 --2- 172.21.15.116:0/3112526642 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd97c0771d0 0x7fd97c077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 --2- 172.21.15.116:0/3112526642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c075f90 0x7fd97c0763b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 -- 172.21.15.116:0/3112526642 >> 172.21.15.116:0/3112526642 conn(0x7fd97c0fed80 msgr2=0x7fd97c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 -- 172.21.15.116:0/3112526642 shutdown_connections 2024-03-04T07:47:42.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.894+0000 7fd982200640 1 -- 172.21.15.116:0/3112526642 wait complete. 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.895+0000 7fd982200640 1 Processor -- start 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.895+0000 7fd982200640 1 -- start start 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.895+0000 7fd982200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c075f90 0x7fd97c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd982200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c0771d0 0x7fd97c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd982200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd97c077b70 0x7fd97c1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd982200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1138e0 con 0x7fd97c075f90 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd982200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c113760 con 0x7fd97c077b70 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd982200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c113a60 con 0x7fd97c0771d0 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c075f90 0x7fd97c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97aa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c0771d0 0x7fd97c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97aa00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c0771d0 0x7fd97c19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:50228/0 (socket says 172.21.15.116:50228) 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97aa00640 1 -- 172.21.15.116:0/3512001005 learned_addr learned my addr 172.21.15.116:0/3512001005 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97be00640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd97c077b70 0x7fd97c1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97b400640 1 -- 172.21.15.116:0/3512001005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c0771d0 msgr2=0x7fd97c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97b400640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c0771d0 0x7fd97c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97b400640 1 -- 172.21.15.116:0/3512001005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd97c077b70 msgr2=0x7fd97c1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97b400640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd97c077b70 0x7fd97c1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97b400640 1 -- 172.21.15.116:0/3512001005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c1a5b00 con 0x7fd97c075f90 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97aa00640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c0771d0 0x7fd97c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.896+0000 7fd97be00640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd97c077b70 0x7fd97c1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.897+0000 7fd97b400640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c075f90 0x7fd97c19dee0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd964036040 tx=0x7fd96402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.897+0000 7fd973e00640 1 -- 172.21.15.116:0/3512001005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964043020 con 0x7fd97c075f90 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.897+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c1a5d90 con 0x7fd97c075f90 2024-03-04T07:47:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.897+0000 7fd973e00640 1 -- 172.21.15.116:0/3512001005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9640054e0 con 0x7fd97c075f90 2024-03-04T07:47:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.897+0000 7fd973e00640 1 -- 172.21.15.116:0/3512001005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964003c30 con 0x7fd97c075f90 2024-03-04T07:47:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.897+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c1a6250 con 0x7fd97c075f90 2024-03-04T07:47:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.900+0000 7fd973e00640 1 -- 172.21.15.116:0/3512001005 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd964046020 con 0x7fd97c075f90 2024-03-04T07:47:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 Processor -- start 2024-03-04T07:47:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 -- start start 2024-03-04T07:47:42.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612ba040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612bcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648611e2090 con 0x5648612ba6b0 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564861219d30 con 0x5648612b5540 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd984f89740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648611c2a30 con 0x5648612b4b50 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd97b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612ba040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.901+0000 7fd97be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:50230/0 (socket says 172.21.15.116:50230) 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 -- 172.21.15.116:0/3155749693 learned_addr learned my addr 172.21.15.116:0/3155749693 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97aa00640 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612bcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 -- 172.21.15.116:0/3155749693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 msgr2=0x5648612ba040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612ba040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 -- 172.21.15.116:0/3155749693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 msgr2=0x5648612bcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612bcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 -- 172.21.15.116:0/3155749693 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564860ff87d0 con 0x5648612b4b50 2024-03-04T07:47:42.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd973e00640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9600a1e00 0x7fd9600a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97be00640 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612b4f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd9740117e0 tx=0x7fd974011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b39995258249f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:42.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd973e00640 1 -- 172.21.15.116:0/3512001005 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd9640efd80 con 0x7fd97c075f90 2024-03-04T07:47:42.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.902+0000 7fd97b400640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9600a1e00 0x7fd9600a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.903+0000 7fd96be00640 1 -- 172.21.15.116:0/3155749693 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974011310 con 0x5648612b4b50 2024-03-04T07:47:42.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.903+0000 7fd96be00640 1 -- 172.21.15.116:0/3155749693 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd974003e50 con 0x5648612b4b50 2024-03-04T07:47:42.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.903+0000 7fd96be00640 1 -- 172.21.15.116:0/3155749693 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974004620 con 0x5648612b4b50 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.903+0000 7fd984f89740 1 -- 172.21.15.116:0/3155749693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 msgr2=0x5648612b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.903+0000 7fd984f89740 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612b4f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd9740117e0 tx=0x7fd974011ca0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.903+0000 7fd97b400640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9600a1e00 0x7fd9600a42c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd9640bc770 tx=0x7fd9640f37c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.903+0000 7fd984f89740 1 -- 172.21.15.116:0/3155749693 shutdown_connections 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612bcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612ba040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 --2- 172.21.15.116:0/3155749693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612b4f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 -- 172.21.15.116:0/3155749693 >> 172.21.15.116:0/3155749693 conn(0x5648612b00c0 msgr2=0x5648612b04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 -- 172.21.15.116:0/3155749693 shutdown_connections 2024-03-04T07:47:42.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 -- 172.21.15.116:0/3155749693 wait complete. 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 -- start start 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.904+0000 7fd984f89740 1 Processor -- start 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd984f89740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612beff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd984f89740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd984f89740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd984f89740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648611b80c0 con 0x5648612b5540 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd984f89740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564861215dd0 con 0x5648612b4b50 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd984f89740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648612bd660 con 0x5648612ba6b0 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd984f89740 10 client.0 _ll_register_callbacks cb 0x5648611d5860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd97be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612beff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd97be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612beff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49968/0 (socket says 172.21.15.116:49968) 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd97aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd97be00640 1 -- 172.21.15.116:0/2747332158 learned_addr learned my addr 172.21.15.116:0/2747332158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:42.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.905+0000 7fd97be00640 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd97aa00640 1 -- 172.21.15.116:0/2747332158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 msgr2=0x5648612c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd97aa00640 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd97aa00640 1 -- 172.21.15.116:0/2747332158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 msgr2=0x5648612beff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd97aa00640 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612beff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd97aa00640 1 -- 172.21.15.116:0/2747332158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564860f2e3f0 con 0x5648612b5540 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd97aa00640 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bf530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009620 tx=0x7fd96c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9e1c93ce55b671 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd968c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd968c00640 1 -- 172.21.15.116:0/2747332158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c012110 con 0x5648612b5540 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd968c00640 1 -- 172.21.15.116:0/2747332158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd96c004050 con 0x5648612b5540 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.906+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd974003e50 con 0x5648612b5540 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd968c00640 1 -- 172.21.15.116:0/2747332158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd96c0122b0 con 0x5648612b5540 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 10 client.16893 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564860eee0f0 con 0x5648612b5540 2024-03-04T07:47:42.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd968c00640 1 -- 172.21.15.116:0/2747332158 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd974003e50 con 0x5648612b5540 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd968c00640 1 -- 172.21.15.116:0/2747332158 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd96c010430 con 0x5648612b5540 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 10 client.16893 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 10 client.16893 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 20 client.16893 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5648612cab30 0x5648612ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 4 client.16893 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564861227270 con 0x5648612cab30 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.907+0000 7fd97be00640 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5648612cab30 0x5648612ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.908+0000 7fd97be00640 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5648612cab30 0x5648612ccf80 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f7a6ed5a01da8ee server_cookie=9f8a0157bc96daa1 in_seq=0 out_seq=0 2024-03-04T07:47:42.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.908+0000 7fd968c00640 10 client.16893 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:42.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.908+0000 7fd968c00640 1 -- 172.21.15.116:0/2747332158 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564861227270 con 0x5648612cab30 2024-03-04T07:47:42.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.908+0000 7fd968c00640 10 client.16893 handle_command_reply: tid=0 2024-03-04T07:47:42.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 client.16893 shutdown 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 msgr2=0x5648612bf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bf530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009620 tx=0x7fd96c00f790 comp rx=0 tx=0).stop 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 shutdown_connections 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5648612cab30 0x5648612ccf80 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bf530 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 --2- 172.21.15.116:0/2747332158 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612beff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd968c00640 0 client.16893 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 >> 172.21.15.116:0/2747332158 conn(0x5648612b00c0 msgr2=0x5648612af360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 shutdown_connections 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 1 -- 172.21.15.116:0/2747332158 wait complete. 2024-03-04T07:47:42.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.909+0000 7fd984f89740 20 client.16893 trim_cache size 0 max 16384 2024-03-04T07:47:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.926+0000 7fd984f89740 1 Processor -- start 2024-03-04T07:47:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd984f89740 1 -- start start 2024-03-04T07:47:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd984f89740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612bda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd984f89740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd984f89740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd984f89740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1ad050 con 0x5648612b5540 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd984f89740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1af630 con 0x5648612b4b50 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd984f89740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1ad650 con 0x5648612ba6b0 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612bda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612bda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49976/0 (socket says 172.21.15.116:49976) 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 -- 172.21.15.116:0/1737604837 learned_addr learned my addr 172.21.15.116:0/1737604837 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97aa00640 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 -- 172.21.15.116:0/1737604837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 msgr2=0x5648612c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 -- 172.21.15.116:0/1737604837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 msgr2=0x5648612bdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.927+0000 7fd97be00640 1 -- 172.21.15.116:0/1737604837 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648612ca910 con 0x5648612b4b50 2024-03-04T07:47:42.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.928+0000 7fd97aa00640 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:47:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.928+0000 7fd97be00640 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612bda80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd974014040 tx=0x7fd974005a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d6b3abd8020155 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.928+0000 7fd96b400640 1 -- 172.21.15.116:0/1737604837 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974023040 con 0x5648612b4b50 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.928+0000 7fd96b400640 1 -- 172.21.15.116:0/1737604837 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd974011440 con 0x5648612b4b50 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.928+0000 7fd96b400640 1 -- 172.21.15.116:0/1737604837 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974005410 con 0x5648612b4b50 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 -- 172.21.15.116:0/1737604837 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 msgr2=0x5648612bda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612bda80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd974014040 tx=0x7fd974005a50 comp rx=0 tx=0).stop 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 -- 172.21.15.116:0/1737604837 shutdown_connections 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612ba6b0 0x5648612c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612b5540 0x5648612bdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 --2- 172.21.15.116:0/1737604837 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b4b50 0x5648612bda80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 -- 172.21.15.116:0/1737604837 >> 172.21.15.116:0/1737604837 conn(0x5648612b00c0 msgr2=0x5648612b6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 -- 172.21.15.116:0/1737604837 shutdown_connections 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 -- 172.21.15.116:0/1737604837 wait complete. 2024-03-04T07:47:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 -- start start 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.929+0000 7fd984f89740 1 Processor -- start 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd984f89740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd984f89740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612c4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd984f89740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd984f89740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1adcb0 con 0x5648612ba6b0 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd984f89740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1aefd0 con 0x5648612b5540 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd984f89740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1ae310 con 0x5648612b4b50 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd984f89740 10 client.0 _ll_register_callbacks cb 0x7fd96c012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612c4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49986/0 (socket says 172.21.15.116:49986) 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 -- 172.21.15.116:0/2476005637 learned_addr learned my addr 172.21.15.116:0/2476005637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97be00640 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 -- 172.21.15.116:0/2476005637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 msgr2=0x5648612c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 -- 172.21.15.116:0/2476005637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 msgr2=0x5648612c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.930+0000 7fd97aa00640 1 -- 172.21.15.116:0/2476005637 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648612c5c30 con 0x5648612b5540 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd97aa00640 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612c4630 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0097f0 tx=0x7fd96c012110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b92de5e003e8dde3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd968c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd968c00640 1 -- 172.21.15.116:0/2476005637 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c005110 con 0x5648612b5540 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd968c00640 1 -- 172.21.15.116:0/2476005637 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd96c0052b0 con 0x5648612b5540 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd974011440 con 0x5648612b5540 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd968c00640 1 -- 172.21.15.116:0/2476005637 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd96c0054b0 con 0x5648612b5540 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd984f89740 10 client.27020 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.931+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564861326840 con 0x5648612b5540 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd968c00640 1 -- 172.21.15.116:0/2476005637 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd974011440 con 0x5648612b5540 2024-03-04T07:47:42.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd968c00640 1 -- 172.21.15.116:0/2476005637 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd96c0305e0 con 0x5648612b5540 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd984f89740 10 client.27020 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd984f89740 10 client.27020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd984f89740 20 client.27020 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd984f89740 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564861327170 0x564861329590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd984f89740 4 client.27020 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5648612215c0 con 0x564861327170 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd97be00640 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564861327170 0x564861329590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd97be00640 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564861327170 0x564861329590 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bfd3760ea8965df server_cookie=b5d9904d65be26a0 in_seq=0 out_seq=0 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.932+0000 7fd968c00640 10 client.27020 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd968c00640 1 -- 172.21.15.116:0/2476005637 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5648612215c0 con 0x564861327170 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd968c00640 10 client.27020 handle_command_reply: tid=0 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 client.27020 shutdown 2024-03-04T07:47:42.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 msgr2=0x5648612c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612c4630 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0097f0 tx=0x7fd96c012110 comp rx=0 tx=0).stop 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 shutdown_connections 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564861327170 0x564861329590 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648612ba6b0 0x5648612c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5648612b5540 0x5648612c4630 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 --2- 172.21.15.116:0/2476005637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5648612b4b50 0x5648612c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd968c00640 0 client.27020 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 >> 172.21.15.116:0/2476005637 conn(0x5648612b00c0 msgr2=0x5648612befb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 shutdown_connections 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 1 -- 172.21.15.116:0/2476005637 wait complete. 2024-03-04T07:47:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.933+0000 7fd984f89740 20 client.27020 trim_cache size 0 max 16384 2024-03-04T07:47:42.942 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:47:42.942 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:47:42.942 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:47:42.942 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:47:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9600a1e00 msgr2=0x7fd9600a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9600a1e00 0x7fd9600a42c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd9640bc770 tx=0x7fd9640f37c0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c075f90 msgr2=0x7fd97c19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c075f90 0x7fd97c19dee0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd964036040 tx=0x7fd96402c650 comp rx=0 tx=0).stop 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 shutdown_connections 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd97c077b70 0x7fd97c1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd9600a1e00 0x7fd9600a42c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd97c0771d0 0x7fd97c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 --2- 172.21.15.116:0/3512001005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c075f90 0x7fd97c19dee0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 >> 172.21.15.116:0/3512001005 conn(0x7fd97c0fed80 msgr2=0x7fd97c100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 shutdown_connections 2024-03-04T07:47:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:42.934+0000 7fd982200640 1 -- 172.21.15.116:0/3512001005 wait complete. 2024-03-04T07:47:42.954 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:47:42.954 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:47:43.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:42 smithi116 ceph-mon[32150]: pgmap v609: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:43.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:42 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:47:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:42 smithi133 ceph-mon[36505]: pgmap v609: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:42 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:47:43.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.152+0000 7f54c1000640 1 -- 172.21.15.116:0/895750445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1027e0 msgr2=0x7f54bc102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.152+0000 7f54c1000640 1 --2- 172.21.15.116:0/895750445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1027e0 0x7f54bc102c60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009860 tx=0x7f54a802bb70 comp rx=0 tx=0).stop 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.152+0000 7f54c1000640 1 -- 172.21.15.116:0/895750445 shutdown_connections 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.153+0000 7f54c1000640 1 --2- 172.21.15.116:0/895750445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc1031a0 0x7f54bc109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.153+0000 7f54c1000640 1 --2- 172.21.15.116:0/895750445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1027e0 0x7f54bc102c60 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.153+0000 7f54c1000640 1 --2- 172.21.15.116:0/895750445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 0x7f54bc101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.153+0000 7f54c1000640 1 -- 172.21.15.116:0/895750445 >> 172.21.15.116:0/895750445 conn(0x7f54bc0fcb80 msgr2=0x7f54bc0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.153+0000 7f54c1000640 1 -- 172.21.15.116:0/895750445 shutdown_connections 2024-03-04T07:47:43.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.153+0000 7f54c1000640 1 -- 172.21.15.116:0/895750445 wait complete. 2024-03-04T07:47:43.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.155+0000 7f54c1000640 1 Processor -- start 2024-03-04T07:47:43.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.156+0000 7f54c1000640 1 -- start start 2024-03-04T07:47:43.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54c1000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 0x7f54bc199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54c1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc1027e0 0x7f54bc199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54c1000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1031a0 0x7f54bc1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54c1000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc10d250 con 0x7f54bc1027e0 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54c1000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc10d0d0 con 0x7f54bc1031a0 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54c1000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc10d3d0 con 0x7f54bc1015f0 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 0x7f54bc199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 0x7f54bc199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:50272/0 (socket says 172.21.15.116:50272) 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 -- 172.21.15.116:0/191896446 learned_addr learned my addr 172.21.15.116:0/191896446 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9400640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc1027e0 0x7f54bc199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54ba800640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1031a0 0x7f54bc1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 -- 172.21.15.116:0/191896446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1031a0 msgr2=0x7f54bc1a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1031a0 0x7f54bc1a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 -- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc1027e0 msgr2=0x7f54bc199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc1027e0 0x7f54bc199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.157+0000 7f54b9e00640 1 -- 172.21.15.116:0/191896446 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc1a16d0 con 0x7f54bc1015f0 2024-03-04T07:47:43.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.158+0000 7f54b9e00640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 0x7f54bc199ab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00e9a0 tx=0x7f54ac00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.158+0000 7f54b2a00640 1 -- 172.21.15.116:0/191896446 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac00cd80 con 0x7f54bc1015f0 2024-03-04T07:47:43.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.158+0000 7f54b2a00640 1 -- 172.21.15.116:0/191896446 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54ac010910 con 0x7f54bc1015f0 2024-03-04T07:47:43.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.158+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc1a19c0 con 0x7f54bc1015f0 2024-03-04T07:47:43.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.158+0000 7f54b2a00640 1 -- 172.21.15.116:0/191896446 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac018a30 con 0x7f54bc1015f0 2024-03-04T07:47:43.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.158+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc1a1f30 con 0x7f54bc1015f0 2024-03-04T07:47:43.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.159+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc1a21c0 con 0x7f54bc1015f0 2024-03-04T07:47:43.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.163+0000 7f54b2a00640 1 -- 172.21.15.116:0/191896446 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f54ac010ac0 con 0x7f54bc1015f0 2024-03-04T07:47:43.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.165+0000 7f54b2a00640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f54a00a1e00 0x7f54a00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.165+0000 7f54b2a00640 1 -- 172.21.15.116:0/191896446 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f54ac0ce230 con 0x7f54bc1015f0 2024-03-04T07:47:43.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.165+0000 7f54b9400640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f54a00a1e00 0x7f54a00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.166+0000 7f54b9400640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f54a00a1e00 0x7f54a00a42c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f54a802c180 tx=0x7f54a8034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.168+0000 7f54b2a00640 1 -- 172.21.15.116:0/191896446 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f54ac053280 con 0x7f54bc1015f0 2024-03-04T07:47:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:42 smithi152 ceph-mon[37154]: pgmap v609: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:42 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:47:43.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.417+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54bc1a2550 con 0x7f54bc1015f0 2024-03-04T07:47:43.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.418+0000 7f54b2a00640 1 -- 172.21.15.116:0/191896446 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f54ac09afd0 con 0x7f54bc1015f0 2024-03-04T07:47:43.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:47:43.419 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:47:43.419 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:47:43.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f54a00a1e00 msgr2=0x7f54a00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f54a00a1e00 0x7f54a00a42c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f54a802c180 tx=0x7f54a8034ab0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 msgr2=0x7f54bc199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 0x7f54bc199ab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00e9a0 tx=0x7f54ac00ee60 comp rx=0 tx=0).stop 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 shutdown_connections 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f54bc1031a0 0x7f54bc1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54bc1027e0 0x7f54bc199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f54a00a1e00 0x7f54a00a42c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 --2- 172.21.15.116:0/191896446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc1015f0 0x7f54bc199ab0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.420+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 >> 172.21.15.116:0/191896446 conn(0x7f54bc0fcb80 msgr2=0x7f54bc073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.421+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 shutdown_connections 2024-03-04T07:47:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.421+0000 7f54c1000640 1 -- 172.21.15.116:0/191896446 wait complete. 2024-03-04T07:47:43.438 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:47:43.438 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:47:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 -- 172.21.15.116:0/3316462542 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8107190 msgr2=0x7f2ff81095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 --2- 172.21.15.116:0/3316462542 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8107190 0x7f2ff81095e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009ca0 tx=0x7f2ff002c100 comp rx=0 tx=0).stop 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 -- 172.21.15.116:0/3316462542 shutdown_connections 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 --2- 172.21.15.116:0/3316462542 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8107190 0x7f2ff81095e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 --2- 172.21.15.116:0/3316462542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ff8104860 0x7f2ff8106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 --2- 172.21.15.116:0/3316462542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ff8101f30 0x7f2ff8104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 -- 172.21.15.116:0/3316462542 >> 172.21.15.116:0/3316462542 conn(0x7f2ff80fb940 msgr2=0x7f2ff80fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 -- 172.21.15.116:0/3316462542 shutdown_connections 2024-03-04T07:47:43.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.617+0000 7f2ffee00640 1 -- 172.21.15.116:0/3316462542 wait complete. 2024-03-04T07:47:43.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.618+0000 7f2ffee00640 1 Processor -- start 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.618+0000 7f2ffee00640 1 -- start start 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffee00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8101f30 0x7f2ff8197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffee00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ff8104860 0x7f2ff8197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffee00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ff8107190 0x7f2ff819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffee00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff810d140 con 0x7f2ff8107190 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffee00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff810cfc0 con 0x7f2ff8101f30 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffee00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff810d2c0 con 0x7f2ff8104860 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8101f30 0x7f2ff8197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8101f30 0x7f2ff8197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50032/0 (socket says 172.21.15.116:50032) 2024-03-04T07:47:43.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 -- 172.21.15.116:0/2338767294 learned_addr learned my addr 172.21.15.116:0/2338767294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffe400640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ff8107190 0x7f2ff819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffd000640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ff8104860 0x7f2ff8197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 -- 172.21.15.116:0/2338767294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ff8104860 msgr2=0x7f2ff8197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ff8104860 0x7f2ff8197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 -- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ff8107190 msgr2=0x7f2ff819ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ff8107190 0x7f2ff819ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffda00640 1 -- 172.21.15.116:0/2338767294 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff819f380 con 0x7f2ff8101f30 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.619+0000 7f2ffe400640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ff8107190 0x7f2ff819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.620+0000 7f2ffda00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8101f30 0x7f2ff8197760 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00e9f0 tx=0x7f2fec00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.620+0000 7f2ff6000640 1 -- 172.21.15.116:0/2338767294 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00cd80 con 0x7f2ff8101f30 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.620+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff819f670 con 0x7f2ff8101f30 2024-03-04T07:47:43.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.620+0000 7f2ff6000640 1 -- 172.21.15.116:0/2338767294 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fec005500 con 0x7f2ff8101f30 2024-03-04T07:47:43.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.620+0000 7f2ff6000640 1 -- 172.21.15.116:0/2338767294 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec010690 con 0x7f2ff8101f30 2024-03-04T07:47:43.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.620+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff819fbc0 con 0x7f2ff8101f30 2024-03-04T07:47:43.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 Processor -- start 2024-03-04T07:47:43.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 -- start start 2024-03-04T07:47:43.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f2ff6000640 1 -- 172.21.15.116:0/2338767294 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2fec010830 con 0x7f2ff8101f30 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e247ffb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffbc70 0x55e248000770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e248003270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e247f5bd60 con 0x55e248000de0 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e247f04cf0 con 0x55e247ffbc70 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f30031e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e247f57e00 con 0x55e247ffb280 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e247ffb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.623+0000 7f2ffda00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffbc70 0x55e248000770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e247ffb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:50294/0 (socket says 172.21.15.116:50294) 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 -- 172.21.15.116:0/1774238596 learned_addr learned my addr 172.21.15.116:0/1774238596 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffd000640 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e248003270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 -- 172.21.15.116:0/1774238596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffbc70 msgr2=0x55e248000770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffbc70 0x55e248000770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 -- 172.21.15.116:0/1774238596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 msgr2=0x55e248003270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e248003270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 -- 172.21.15.116:0/1774238596 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e247c703f0 con 0x55e247ffb280 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ffe400640 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e247ffb6a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00324c0 tx=0x7f2ff0005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c5e5dc17d438a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2ff6000640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2fe00a1e00 0x7f2fe00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.624+0000 7f2feaa00640 1 -- 172.21.15.116:0/1774238596 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0043020 con 0x55e247ffb280 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.625+0000 7f2feaa00640 1 -- 172.21.15.116:0/1774238596 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff0003970 con 0x55e247ffb280 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.625+0000 7f2feaa00640 1 -- 172.21.15.116:0/1774238596 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0039040 con 0x55e247ffb280 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.625+0000 7f2ff6000640 1 -- 172.21.15.116:0/2338767294 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2fec0ce210 con 0x7f2ff8101f30 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.625+0000 7f2ffda00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2fe00a1e00 0x7f2fe00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.625+0000 7f2ffda00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2fe00a1e00 0x7f2fe00a42c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2fec09ae90 tx=0x7f2fec0d1c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.625+0000 7f30031e6740 1 -- 172.21.15.116:0/1774238596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 msgr2=0x55e247ffb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.625+0000 7f30031e6740 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e247ffb6a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00324c0 tx=0x7f2ff0005f70 comp rx=0 tx=0).stop 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 -- 172.21.15.116:0/1774238596 shutdown_connections 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e248003270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffbc70 0x55e248000770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 --2- 172.21.15.116:0/1774238596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e247ffb6a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 -- 172.21.15.116:0/1774238596 >> 172.21.15.116:0/1774238596 conn(0x55e247ff67f0 msgr2=0x55e247ff6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 -- 172.21.15.116:0/1774238596 shutdown_connections 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 -- 172.21.15.116:0/1774238596 wait complete. 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.626+0000 7f30031e6740 1 -- start start 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 1 Processor -- start 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e248005cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e24800cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e248003d30 con 0x55e247ffbc70 2024-03-04T07:47:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e247efa220 con 0x55e248000de0 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e248003eb0 con 0x55e247ffb280 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f30031e6740 10 client.0 _ll_register_callbacks cb 0x55e247c6fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f2ffd000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e248005cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e24800cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:50310/0 (socket says 172.21.15.116:50310) 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.627+0000 7f2ffe400640 1 -- 172.21.15.116:0/3157300796 learned_addr learned my addr 172.21.15.116:0/3157300796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2ffe400640 1 -- 172.21.15.116:0/3157300796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 msgr2=0x55e24800cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2ffe400640 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e24800cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2ffe400640 1 -- 172.21.15.116:0/3157300796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 msgr2=0x55e248005cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2ffe400640 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e248005cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2ffe400640 1 -- 172.21.15.116:0/3157300796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e247d3a7d0 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2ffe400640 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009bf0 tx=0x7f2ff002c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ab3fa9a314491bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2fdf400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2fdf400640 1 -- 172.21.15.116:0/3157300796 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0043040 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ff0003970 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2fdf400640 1 -- 172.21.15.116:0/3157300796 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2ff0004230 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.628+0000 7f2fdf400640 1 -- 172.21.15.116:0/3157300796 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ff0039040 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 10 client.26938 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e248003d80 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f2fdf400640 1 -- 172.21.15.116:0/3157300796 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff0003970 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f2fdf400640 1 -- 172.21.15.116:0/3157300796 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2ff003f600 con 0x55e247ffb280 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 10 client.26938 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 10 client.26938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 20 client.26938 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2480112b0 0x55e2480136a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 4 client.26938 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e247f692a0 con 0x55e2480112b0 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.629+0000 7f2ffd000640 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2480112b0 0x55e2480136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.630+0000 7f2ffd000640 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2480112b0 0x55e2480136a0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38337ba097259ad7 server_cookie=560cb5ede0c6bfa9 in_seq=0 out_seq=0 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.630+0000 7f2fdf400640 10 client.26938 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.630+0000 7f2fdf400640 1 -- 172.21.15.116:0/3157300796 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e247f692a0 con 0x55e2480112b0 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.630+0000 7f2fdf400640 10 client.26938 handle_command_reply: tid=0 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 client.26938 shutdown 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 msgr2=0x55e248005780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009bf0 tx=0x7f2ff002c100 comp rx=0 tx=0).stop 2024-03-04T07:47:43.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 shutdown_connections 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e2480112b0 0x55e2480136a0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e24800cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e248005cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f2fdf400640 0 client.26938 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 --2- 172.21.15.116:0/3157300796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 >> 172.21.15.116:0/3157300796 conn(0x55e247ff67f0 msgr2=0x55e247ff1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 shutdown_connections 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 1 -- 172.21.15.116:0/3157300796 wait complete. 2024-03-04T07:47:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.631+0000 7f30031e6740 20 client.26938 trim_cache size 0 max 16384 2024-03-04T07:47:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 Processor -- start 2024-03-04T07:47:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 -- start start 2024-03-04T07:47:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e24800ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e247ff1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff81a4660 con 0x55e247ffbc70 2024-03-04T07:47:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff81a6c40 con 0x55e248000de0 2024-03-04T07:47:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.648+0000 7f30031e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff81a4c60 con 0x55e247ffb280 2024-03-04T07:47:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e247ff1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e247ff1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50070/0 (socket says 172.21.15.116:50070) 2024-03-04T07:47:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 -- 172.21.15.116:0/4067624037 learned_addr learned my addr 172.21.15.116:0/4067624037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffd000640 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e24800ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 -- 172.21.15.116:0/4067624037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 msgr2=0x55e248005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 -- 172.21.15.116:0/4067624037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 msgr2=0x55e24800ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e24800ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffe400640 1 -- 172.21.15.116:0/4067624037 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e248013a10 con 0x55e248000de0 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.649+0000 7f2ffd000640 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e24800ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.650+0000 7f2ffe400640 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e247ff1580 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009bc0 tx=0x7f2ff002c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b7828816da1f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.650+0000 7f2fea000640 1 -- 172.21.15.116:0/4067624037 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0043040 con 0x55e248000de0 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.650+0000 7f2fea000640 1 -- 172.21.15.116:0/4067624037 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff0005c30 con 0x55e248000de0 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.650+0000 7f2fea000640 1 -- 172.21.15.116:0/4067624037 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00042a0 con 0x55e248000de0 2024-03-04T07:47:43.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.650+0000 7f30031e6740 1 -- 172.21.15.116:0/4067624037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 msgr2=0x55e247ff1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.650+0000 7f30031e6740 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e247ff1580 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009bc0 tx=0x7f2ff002c100 comp rx=0 tx=0).stop 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 -- 172.21.15.116:0/4067624037 shutdown_connections 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e248000de0 0x55e247ff1580 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e247ffbc70 0x55e24800ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 --2- 172.21.15.116:0/4067624037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffb280 0x55e248005c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 -- 172.21.15.116:0/4067624037 >> 172.21.15.116:0/4067624037 conn(0x55e247ff67f0 msgr2=0x55e247ff0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 -- 172.21.15.116:0/4067624037 shutdown_connections 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 -- 172.21.15.116:0/4067624037 wait complete. 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 -- start start 2024-03-04T07:47:43.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.651+0000 7f30031e6740 1 Processor -- start 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f30031e6740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffb280 0x55e24800c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f30031e6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffbc70 0x55e24800cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f30031e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e24800ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f30031e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff81a52c0 con 0x55e248000de0 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f30031e6740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff81a65e0 con 0x55e247ffb280 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f30031e6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff81a5920 con 0x55e247ffbc70 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f30031e6740 10 client.0 _ll_register_callbacks cb 0x7f2fec0d1b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:43.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffb280 0x55e24800c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffe400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffb280 0x55e24800c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50072/0 (socket says 172.21.15.116:50072) 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffe400640 1 -- 172.21.15.116:0/2149210616 learned_addr learned my addr 172.21.15.116:0/2149210616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffe400640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e24800ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffd000640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffbc70 0x55e24800cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffd000640 1 -- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffb280 msgr2=0x55e24800c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffd000640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffb280 0x55e24800c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffd000640 1 -- 172.21.15.116:0/2149210616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 msgr2=0x55e24800ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffd000640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e24800ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffd000640 1 -- 172.21.15.116:0/2149210616 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e24800b550 con 0x55e247ffbc70 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.652+0000 7f2ffe400640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffb280 0x55e24800c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f2ffd000640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffbc70 0x55e24800cbe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55e248013ed0 tx=0x7f2fe400be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9762aff142f4d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f2fdf400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:47:43.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f2fdf400640 1 -- 172.21.15.116:0/2149210616 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe40035f0 con 0x55e247ffbc70 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f2fdf400640 1 -- 172.21.15.116:0/2149210616 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2fe4003790 con 0x55e247ffbc70 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ff0005c30 con 0x55e247ffbc70 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f2fdf400640 1 -- 172.21.15.116:0/2149210616 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fe402d050 con 0x55e247ffbc70 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f30031e6740 10 client.26944 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e24806d700 con 0x55e247ffbc70 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f2fdf400640 1 -- 172.21.15.116:0/2149210616 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff0005c30 con 0x55e247ffbc70 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f2fdf400640 1 -- 172.21.15.116:0/2149210616 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2fe4026a20 con 0x55e247ffbc70 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f30031e6740 10 client.26944 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f30031e6740 10 client.26944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f30031e6740 20 client.26944 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.653+0000 7f30031e6740 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24806e010 0x55e248070430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f30031e6740 4 client.26944 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e247f635f0 con 0x55e24806e010 2024-03-04T07:47:43.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f2ffe400640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24806e010 0x55e248070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f2ffe400640 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24806e010 0x55e248070430 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d0b014dc6da0c76 server_cookie=a987b43172736e3 in_seq=0 out_seq=0 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f2fdf400640 10 client.26944 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f2fdf400640 1 -- 172.21.15.116:0/2149210616 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e247f635f0 con 0x55e24806e010 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f2fdf400640 10 client.26944 handle_command_reply: tid=0 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f30031e6740 1 client.26944 shutdown 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffbc70 msgr2=0x55e24800cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.654+0000 7f30031e6740 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffbc70 0x55e24800cbe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55e248013ed0 tx=0x7f2fe400be40 comp rx=0 tx=0).stop 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 shutdown_connections 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e24806e010 0x55e248070430 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e248000de0 0x55e24800ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e247ffbc70 0x55e24800cbe0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 --2- 172.21.15.116:0/2149210616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e247ffb280 0x55e24800c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 >> 172.21.15.116:0/2149210616 conn(0x55e247ff67f0 msgr2=0x55e248005aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2fdf400640 0 client.26944 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 shutdown_connections 2024-03-04T07:47:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 1 -- 172.21.15.116:0/2149210616 wait complete. 2024-03-04T07:47:43.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f30031e6740 20 client.26944 trim_cache size 0 max 16384 2024-03-04T07:47:43.662 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:47:43.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2fe00a1e00 msgr2=0x7f2fe00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2fe00a1e00 0x7f2fe00a42c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2fec09ae90 tx=0x7f2fec0d1c50 comp rx=0 tx=0).stop 2024-03-04T07:47:43.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8101f30 msgr2=0x7f2ff8197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:43.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8101f30 0x7f2ff8197760 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00e9f0 tx=0x7f2fec00edf0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 shutdown_connections 2024-03-04T07:47:43.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ff8107190 0x7f2ff819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2fe00a1e00 0x7f2fe00a42c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ff8104860 0x7f2ff8197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 --2- 172.21.15.116:0/2338767294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ff8101f30 0x7f2ff8197760 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:43.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.655+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 >> 172.21.15.116:0/2338767294 conn(0x7f2ff80fb940 msgr2=0x7f2ff8102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:43.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.656+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 shutdown_connections 2024-03-04T07:47:43.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:43.656+0000 7f2ffee00640 1 -- 172.21.15.116:0/2338767294 wait complete. 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/191896446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:47:44.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:43 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:47:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/191896446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:47:44.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:47:44.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:47:44.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:43 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/191896446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:47:44.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:43 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:47:44.673 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:47:44.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:47:44.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:47:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 -- 172.21.15.116:0/293737082 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c075610 msgr2=0x7f0b5c073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 --2- 172.21.15.116:0/293737082 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c075610 0x7f0b5c073a60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009900 tx=0x7f0b5002bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 -- 172.21.15.116:0/293737082 shutdown_connections 2024-03-04T07:47:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 --2- 172.21.15.116:0/293737082 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b5c1094a0 0x7f0b5c10b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 --2- 172.21.15.116:0/293737082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c074030 0x7f0b5c0744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 --2- 172.21.15.116:0/293737082 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c075610 0x7f0b5c073a60 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 -- 172.21.15.116:0/293737082 >> 172.21.15.116:0/293737082 conn(0x7f0b5c0fcb50 msgr2=0x7f0b5c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:44.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 -- 172.21.15.116:0/293737082 shutdown_connections 2024-03-04T07:47:44.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.902+0000 7f0b64000640 1 -- 172.21.15.116:0/293737082 wait complete. 2024-03-04T07:47:44.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.903+0000 7f0b64000640 1 Processor -- start 2024-03-04T07:47:44.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.903+0000 7f0b64000640 1 -- start start 2024-03-04T07:47:44.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.903+0000 7f0b64000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b5c074030 0x7f0b5c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b64000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c075610 0x7f0b5c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b64000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c1094a0 0x7f0b5c1a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b64000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c10f490 con 0x7f0b5c075610 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b64000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c10f310 con 0x7f0b5c1094a0 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b64000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c10f610 con 0x7f0b5c074030 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c075610 0x7f0b5c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c075610 0x7f0b5c19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37408/0 (socket says 172.21.15.116:37408) 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 -- 172.21.15.116:0/2043560392 learned_addr learned my addr 172.21.15.116:0/2043560392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b62000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c1094a0 0x7f0b5c1a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b61600640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b5c074030 0x7f0b5c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 -- 172.21.15.116:0/2043560392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b5c074030 msgr2=0x7f0b5c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b5c074030 0x7f0b5c199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 -- 172.21.15.116:0/2043560392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c1094a0 msgr2=0x7f0b5c1a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c1094a0 0x7f0b5c1a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b60c00640 1 -- 172.21.15.116:0/2043560392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c1a1740 con 0x7f0b5c075610 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.904+0000 7f0b62000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c1094a0 0x7f0b5c1a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.905+0000 7f0b60c00640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c075610 0x7f0b5c19a060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800ea60 tx=0x7f0b4800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.905+0000 7f0b5a000640 1 -- 172.21.15.116:0/2043560392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4800cde0 con 0x7f0b5c075610 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.905+0000 7f0b5a000640 1 -- 172.21.15.116:0/2043560392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b480035f0 con 0x7f0b5c075610 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.905+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c1a1a30 con 0x7f0b5c075610 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.905+0000 7f0b5a000640 1 -- 172.21.15.116:0/2043560392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48010670 con 0x7f0b5c075610 2024-03-04T07:47:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.905+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c1a1f80 con 0x7f0b5c075610 2024-03-04T07:47:44.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.908+0000 7f0b5a000640 1 -- 172.21.15.116:0/2043560392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f0b480037a0 con 0x7f0b5c075610 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.908+0000 7f0b66dde740 1 Processor -- start 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b66dde740 1 -- start start 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6645000d0 0x55b6645004f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b6645055c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b664505c30 0x55b6645080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b66dde740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b664413e20 con 0x55b664505c30 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b66dde740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b664465a20 con 0x55b6645000d0 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b66dde740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b66440eb00 con 0x55b664500ac0 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b62000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6645000d0 0x55b6645004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b62000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6645000d0 0x55b6645004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50106/0 (socket says 172.21.15.116:50106) 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b62000640 1 -- 172.21.15.116:0/1955204851 learned_addr learned my addr 172.21.15.116:0/1955204851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:44.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b61600640 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b6645055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.909+0000 7f0b60c00640 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b664505c30 0x55b6645080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b62000640 1 -- 172.21.15.116:0/1955204851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 msgr2=0x55b6645055c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b62000640 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b6645055c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b62000640 1 -- 172.21.15.116:0/1955204851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b664505c30 msgr2=0x55b6645080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b62000640 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b664505c30 0x55b6645080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b62000640 1 -- 172.21.15.116:0/1955204851 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b664423e10 con 0x55b6645000d0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b62000640 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6645000d0 0x55b6645004f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0b540117e0 tx=0x7f0b54011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee4642e2d48d452a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b4e600640 1 -- 172.21.15.116:0/1955204851 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b54011310 con 0x55b6645000d0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b4e600640 1 -- 172.21.15.116:0/1955204851 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b54003e50 con 0x55b6645000d0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b4e600640 1 -- 172.21.15.116:0/1955204851 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b54004620 con 0x55b6645000d0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b5a000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b440a1d80 0x7f0b440a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.910+0000 7f0b5a000640 1 -- 172.21.15.116:0/2043560392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0b480ce060 con 0x7f0b5c075610 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b61600640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b440a1d80 0x7f0b440a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 -- 172.21.15.116:0/1955204851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6645000d0 msgr2=0x55b6645004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6645000d0 0x55b6645004f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0b540117e0 tx=0x7f0b54011ca0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 -- 172.21.15.116:0/1955204851 shutdown_connections 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b664505c30 0x55b6645080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b6645055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1955204851 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b6645000d0 0x55b6645004f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 -- 172.21.15.116:0/1955204851 >> 172.21.15.116:0/1955204851 conn(0x55b6644fb640 msgr2=0x55b6644fba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 -- 172.21.15.116:0/1955204851 shutdown_connections 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b66dde740 1 -- 172.21.15.116:0/1955204851 wait complete. 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.911+0000 7f0b61600640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b440a1d80 0x7f0b440a4240 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0b5002c240 tx=0x7f0b5002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 -- start start 2024-03-04T07:47:44.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 Processor -- start 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b66450a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b664511a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b664421f40 con 0x55b6645000d0 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b664461ac0 con 0x55b664500ac0 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.912+0000 7f0b66dde740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b664508be0 con 0x55b664505c30 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b66dde740 10 client.0 _ll_register_callbacks cb 0x55b664460aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b66450a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b61600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b66450a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50114/0 (socket says 172.21.15.116:50114) 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 -- 172.21.15.116:0/1854384349 learned_addr learned my addr 172.21.15.116:0/1854384349 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:44.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b62000640 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b664511a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 -- 172.21.15.116:0/1854384349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 msgr2=0x55b664511a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b664511a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 -- 172.21.15.116:0/1854384349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 msgr2=0x55b66450a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 -- 172.21.15.116:0/1854384349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6642427e0 con 0x55b664500ac0 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b62000640 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b664511a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b60c00640 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b66450a9d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0b480a3000 tx=0x7f0b480095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6211b541e8e09645 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.913+0000 7f0b42a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b42a00640 1 -- 172.21.15.116:0/1854384349 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4809a6f0 con 0x55b664500ac0 2024-03-04T07:47:44.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b42a00640 1 -- 172.21.15.116:0/1854384349 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0b48097400 con 0x55b664500ac0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b54003e50 con 0x55b664500ac0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b42a00640 1 -- 172.21.15.116:0/1854384349 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b4809b490 con 0x55b664500ac0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 10 client.27056 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b6642a2f70 con 0x55b664500ac0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b42a00640 1 -- 172.21.15.116:0/1854384349 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b54003e50 con 0x55b664500ac0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b42a00640 1 -- 172.21.15.116:0/1854384349 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0b480d00f0 con 0x55b664500ac0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 10 client.27056 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 10 client.27056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 20 client.27056 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6645162c0 0x55b6645186f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 4 client.27056 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.914+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b664472f60 con 0x55b6645162c0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.915+0000 7f0b61600640 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6645162c0 0x55b6645186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.915+0000 7f0b61600640 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6645162c0 0x55b6645186f0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=794af456eab4eacf server_cookie=2353e6f825725339 in_seq=0 out_seq=0 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.915+0000 7f0b42a00640 10 client.27056 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b42a00640 1 -- 172.21.15.116:0/1854384349 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b664472f60 con 0x55b6645162c0 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b42a00640 10 client.27056 handle_command_reply: tid=0 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 client.27056 shutdown 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 msgr2=0x55b66450a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b66450a9d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0b480a3000 tx=0x7f0b480095a0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 shutdown_connections 2024-03-04T07:47:44.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b6645162c0 0x55b6645186f0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b664511a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b66450a9d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 --2- 172.21.15.116:0/1854384349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b42a00640 0 client.27056 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 >> 172.21.15.116:0/1854384349 conn(0x55b6644fb640 msgr2=0x55b6644fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 shutdown_connections 2024-03-04T07:47:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 1 -- 172.21.15.116:0/1854384349 wait complete. 2024-03-04T07:47:44.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.916+0000 7f0b66dde740 20 client.27056 trim_cache size 0 max 16384 2024-03-04T07:47:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 Processor -- start 2024-03-04T07:47:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 -- start start 2024-03-04T07:47:44.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b6644faa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b6645119d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b6644fefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a8cd0 con 0x55b6645000d0 2024-03-04T07:47:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1ab2b0 con 0x55b664500ac0 2024-03-04T07:47:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b66dde740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a92d0 con 0x55b664505c30 2024-03-04T07:47:44.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b61600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b6644faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b61600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b6644faa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37428/0 (socket says 172.21.15.116:37428) 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b61600640 1 -- 172.21.15.116:0/4221824028 learned_addr learned my addr 172.21.15.116:0/4221824028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b60c00640 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b6645119d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b62000640 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b6644fefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.934+0000 7f0b61600640 1 -- 172.21.15.116:0/4221824028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 msgr2=0x55b6644fefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b61600640 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b6644fefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b61600640 1 -- 172.21.15.116:0/4221824028 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 msgr2=0x55b6645119d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b61600640 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b6645119d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b61600640 1 -- 172.21.15.116:0/4221824028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b66413a170 con 0x55b6645000d0 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b60c00640 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b6645119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b61600640 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b6644faa60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0b50035000 tx=0x7f0b50042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725c7c0762356e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b4dc00640 1 -- 172.21.15.116:0/4221824028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50046070 con 0x55b6645000d0 2024-03-04T07:47:44.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b4dc00640 1 -- 172.21.15.116:0/4221824028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b50004420 con 0x55b6645000d0 2024-03-04T07:47:44.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.935+0000 7f0b4dc00640 1 -- 172.21.15.116:0/4221824028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50058a50 con 0x55b6645000d0 2024-03-04T07:47:44.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 -- 172.21.15.116:0/4221824028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 msgr2=0x55b6644faa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b6644faa60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0b50035000 tx=0x7f0b50042000 comp rx=0 tx=0).stop 2024-03-04T07:47:44.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 -- 172.21.15.116:0/4221824028 shutdown_connections 2024-03-04T07:47:44.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664505c30 0x55b6644fefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664500ac0 0x55b6645119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4221824028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b6644faa60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 -- 172.21.15.116:0/4221824028 >> 172.21.15.116:0/4221824028 conn(0x55b6644fb640 msgr2=0x55b6644fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 -- 172.21.15.116:0/4221824028 shutdown_connections 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 -- 172.21.15.116:0/4221824028 wait complete. 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.936+0000 7f0b66dde740 1 -- start start 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 1 Processor -- start 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b66450fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664505c30 0x55b664510ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a9930 con 0x55b6645000d0 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1aac50 con 0x55b664505c30 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a9f90 con 0x55b664500ac0 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b66dde740 10 client.0 _ll_register_callbacks cb 0x7f0b48003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b61600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b60c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b66450fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b61600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37436/0 (socket says 172.21.15.116:37436) 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b61600640 1 -- 172.21.15.116:0/4043347517 learned_addr learned my addr 172.21.15.116:0/4043347517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.937+0000 7f0b62000640 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664505c30 0x55b664510ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b61600640 1 -- 172.21.15.116:0/4043347517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 msgr2=0x55b66450fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b61600640 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b66450fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b61600640 1 -- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664505c30 msgr2=0x55b664510ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b61600640 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664505c30 0x55b664510ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b61600640 1 -- 172.21.15.116:0/4043347517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b664511290 con 0x55b6645000d0 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b61600640 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450f880 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0b5002cde0 tx=0x7f0b500581b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0dfca7a25668aa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b42a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b42a00640 1 -- 172.21.15.116:0/4043347517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50004b90 con 0x55b6645000d0 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b50004420 con 0x55b6645000d0 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.938+0000 7f0b42a00640 1 -- 172.21.15.116:0/4043347517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0b500587d0 con 0x55b6645000d0 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b42a00640 1 -- 172.21.15.116:0/4043347517 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b500589d0 con 0x55b6645000d0 2024-03-04T07:47:44.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 10 client.16935 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b664572240 con 0x55b6645000d0 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b42a00640 1 -- 172.21.15.116:0/4043347517 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b50004420 con 0x55b6645000d0 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b42a00640 1 -- 172.21.15.116:0/4043347517 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0b500035f0 con 0x55b6645000d0 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 10 client.16935 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 10 client.16935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 20 client.16935 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b664572bc0 0x55b664574fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 4 client.16935 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b66446d2b0 con 0x55b664572bc0 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.939+0000 7f0b60c00640 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b664572bc0 0x55b664574fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.940+0000 7f0b60c00640 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b664572bc0 0x55b664574fe0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d51424636674da51 server_cookie=e7feb3ed65b64b42 in_seq=0 out_seq=0 2024-03-04T07:47:44.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.940+0000 7f0b42a00640 10 client.16935 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b42a00640 1 -- 172.21.15.116:0/4043347517 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b66446d2b0 con 0x55b664572bc0 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b42a00640 10 client.16935 handle_command_reply: tid=0 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 client.16935 shutdown 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 msgr2=0x55b66450f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450f880 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0b5002cde0 tx=0x7f0b500581b0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 shutdown_connections 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b664572bc0 0x55b664574fe0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b664505c30 0x55b664510ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b664500ac0 0x55b66450fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 --2- 172.21.15.116:0/4043347517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6645000d0 0x55b66450f880 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b42a00640 0 client.16935 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 >> 172.21.15.116:0/4043347517 conn(0x55b6644fb640 msgr2=0x55b6644ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:44.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 shutdown_connections 2024-03-04T07:47:44.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 1 -- 172.21.15.116:0/4043347517 wait complete. 2024-03-04T07:47:44.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.941+0000 7f0b66dde740 20 client.16935 trim_cache size 0 max 16384 2024-03-04T07:47:44.949 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:47:44.949 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:47:44.949 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "036313f6-e14a-4d45-af4d-3d1afcb8d486", 2024-03-04T07:47:44.949 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:47:44.949 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:47:44.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b440a1d80 msgr2=0x7f0b440a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b440a1d80 0x7f0b440a4240 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0b5002c240 tx=0x7f0b5002df90 comp rx=0 tx=0).stop 2024-03-04T07:47:44.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c075610 msgr2=0x7f0b5c19a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:44.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c075610 0x7f0b5c19a060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800ea60 tx=0x7f0b4800ef20 comp rx=0 tx=0).stop 2024-03-04T07:47:44.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 shutdown_connections 2024-03-04T07:47:44.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b5c1094a0 0x7f0b5c1a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b5c075610 0x7f0b5c19a060 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f0b440a1d80 0x7f0b440a4240 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 --2- 172.21.15.116:0/2043560392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b5c074030 0x7f0b5c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 >> 172.21.15.116:0/2043560392 conn(0x7f0b5c0fcb50 msgr2=0x7f0b5c1053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 shutdown_connections 2024-03-04T07:47:44.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:44.942+0000 7f0b64000640 1 -- 172.21.15.116:0/2043560392 wait complete. 2024-03-04T07:47:44.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:47:44.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:47:45.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:44 smithi116 ceph-mon[32150]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-04T07:47:45.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:44 smithi116 ceph-mon[32150]: Unable to set osd_memory_target on smithi152 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-04T07:47:45.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:44 smithi116 ceph-mon[32150]: pgmap v610: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:45.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3799454364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec1097a0 msgr2=0x7fe3ec110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3799454364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec1097a0 0x7fe3ec110030 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009ca0 tx=0x7fe3e402c100 comp rx=0 tx=0).stop 2024-03-04T07:47:45.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3799454364 shutdown_connections 2024-03-04T07:47:45.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3799454364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec1097a0 0x7fe3ec110030 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3799454364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3ec108de0 0x7fe3ec109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3799454364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec107bf0 0x7fe3ec108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3799454364 >> 172.21.15.116:0/3799454364 conn(0x7fe3ec076b90 msgr2=0x7fe3ec078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:45.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3799454364 shutdown_connections 2024-03-04T07:47:45.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.168+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3799454364 wait complete. 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.169+0000 7fe3f2a00640 1 Processor -- start 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.169+0000 7fe3f2a00640 1 -- start start 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.169+0000 7fe3f2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec107bf0 0x7fe3ec19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3f2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec108de0 0x7fe3ec19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3f2a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3ec1097a0 0x7fe3ec1a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3f2a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec113870 con 0x7fe3ec108de0 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3f2a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1136f0 con 0x7fe3ec107bf0 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3f2a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1139f0 con 0x7fe3ec1097a0 2024-03-04T07:47:45.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec107bf0 0x7fe3ec19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3e3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec108de0 0x7fe3ec19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec107bf0 0x7fe3ec19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50140/0 (socket says 172.21.15.116:50140) 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 -- 172.21.15.116:0/3406570060 learned_addr learned my addr 172.21.15.116:0/3406570060 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3ec1097a0 0x7fe3ec1a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 -- 172.21.15.116:0/3406570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3ec1097a0 msgr2=0x7fe3ec1a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3ec1097a0 0x7fe3ec1a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 -- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec108de0 msgr2=0x7fe3ec19e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec108de0 0x7fe3ec19e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.170+0000 7fe3ebe00640 1 -- 172.21.15.116:0/3406570060 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec1a5aa0 con 0x7fe3ec107bf0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.171+0000 7fe3ebe00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec107bf0 0x7fe3ec19de80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00e9c0 tx=0x7fe3dc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.171+0000 7fe3e9600640 1 -- 172.21.15.116:0/3406570060 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc00cd80 con 0x7fe3ec107bf0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.171+0000 7fe3e9600640 1 -- 172.21.15.116:0/3406570060 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3dc0035f0 con 0x7fe3ec107bf0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.171+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec1a5d90 con 0x7fe3ec107bf0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.171+0000 7fe3e9600640 1 -- 172.21.15.116:0/3406570060 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc010690 con 0x7fe3ec107bf0 2024-03-04T07:47:45.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.171+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec1a6220 con 0x7fe3ec107bf0 2024-03-04T07:47:45.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.174+0000 7fe3f57bf740 1 Processor -- start 2024-03-04T07:47:45.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.174+0000 7fe3e9600640 1 -- 172.21.15.116:0/3406570060 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe3dc0037a0 con 0x7fe3ec107bf0 2024-03-04T07:47:45.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.174+0000 7fe3f57bf740 1 -- start start 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb251f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb257040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb259b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eb17f090 con 0x5624eb2576b0 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eb1b6d30 con 0x5624eb251b50 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eb15fa30 con 0x5624eb252540 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb251f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3e3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb259b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb251f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50144/0 (socket says 172.21.15.116:50144) 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb257040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 -- 172.21.15.116:0/3234849947 learned_addr learned my addr 172.21.15.116:0/3234849947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 -- 172.21.15.116:0/3234849947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 msgr2=0x5624eb257040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/3234849947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb257040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 -- 172.21.15.116:0/3234849947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 msgr2=0x5624eb259b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/3234849947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb259b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.175+0000 7fe3f0a00640 1 -- 172.21.15.116:0/3234849947 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624eaf957d0 con 0x5624eb251b50 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.176+0000 7fe3e9600640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3d00a1d80 0x7fe3d00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.176+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/3234849947 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb251f70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009ca0 tx=0x7fe3e40323c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b550e552cebe8a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.176+0000 7fe3e9600640 1 -- 172.21.15.116:0/3406570060 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe3dc0ce170 con 0x7fe3ec107bf0 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.176+0000 7fe3ebe00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3d00a1d80 0x7fe3d00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.176+0000 7fe3e1200640 1 -- 172.21.15.116:0/3234849947 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4043070 con 0x5624eb251b50 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.176+0000 7fe3e1200640 1 -- 172.21.15.116:0/3234849947 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3e4003970 con 0x5624eb251b50 2024-03-04T07:47:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.176+0000 7fe3e1200640 1 -- 172.21.15.116:0/3234849947 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4039040 con 0x5624eb251b50 2024-03-04T07:47:45.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 -- 172.21.15.116:0/3234849947 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 msgr2=0x5624eb251f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/3234849947 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb251f70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009ca0 tx=0x7fe3e40323c0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3ebe00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3d00a1d80 0x7fe3d00a4240 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc097830 tx=0x7fe3dc0d1bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:45.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 -- 172.21.15.116:0/3234849947 shutdown_connections 2024-03-04T07:47:45.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/3234849947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb259b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/3234849947 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb257040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/3234849947 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb251f70 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 -- 172.21.15.116:0/3234849947 >> 172.21.15.116:0/3234849947 conn(0x5624eb24d0c0 msgr2=0x5624eb24d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 -- 172.21.15.116:0/3234849947 shutdown_connections 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 -- 172.21.15.116:0/3234849947 wait complete. 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.177+0000 7fe3f57bf740 1 -- start start 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 1 Processor -- start 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb251b50 0x5624eb25bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb25c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb263440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eb1550c0 con 0x5624eb251b50 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eb1b2dd0 con 0x5624eb2576b0 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eb25a600 con 0x5624eb252540 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f57bf740 10 client.0 _ll_register_callbacks cb 0x5624eb172860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb251b50 0x5624eb25bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3e3400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb25c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3e3400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb25c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:50414/0 (socket says 172.21.15.116:50414) 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3e3400640 1 -- 172.21.15.116:0/260793275 learned_addr learned my addr 172.21.15.116:0/260793275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.178+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb263440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3e3400640 1 -- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 msgr2=0x5624eb263440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3e3400640 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb263440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3e3400640 1 -- 172.21.15.116:0/260793275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb251b50 msgr2=0x5624eb25bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3e3400640 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb251b50 0x5624eb25bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3e3400640 1 -- 172.21.15.116:0/260793275 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624eaecb3f0 con 0x5624eb252540 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb263440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3e3400640 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb25c400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4012720 tx=0x7fe3d4012be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d94564d550cf7ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3d9600640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3d9600640 1 -- 172.21.15.116:0/260793275 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d400f8c0 con 0x5624eb252540 2024-03-04T07:47:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3d9600640 1 -- 172.21.15.116:0/260793275 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe3d400fa60 con 0x5624eb252540 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.179+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3e4003970 con 0x5624eb252540 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3d9600640 1 -- 172.21.15.116:0/260793275 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3d400fc60 con 0x5624eb252540 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 10 client.26971 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5624eae8b0f0 con 0x5624eb252540 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3d9600640 1 -- 172.21.15.116:0/260793275 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3e4003970 con 0x5624eb252540 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3d9600640 1 -- 172.21.15.116:0/260793275 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3d401c4b0 con 0x5624eb252540 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 10 client.26971 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 10 client.26971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 20 client.26971 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb267990 0x5624eb269dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 4 client.26971 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.180+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624eb1c4270 con 0x5624eb267990 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.181+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb267990 0x5624eb269dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.181+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb267990 0x5624eb269dd0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=163ca9d6787233f7 server_cookie=79cebb5bffa401c8 in_seq=0 out_seq=0 2024-03-04T07:47:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.181+0000 7fe3d9600640 10 client.26971 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3d9600640 1 -- 172.21.15.116:0/260793275 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5624eb1c4270 con 0x5624eb267990 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3d9600640 10 client.26971 handle_command_reply: tid=0 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 client.26971 shutdown 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 msgr2=0x5624eb25c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb25c400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4012720 tx=0x7fe3d4012be0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 shutdown_connections 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb267990 0x5624eb269dd0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb263440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb25c400 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/260793275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb251b50 0x5624eb25bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 >> 172.21.15.116:0/260793275 conn(0x5624eb24d0c0 msgr2=0x5624eb24c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3d9600640 0 client.26971 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:45.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 shutdown_connections 2024-03-04T07:47:45.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 1 -- 172.21.15.116:0/260793275 wait complete. 2024-03-04T07:47:45.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.182+0000 7fe3f57bf740 20 client.26971 trim_cache size 0 max 16384 2024-03-04T07:47:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.200+0000 7fe3f57bf740 1 Processor -- start 2024-03-04T07:47:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f57bf740 1 -- start start 2024-03-04T07:47:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb24bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb263400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb24c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1ad070 con 0x5624eb2576b0 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1af650 con 0x5624eb251b50 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1ad670 con 0x5624eb252540 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb24bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb24bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:50160/0 (socket says 172.21.15.116:50160) 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f0a00640 1 -- 172.21.15.116:0/9015991 learned_addr learned my addr 172.21.15.116:0/9015991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb24c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.201+0000 7fe3e3400640 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb263400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3f0a00640 1 -- 172.21.15.116:0/9015991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 msgr2=0x5624eb263400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb263400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3f0a00640 1 -- 172.21.15.116:0/9015991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 msgr2=0x5624eb24c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb24c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3f0a00640 1 -- 172.21.15.116:0/9015991 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624eb24de30 con 0x5624eb251b50 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3e3400640 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb263400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb24bad0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe3e402c640 tx=0x7fe3e404a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f53eedba97daf51b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3dbe00640 1 -- 172.21.15.116:0/9015991 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4039040 con 0x5624eb251b50 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3dbe00640 1 -- 172.21.15.116:0/9015991 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3e4003e50 con 0x5624eb251b50 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.202+0000 7fe3dbe00640 1 -- 172.21.15.116:0/9015991 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4043040 con 0x5624eb251b50 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 -- 172.21.15.116:0/9015991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 msgr2=0x5624eb24bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb24bad0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe3e402c640 tx=0x7fe3e404a040 comp rx=0 tx=0).stop 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 -- 172.21.15.116:0/9015991 shutdown_connections 2024-03-04T07:47:45.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb2576b0 0x5624eb24c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb252540 0x5624eb263400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/9015991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb251b50 0x5624eb24bad0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 -- 172.21.15.116:0/9015991 >> 172.21.15.116:0/9015991 conn(0x5624eb24d0c0 msgr2=0x5624eb250320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 -- 172.21.15.116:0/9015991 shutdown_connections 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.203+0000 7fe3f57bf740 1 -- 172.21.15.116:0/9015991 wait complete. 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 -- start start 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 Processor -- start 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb251b50 0x5624eb261530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb252540 0x5624eb261a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb262910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1adcd0 con 0x5624eb252540 2024-03-04T07:47:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1aeff0 con 0x5624eb2576b0 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1ae330 con 0x5624eb251b50 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3f57bf740 10 client.0 _ll_register_callbacks cb 0x7fe3d4009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3e3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb252540 0x5624eb261a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3e3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb252540 0x5624eb261a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37472/0 (socket says 172.21.15.116:37472) 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.204+0000 7fe3e3400640 1 -- 172.21.15.116:0/2637988889 learned_addr learned my addr 172.21.15.116:0/2637988889 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb262910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb251b50 0x5624eb261530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3e3400640 1 -- 172.21.15.116:0/2637988889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb251b50 msgr2=0x5624eb261530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3e3400640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb251b50 0x5624eb261530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3e3400640 1 -- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 msgr2=0x5624eb262910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3e3400640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb262910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3e3400640 1 -- 172.21.15.116:0/2637988889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624eb262fb0 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb262910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3e3400640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb252540 0x5624eb261a70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5624eb25d3a0 tx=0x7fe3d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85951f65ec2e9a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3d9600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3d9600640 1 -- 172.21.15.116:0/2637988889 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d401c630 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3d9600640 1 -- 172.21.15.116:0/2637988889 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe3d401cc50 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.205+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3e4003e50 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3d9600640 1 -- 172.21.15.116:0/2637988889 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3d4036050 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 10 client.16953 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5624eb2c3cf0 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3d9600640 1 -- 172.21.15.116:0/2637988889 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3e4003e50 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3d9600640 1 -- 172.21.15.116:0/2637988889 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe3d4003890 con 0x5624eb252540 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 10 client.16953 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:45.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 10 client.16953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 20 client.16953 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb2c45f0 0x5624eb2c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 4 client.16953 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5624eb1be5c0 con 0x5624eb2c45f0 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.206+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb2c45f0 0x5624eb2c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f0a00640 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb2c45f0 0x5624eb2c6a10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17a3a53512d83a38 server_cookie=a2b7a7a903e4a5f3 in_seq=0 out_seq=0 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3d9600640 10 client.16953 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3d9600640 1 -- 172.21.15.116:0/2637988889 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5624eb1be5c0 con 0x5624eb2c45f0 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3d9600640 10 client.16953 handle_command_reply: tid=0 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 client.16953 shutdown 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb252540 msgr2=0x5624eb261a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb252540 0x5624eb261a70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x5624eb25d3a0 tx=0x7fe3d4005880 comp rx=0 tx=0).stop 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 shutdown_connections 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5624eb2c45f0 0x5624eb2c6a10 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5624eb2576b0 0x5624eb262910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624eb252540 0x5624eb261a70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3d9600640 0 client.16953 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 --2- 172.21.15.116:0/2637988889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5624eb251b50 0x5624eb261530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 >> 172.21.15.116:0/2637988889 conn(0x5624eb24d0c0 msgr2=0x5624eb24bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.207+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 shutdown_connections 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f57bf740 1 -- 172.21.15.116:0/2637988889 wait complete. 2024-03-04T07:47:45.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f57bf740 20 client.16953 trim_cache size 0 max 16384 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: "036313f6-e14a-4d45-af4d-3d1afcb8d486": { 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "036313f6-e14a-4d45-af4d-3d1afcb8d486", 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:47:45.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:47:45.217 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:47:45.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3d00a1d80 msgr2=0x7fe3d00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3d00a1d80 0x7fe3d00a4240 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc097830 tx=0x7fe3dc0d1bb0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec107bf0 msgr2=0x7fe3ec19de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec107bf0 0x7fe3ec19de80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00e9c0 tx=0x7fe3dc00ee80 comp rx=0 tx=0).stop 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 shutdown_connections 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3d00a1d80 0x7fe3d00a4240 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3ec1097a0 0x7fe3ec1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3ec108de0 0x7fe3ec19e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 --2- 172.21.15.116:0/3406570060 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe3ec107bf0 0x7fe3ec19de80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 >> 172.21.15.116:0/3406570060 conn(0x7fe3ec076b90 msgr2=0x7fe3ec10c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 shutdown_connections 2024-03-04T07:47:45.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:47:45.208+0000 7fe3f2a00640 1 -- 172.21.15.116:0/3406570060 wait complete. 2024-03-04T07:47:45.226 INFO:tasks.cephfs.filesystem:scrub status for tag:036313f6-e14a-4d45-af4d-3d1afcb8d486 - {'path': '/', 'tag': '036313f6-e14a-4d45-af4d-3d1afcb8d486', 'options': 'recursive,force'} 2024-03-04T07:47:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:44 smithi152 ceph-mon[37154]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-04T07:47:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:44 smithi152 ceph-mon[37154]: Unable to set osd_memory_target on smithi152 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-04T07:47:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:44 smithi152 ceph-mon[37154]: pgmap v610: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:44 smithi133 ceph-mon[36505]: Adjusting osd_memory_target on smithi152 to 174.1M 2024-03-04T07:47:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:44 smithi133 ceph-mon[36505]: Unable to set osd_memory_target on smithi152 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-04T07:47:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:44 smithi133 ceph-mon[36505]: pgmap v610: 97 pgs: 97 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:45 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:47:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:45 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:45 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:47:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:45 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:45 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:47:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:45 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:45 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:47:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:45 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:46.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:45 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:47:46.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:45 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:46.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:45 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:47:46.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:45 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:47:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 -- 172.21.15.116:0/2919689797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 msgr2=0x7f8e60101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2919689797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60101860 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009900 tx=0x7f8e5002bc30 comp rx=0 tx=0).stop 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 -- 172.21.15.116:0/2919689797 shutdown_connections 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2919689797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60102ff0 0x7f8e60109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2919689797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60102630 0x7f8e60102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2919689797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60101860 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 -- 172.21.15.116:0/2919689797 >> 172.21.15.116:0/2919689797 conn(0x7f8e600fc9b0 msgr2=0x7f8e600fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.251+0000 7f8e5f200640 1 -- 172.21.15.116:0/2919689797 shutdown_connections 2024-03-04T07:47:47.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.252+0000 7f8e5f200640 1 -- 172.21.15.116:0/2919689797 wait complete. 2024-03-04T07:47:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.252+0000 7f8e5f200640 1 Processor -- start 2024-03-04T07:47:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5f200640 1 -- start start 2024-03-04T07:47:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5f200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5f200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60102630 0x7f8e600739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5f200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60102ff0 0x7f8e60073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5f200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6010d0f0 con 0x7f8e60102ff0 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5f200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6010cf70 con 0x7f8e60101440 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5f200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6010d270 con 0x7f8e60102630 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60102ff0 0x7f8e60073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60075380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37012/0 (socket says 172.21.15.116:37012) 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e5de00640 1 -- 172.21.15.116:0/2704855867 learned_addr learned my addr 172.21.15.116:0/2704855867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.253+0000 7f8e55400640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60102630 0x7f8e600739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5de00640 1 -- 172.21.15.116:0/2704855867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60102630 msgr2=0x7f8e600739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5de00640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60102630 0x7f8e600739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5de00640 1 -- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60102ff0 msgr2=0x7f8e60073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5de00640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60102ff0 0x7f8e60073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5de00640 1 -- 172.21.15.116:0/2704855867 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e60074790 con 0x7f8e60101440 2024-03-04T07:47:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5de00640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60075380 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8e50000c00 tx=0x7f8e5002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e57400640 1 -- 172.21.15.116:0/2704855867 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50043020 con 0x7f8e60101440 2024-03-04T07:47:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e57400640 1 -- 172.21.15.116:0/2704855867 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e500054e0 con 0x7f8e60101440 2024-03-04T07:47:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e601a1880 con 0x7f8e60101440 2024-03-04T07:47:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e57400640 1 -- 172.21.15.116:0/2704855867 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50003ce0 con 0x7f8e60101440 2024-03-04T07:47:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.254+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e601a1df0 con 0x7f8e60101440 2024-03-04T07:47:47.257 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:46 smithi116 ceph-mon[32150]: pgmap v611: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.257+0000 7f8e57400640 1 -- 172.21.15.116:0/2704855867 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8e50046020 con 0x7f8e60101440 2024-03-04T07:47:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 Processor -- start 2024-03-04T07:47:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 -- start start 2024-03-04T07:47:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a911ffe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91204f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91207a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a91165370 con 0x558a911ffa10 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a9110e550 con 0x558a91205570 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e6767a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a91161410 con 0x558a91200400 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e5e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a911ffe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.258+0000 7f8e55400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91207a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e5e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a911ffe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56524/0 (socket says 172.21.15.116:56524) 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e5e800640 1 -- 172.21.15.116:0/3237507857 learned_addr learned my addr 172.21.15.116:0/3237507857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e5de00640 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91204f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e55400640 1 -- 172.21.15.116:0/3237507857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 msgr2=0x558a91204f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e55400640 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91204f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e55400640 1 -- 172.21.15.116:0/3237507857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 msgr2=0x558a911ffe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e55400640 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a911ffe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e55400640 1 -- 172.21.15.116:0/3237507857 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a911223c0 con 0x558a91205570 2024-03-04T07:47:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e5e800640 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a911ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e57400640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1df0 0x7f8e440a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e55400640 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91207a00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800bbc0 tx=0x7f8e480107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7592848df630bbe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e57400640 1 -- 172.21.15.116:0/2704855867 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8e500efe00 con 0x7f8e60101440 2024-03-04T07:47:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e5e800640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1df0 0x7f8e440a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e4f000640 1 -- 172.21.15.116:0/3237507857 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48004030 con 0x558a91205570 2024-03-04T07:47:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.259+0000 7f8e4f000640 1 -- 172.21.15.116:0/3237507857 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e480041d0 con 0x558a91205570 2024-03-04T07:47:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e4f000640 1 -- 172.21.15.116:0/3237507857 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e480116a0 con 0x558a91205570 2024-03-04T07:47:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e6767a740 1 -- 172.21.15.116:0/3237507857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 msgr2=0x558a91207a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e5e800640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1df0 0x7f8e440a42b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x558a91200c60 tx=0x7f8e58009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e6767a740 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91207a00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800bbc0 tx=0x7f8e480107a0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e6767a740 1 -- 172.21.15.116:0/3237507857 shutdown_connections 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e6767a740 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91207a00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e6767a740 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91204f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e6767a740 1 --2- 172.21.15.116:0/3237507857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a911ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.260+0000 7f8e6767a740 1 -- 172.21.15.116:0/3237507857 >> 172.21.15.116:0/3237507857 conn(0x558a911faf80 msgr2=0x558a911fb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.261+0000 7f8e6767a740 1 -- 172.21.15.116:0/3237507857 shutdown_connections 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.261+0000 7f8e6767a740 1 -- 172.21.15.116:0/3237507857 wait complete. 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.261+0000 7f8e6767a740 1 -- start start 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.261+0000 7f8e6767a740 1 Processor -- start 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a912112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e6767a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a91208520 con 0x558a911ffa10 2024-03-04T07:47:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e6767a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a911219b0 con 0x558a91200400 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e6767a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a912086a0 con 0x558a91205570 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e6767a740 10 client.0 _ll_register_callbacks cb 0x558a9115d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e55400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e55400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37036/0 (socket says 172.21.15.116:37036) 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e55400640 1 -- 172.21.15.116:0/1806856960 learned_addr learned my addr 172.21.15.116:0/1806856960 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e5de00640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e5e800640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a912112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e5de00640 1 -- 172.21.15.116:0/1806856960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 msgr2=0x558a912112a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e5de00640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a912112a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e5de00640 1 -- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 msgr2=0x558a9120a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e5de00640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e5de00640 1 -- 172.21.15.116:0/1806856960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a90e7a210 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.262+0000 7f8e55400640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e5de00640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209d20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x558a91200e70 tx=0x7f8e500bc8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab112a433b99c06 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e43400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e43400640 1 -- 172.21.15.116:0/1806856960 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50004c80 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e43400640 1 -- 172.21.15.116:0/1806856960 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8e500ed9d0 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e480041d0 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e43400640 1 -- 172.21.15.116:0/1806856960 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e500044a0 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e6767a740 10 client.16965 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558a90f46620 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e43400640 1 -- 172.21.15.116:0/1806856960 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e480041d0 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.263+0000 7f8e43400640 1 -- 172.21.15.116:0/1806856960 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8e500c4e00 con 0x558a911ffa10 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e6767a740 10 client.16965 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e6767a740 10 client.16965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e6767a740 20 client.16965 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e6767a740 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91215ba0 0x558a91218010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e6767a740 4 client.16965 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a911728b0 con 0x558a91215ba0 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e55400640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91215ba0 0x558a91218010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e55400640 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91215ba0 0x558a91218010 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e27bf6b067151953 server_cookie=89f399af88119447 in_seq=0 out_seq=0 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.264+0000 7f8e43400640 10 client.16965 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e43400640 1 -- 172.21.15.116:0/1806856960 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558a911728b0 con 0x558a91215ba0 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e43400640 10 client.16965 handle_command_reply: tid=0 2024-03-04T07:47:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 client.16965 shutdown 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 msgr2=0x558a91209d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209d20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x558a91200e70 tx=0x7f8e500bc8c0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 shutdown_connections 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91215ba0 0x558a91218010 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a912112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 --2- 172.21.15.116:0/1806856960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209d20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e43400640 0 client.16965 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.265+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 >> 172.21.15.116:0/1806856960 conn(0x558a911faf80 msgr2=0x558a911fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.266+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 shutdown_connections 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.266+0000 7f8e6767a740 1 -- 172.21.15.116:0/1806856960 wait complete. 2024-03-04T07:47:47.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.266+0000 7f8e6767a740 20 client.16965 trim_cache size 0 max 16384 2024-03-04T07:47:47.273 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:46 smithi152 ceph-mon[37154]: pgmap v611: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.281+0000 7f8e6767a740 1 Processor -- start 2024-03-04T07:47:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.281+0000 7f8e6767a740 1 -- start start 2024-03-04T07:47:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91211260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91209e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e6767a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601a8a50 con 0x558a911ffa10 2024-03-04T07:47:47.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e6767a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601ab030 con 0x558a91205570 2024-03-04T07:47:47.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e6767a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601a9050 con 0x558a91200400 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91209e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91209e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37052/0 (socket says 172.21.15.116:37052) 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5e800640 1 -- 172.21.15.116:0/4101538342 learned_addr learned my addr 172.21.15.116:0/4101538342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e55400640 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91211260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5de00640 1 -- 172.21.15.116:0/4101538342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 msgr2=0x558a91211260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5de00640 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91211260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5de00640 1 -- 172.21.15.116:0/4101538342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 msgr2=0x558a91209e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5de00640 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91209e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5de00640 1 -- 172.21.15.116:0/4101538342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a911ff000 con 0x558a911ffa10 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.282+0000 7f8e5e800640 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91209e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.283+0000 7f8e5de00640 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8e500bca80 tx=0x7f8e500c0d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1468f42a9b48aa5f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.283+0000 7f8e4e600640 1 -- 172.21.15.116:0/4101538342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500bc630 con 0x558a911ffa10 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.283+0000 7f8e4e600640 1 -- 172.21.15.116:0/4101538342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e50004c80 con 0x558a911ffa10 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.283+0000 7f8e4e600640 1 -- 172.21.15.116:0/4101538342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500ed940 con 0x558a911ffa10 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.283+0000 7f8e6767a740 1 -- 172.21.15.116:0/4101538342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 msgr2=0x558a91209930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.283+0000 7f8e6767a740 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8e500bca80 tx=0x7f8e500c0d30 comp rx=0 tx=0).stop 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 -- 172.21.15.116:0/4101538342 shutdown_connections 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91205570 0x558a91209e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91200400 0x558a91211260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 --2- 172.21.15.116:0/4101538342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a91209930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 -- 172.21.15.116:0/4101538342 >> 172.21.15.116:0/4101538342 conn(0x558a911faf80 msgr2=0x558a91211cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 -- 172.21.15.116:0/4101538342 shutdown_connections 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 -- 172.21.15.116:0/4101538342 wait complete. 2024-03-04T07:47:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 -- start start 2024-03-04T07:47:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.284+0000 7f8e6767a740 1 Processor -- start 2024-03-04T07:47:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a9120e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e6767a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a9120fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e6767a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601a96b0 con 0x558a911ffa10 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e6767a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601aa9d0 con 0x558a91200400 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e6767a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601a9d10 con 0x558a91205570 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e6767a740 10 client.0 _ll_register_callbacks cb 0x7f8e50003ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a9120e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e55400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a9120e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56552/0 (socket says 172.21.15.116:56552) 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e55400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:37062/0 (socket says 172.21.15.116:37062) 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 -- 172.21.15.116:0/719253523 learned_addr learned my addr 172.21.15.116:0/719253523 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5e800640 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a9120fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 -- 172.21.15.116:0/719253523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 msgr2=0x558a9120fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a9120fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 -- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 msgr2=0x558a9120eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e5de00640 1 -- 172.21.15.116:0/719253523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a91210150 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.285+0000 7f8e55400640 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e5de00640 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a9120e610 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8e500b8f50 tx=0x7f8e50009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43656ccd61ffa5fc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e43400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e43400640 1 -- 172.21.15.116:0/719253523 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50005690 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e43400640 1 -- 172.21.15.116:0/719253523 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8e500bc150 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e50004c80 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e43400640 1 -- 172.21.15.116:0/719253523 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e500c5290 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 10 client.16971 fetch_fsmap learned FSMap version 19 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558a91210150 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e43400640 1 -- 172.21.15.116:0/719253523 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e50004c80 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e43400640 1 -- 172.21.15.116:0/719253523 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8e50005060 con 0x558a911ffa10 2024-03-04T07:47:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 10 client.16971 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 10 client.16971 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 20 client.16971 populate_metadata read hostname 'smithi116' 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91271600 0x558a912739f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 4 client.16971 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.286+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558a9116cc00 con 0x558a91271600 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.287+0000 7f8e55400640 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91271600 0x558a912739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.287+0000 7f8e55400640 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91271600 0x558a912739f0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1646857386ad0663 server_cookie=41f3348e843366f0 in_seq=0 out_seq=0 2024-03-04T07:47:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.287+0000 7f8e43400640 10 client.16971 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.301+0000 7f8e43400640 1 -- 172.21.15.116:0/719253523 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558a9116cc00 con 0x558a91271600 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.301+0000 7f8e43400640 10 client.16971 handle_command_reply: tid=0 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 client.16971 shutdown 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 msgr2=0x558a9120e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a9120e610 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8e500b8f50 tx=0x7f8e50009560 comp rx=0 tx=0).stop 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 shutdown_connections 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a91271600 0x558a912739f0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a91205570 0x558a9120fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a91200400 0x558a9120eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 --2- 172.21.15.116:0/719253523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a911ffa10 0x558a9120e610 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e43400640 0 client.16971 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 >> 172.21.15.116:0/719253523 conn(0x558a911faf80 msgr2=0x558a912098c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 shutdown_connections 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 1 -- 172.21.15.116:0/719253523 wait complete. 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.302+0000 7f8e6767a740 20 client.16971 trim_cache size 0 max 16384 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:47:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:47:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:47:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:47:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1df0 msgr2=0x7f8e440a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1df0 0x7f8e440a42b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x558a91200c60 tx=0x7f8e58009500 comp rx=0 tx=0).stop 2024-03-04T07:47:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 msgr2=0x7f8e60075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:47:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60075380 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8e50000c00 tx=0x7f8e5002c6c0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 shutdown_connections 2024-03-04T07:47:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e60102ff0 0x7f8e60073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8e440a1df0 0x7f8e440a42b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e60102630 0x7f8e600739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 --2- 172.21.15.116:0/2704855867 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8e60101440 0x7f8e60075380 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:47:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 >> 172.21.15.116:0/2704855867 conn(0x7f8e600fc9b0 msgr2=0x7f8e60105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:47:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 shutdown_connections 2024-03-04T07:47:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:47:47.303+0000 7f8e5f200640 1 -- 172.21.15.116:0/2704855867 wait complete. 2024-03-04T07:47:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:46 smithi133 ceph-mon[36505]: pgmap v611: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:47:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:47 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:47:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:47 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:47:48.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:47 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:47:49.172 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:48 smithi152 ceph-mon[37154]: pgmap v612: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:48 smithi116 ceph-mon[32150]: pgmap v612: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:49.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:48 smithi133 ceph-mon[36505]: pgmap v612: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:47:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:50 smithi116 ceph-mon[32150]: pgmap v613: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:47:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:50 smithi152 ceph-mon[37154]: pgmap v613: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:47:51.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:50 smithi133 ceph-mon[36505]: pgmap v613: 97 pgs: 97 active+clean; 1.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:47:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:52 smithi116 ceph-mon[32150]: pgmap v614: 97 pgs: 97 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:52 smithi152 ceph-mon[37154]: pgmap v614: 97 pgs: 97 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:52 smithi133 ceph-mon[36505]: pgmap v614: 97 pgs: 97 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:47:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:54 smithi116 ceph-mon[32150]: pgmap v615: 97 pgs: 97 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:54 smithi152 ceph-mon[37154]: pgmap v615: 97 pgs: 97 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:55.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:54 smithi133 ceph-mon[36505]: pgmap v615: 97 pgs: 97 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:47:57.139 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:56 smithi133 ceph-mon[36505]: pgmap v616: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:47:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:56 smithi116 ceph-mon[32150]: pgmap v616: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:47:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:56 smithi152 ceph-mon[37154]: pgmap v616: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:47:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:47:58 smithi116 ceph-mon[32150]: pgmap v617: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:47:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:47:58 smithi152 ceph-mon[37154]: pgmap v617: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:47:59.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:47:58 smithi133 ceph-mon[36505]: pgmap v617: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.444+0000 7f6939c00640 1 -- 172.21.15.116:0/4159618357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 msgr2=0x7f6934110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.444+0000 7f6939c00640 1 --2- 172.21.15.116:0/4159618357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 0x7f6934110070 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f692c009ca0 tx=0x7f692c02c0f0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.445+0000 7f6939c00640 1 -- 172.21.15.116:0/4159618357 shutdown_connections 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.445+0000 7f6939c00640 1 --2- 172.21.15.116:0/4159618357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 0x7f6934110070 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.445+0000 7f6939c00640 1 --2- 172.21.15.116:0/4159618357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934108e20 0x7f69341092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.445+0000 7f6939c00640 1 --2- 172.21.15.116:0/4159618357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6934107c30 0x7f6934108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.445+0000 7f6939c00640 1 -- 172.21.15.116:0/4159618357 >> 172.21.15.116:0/4159618357 conn(0x7f6934076be0 msgr2=0x7f6934079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.445+0000 7f6939c00640 1 -- 172.21.15.116:0/4159618357 shutdown_connections 2024-03-04T07:48:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.445+0000 7f6939c00640 1 -- 172.21.15.116:0/4159618357 wait complete. 2024-03-04T07:48:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.446+0000 7f6939c00640 1 Processor -- start 2024-03-04T07:48:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.446+0000 7f6939c00640 1 -- start start 2024-03-04T07:48:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6939c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934107c30 0x7f693419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6939c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6934108e20 0x7f693419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6939c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 0x7f69341a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6939c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69341138b0 con 0x7f6934107c30 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6939c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934113730 con 0x7f69341097e0 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6939c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6934113a30 con 0x7f6934108e20 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6932a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934107c30 0x7f693419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6932a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934107c30 0x7f693419de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41552/0 (socket says 172.21.15.116:41552) 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6932a00640 1 -- 172.21.15.116:0/3398486889 learned_addr learned my addr 172.21.15.116:0/3398486889 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6933400640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 0x7f69341a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f692be00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6934108e20 0x7f693419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6933400640 1 -- 172.21.15.116:0/3398486889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6934108e20 msgr2=0x7f693419e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6933400640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6934108e20 0x7f693419e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6933400640 1 -- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934107c30 msgr2=0x7f693419de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6933400640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934107c30 0x7f693419de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6933400640 1 -- 172.21.15.116:0/3398486889 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69341a5a80 con 0x7f69341097e0 2024-03-04T07:48:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.447+0000 7f6932a00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934107c30 0x7f693419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.448+0000 7f6933400640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 0x7f69341a5410 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f692c037040 tx=0x7f692c032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.448+0000 7f692b400640 1 -- 172.21.15.116:0/3398486889 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c041020 con 0x7f69341097e0 2024-03-04T07:48:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.448+0000 7f692b400640 1 -- 172.21.15.116:0/3398486889 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692c034cd0 con 0x7f69341097e0 2024-03-04T07:48:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.448+0000 7f692b400640 1 -- 172.21.15.116:0/3398486889 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c039040 con 0x7f69341097e0 2024-03-04T07:48:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.448+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69341a5d10 con 0x7f69341097e0 2024-03-04T07:48:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.448+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69341a6180 con 0x7f69341097e0 2024-03-04T07:48:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.451+0000 7f692b400640 1 -- 172.21.15.116:0/3398486889 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f692c044020 con 0x7f69341097e0 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.451+0000 7f693c861740 1 Processor -- start 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.451+0000 7f693c861740 1 -- start start 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f693c861740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd777c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f693c861740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd778240 0x55aebd77cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f693c861740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd77f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f693c861740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aebd6dd370 con 0x55aebd77d3b0 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f693c861740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aebd686550 con 0x55aebd777850 2024-03-04T07:48:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f693c861740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aebd6d9410 con 0x55aebd778240 2024-03-04T07:48:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f692be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd777c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f692be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd777c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51178/0 (socket says 172.21.15.116:51178) 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f692be00640 1 -- 172.21.15.116:0/1429708451 learned_addr learned my addr 172.21.15.116:0/1429708451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f6932a00640 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd778240 0x55aebd77cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.452+0000 7f692be00640 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd77f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6932a00640 1 -- 172.21.15.116:0/1429708451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 msgr2=0x55aebd777c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6932a00640 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd777c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6932a00640 1 -- 172.21.15.116:0/1429708451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 msgr2=0x55aebd77f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6932a00640 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd77f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6932a00640 1 -- 172.21.15.116:0/1429708451 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aebd69a3c0 con 0x55aebd778240 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6932a00640 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd778240 0x55aebd77cd40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f692400bfd0 tx=0x7f6924010930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80fe600974aead75 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f692b400640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69180a1e20 0x7f69180a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f692b400640 1 -- 172.21.15.116:0/3398486889 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f692c0efc20 con 0x7f69341097e0 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6923e00640 1 -- 172.21.15.116:0/1429708451 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924011760 con 0x55aebd778240 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f692be00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69180a1e20 0x7f69180a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6923e00640 1 -- 172.21.15.116:0/1429708451 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6924011de0 con 0x55aebd778240 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.453+0000 7f6923e00640 1 -- 172.21.15.116:0/1429708451 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692401a380 con 0x55aebd778240 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 -- 172.21.15.116:0/1429708451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd778240 msgr2=0x55aebd77cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f692be00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69180a1e20 0x7f69180a42e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55aebd778aa0 tx=0x7f691c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd778240 0x55aebd77cd40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f692400bfd0 tx=0x7f6924010930 comp rx=0 tx=0).stop 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 -- 172.21.15.116:0/1429708451 shutdown_connections 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd77f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd778240 0x55aebd77cd40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 --2- 172.21.15.116:0/1429708451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd777c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 -- 172.21.15.116:0/1429708451 >> 172.21.15.116:0/1429708451 conn(0x55aebd772dc0 msgr2=0x55aebd7731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 -- 172.21.15.116:0/1429708451 shutdown_connections 2024-03-04T07:48:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.454+0000 7f693c861740 1 -- 172.21.15.116:0/1429708451 wait complete. 2024-03-04T07:48:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 -- start start 2024-03-04T07:48:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 Processor -- start 2024-03-04T07:48:00.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd777850 0x55aebd781b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd7820a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd789070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aebd780360 con 0x55aebd77d3b0 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aebd6999b0 con 0x55aebd778240 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aebd7804e0 con 0x55aebd777850 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.455+0000 7f693c861740 10 client.0 _ll_register_callbacks cb 0x55aebd6d5340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd7820a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd7820a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51188/0 (socket says 172.21.15.116:51188) 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f6933400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd789070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 -- 172.21.15.116:0/3164488144 learned_addr learned my addr 172.21.15.116:0/3164488144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f6932a00640 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd777850 0x55aebd781b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 -- 172.21.15.116:0/3164488144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd777850 msgr2=0x55aebd781b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd777850 0x55aebd781b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 -- 172.21.15.116:0/3164488144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 msgr2=0x55aebd789070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd789070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 -- 172.21.15.116:0/3164488144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aebd3f2210 con 0x55aebd778240 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f6933400640 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd789070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f692be00640 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd7820a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f691c003b10 tx=0x7f691c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85ffa6b80e38ae2e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.456+0000 7f6920c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f6920c00640 1 -- 172.21.15.116:0/3164488144 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c005020 con 0x55aebd778240 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f6920c00640 1 -- 172.21.15.116:0/3164488144 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f691c01a050 con 0x55aebd778240 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6924011de0 con 0x55aebd778240 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f6920c00640 1 -- 172.21.15.116:0/3164488144 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f691c0051c0 con 0x55aebd778240 2024-03-04T07:48:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f693c861740 10 client.27110 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55aebd4be620 con 0x55aebd778240 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f6920c00640 1 -- 172.21.15.116:0/3164488144 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6924011de0 con 0x55aebd778240 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f6920c00640 1 -- 172.21.15.116:0/3164488144 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f691c0206b0 con 0x55aebd778240 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f693c861740 10 client.27110 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.457+0000 7f693c861740 10 client.27110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.458+0000 7f693c861740 20 client.27110 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.458+0000 7f693c861740 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd78fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.458+0000 7f693c861740 4 client.27110 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.458+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aebd6ea8b0 con 0x55aebd78da00 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.458+0000 7f6932a00640 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd78fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.458+0000 7f6932a00640 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd78fe40 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a383f694cb55f957 server_cookie=c3992ea6ed2a08c0 in_seq=0 out_seq=0 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.458+0000 7f6920c00640 10 client.27110 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f6920c00640 1 -- 172.21.15.116:0/3164488144 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aebd6ea8b0 con 0x55aebd78da00 2024-03-04T07:48:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f6920c00640 10 client.27110 handle_command_reply: tid=0 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 client.27110 shutdown 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 msgr2=0x55aebd7820a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd7820a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f691c003b10 tx=0x7f691c004730 comp rx=0 tx=0).stop 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 shutdown_connections 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd78fe40 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f6920c00640 0 client.27110 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd77d3b0 0x55aebd789070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd7820a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 --2- 172.21.15.116:0/3164488144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd777850 0x55aebd781b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.459+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 >> 172.21.15.116:0/3164488144 conn(0x55aebd772dc0 msgr2=0x55aebd771fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.460+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 shutdown_connections 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.460+0000 7f693c861740 1 -- 172.21.15.116:0/3164488144 wait complete. 2024-03-04T07:48:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.460+0000 7f693c861740 20 client.27110 trim_cache size 0 max 16384 2024-03-04T07:48:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.474+0000 7f693c861740 1 Processor -- start 2024-03-04T07:48:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f693c861740 1 -- start start 2024-03-04T07:48:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f693c861740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd776730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f693c861740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd778240 0x55aebd789030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f693c861740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd794740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f693c861740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69341acfa0 con 0x55aebd778240 2024-03-04T07:48:00.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f693c861740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69341af580 con 0x55aebd777850 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f693c861740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69341ad5a0 con 0x55aebd77d3b0 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f6932a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd776730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f6932a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd776730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51198/0 (socket says 172.21.15.116:51198) 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f6932a00640 1 -- 172.21.15.116:0/1255505115 learned_addr learned my addr 172.21.15.116:0/1255505115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f692be00640 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd778240 0x55aebd789030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f6933400640 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd794740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f6932a00640 1 -- 172.21.15.116:0/1255505115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 msgr2=0x55aebd794740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.475+0000 7f6932a00640 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd794740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f6932a00640 1 -- 172.21.15.116:0/1255505115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd778240 msgr2=0x55aebd789030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f6932a00640 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd778240 0x55aebd789030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f6932a00640 1 -- 172.21.15.116:0/1255505115 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aebd76e610 con 0x55aebd777850 2024-03-04T07:48:00.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f692be00640 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd778240 0x55aebd789030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:00.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f6932a00640 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd776730 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f692400bfd0 tx=0x7f692400b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f87723b9c9bc52c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:00.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f6923400640 1 -- 172.21.15.116:0/1255505115 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924029030 con 0x55aebd777850 2024-03-04T07:48:00.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f6923400640 1 -- 172.21.15.116:0/1255505115 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692402d020 con 0x55aebd777850 2024-03-04T07:48:00.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.476+0000 7f6923400640 1 -- 172.21.15.116:0/1255505115 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924029700 con 0x55aebd777850 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 -- 172.21.15.116:0/1255505115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 msgr2=0x55aebd776730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd776730 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f692400bfd0 tx=0x7f692400b410 comp rx=0 tx=0).stop 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 -- 172.21.15.116:0/1255505115 shutdown_connections 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd794740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd778240 0x55aebd789030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 --2- 172.21.15.116:0/1255505115 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd777850 0x55aebd776730 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 -- 172.21.15.116:0/1255505115 >> 172.21.15.116:0/1255505115 conn(0x55aebd772dc0 msgr2=0x55aebd776020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 -- 172.21.15.116:0/1255505115 shutdown_connections 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 -- 172.21.15.116:0/1255505115 wait complete. 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.477+0000 7f693c861740 1 -- start start 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 1 Processor -- start 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd777850 0x55aebd7865d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd786b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd787ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69341adc00 con 0x55aebd777850 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69341aef20 con 0x55aebd778240 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69341ae260 con 0x55aebd77d3b0 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f693c861740 10 client.0 _ll_register_callbacks cb 0x7f691c003900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f6932a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd777850 0x55aebd7865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f692be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd786b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f692be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd786b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:51200/0 (socket says 172.21.15.116:51200) 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f692be00640 1 -- 172.21.15.116:0/3013897332 learned_addr learned my addr 172.21.15.116:0/3013897332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f6932a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd777850 0x55aebd7865d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41596/0 (socket says 172.21.15.116:41596) 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.478+0000 7f6933400640 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd787ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f692be00640 1 -- 172.21.15.116:0/3013897332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 msgr2=0x55aebd787ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f692be00640 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd787ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f692be00640 1 -- 172.21.15.116:0/3013897332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd777850 msgr2=0x55aebd7865d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f692be00640 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd777850 0x55aebd7865d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f692be00640 1 -- 172.21.15.116:0/3013897332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aebd7880a0 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f692be00640 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd786b10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f691c020ac0 tx=0x7f691c011540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e350f648237933f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f6920c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f6920c00640 1 -- 172.21.15.116:0/3013897332 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c03a070 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f6920c00640 1 -- 172.21.15.116:0/3013897332 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f691c028040 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.479+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f692402d020 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f6920c00640 1 -- 172.21.15.116:0/3013897332 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f691c003fd0 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 10 client.27116 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55aebd7e9310 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f6920c00640 1 -- 172.21.15.116:0/3013897332 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692402d020 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f6920c00640 1 -- 172.21.15.116:0/3013897332 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f691c036410 con 0x55aebd778240 2024-03-04T07:48:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 10 client.27116 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 10 client.27116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 20 client.27116 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd7ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 4 client.27116 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55aebd6e4c00 con 0x55aebd78da00 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.480+0000 7f6932a00640 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd7ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.481+0000 7f6932a00640 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd7ebb40 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50695eb0a9eb0999 server_cookie=f897d97c103322f9 in_seq=0 out_seq=0 2024-03-04T07:48:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.481+0000 7f6920c00640 10 client.27116 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f6920c00640 1 -- 172.21.15.116:0/3013897332 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55aebd6e4c00 con 0x55aebd78da00 2024-03-04T07:48:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f6920c00640 10 client.27116 handle_command_reply: tid=0 2024-03-04T07:48:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 client.27116 shutdown 2024-03-04T07:48:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 msgr2=0x55aebd786b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd786b10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f691c020ac0 tx=0x7f691c011540 comp rx=0 tx=0).stop 2024-03-04T07:48:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 shutdown_connections 2024-03-04T07:48:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55aebd78da00 0x55aebd7ebb40 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aebd77d3b0 0x55aebd787ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55aebd778240 0x55aebd786b10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 --2- 172.21.15.116:0/3013897332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aebd777850 0x55aebd7865d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f6920c00640 0 client.27116 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.495+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 >> 172.21.15.116:0/3013897332 conn(0x55aebd772dc0 msgr2=0x55aebd7768b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.496+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 shutdown_connections 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.496+0000 7f693c861740 1 -- 172.21.15.116:0/3013897332 wait complete. 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.496+0000 7f693c861740 20 client.27116 trim_cache size 0 max 16384 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:48:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:48:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.496+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69180a1e20 msgr2=0x7f69180a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.496+0000 7f6939c00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69180a1e20 0x7f69180a42e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55aebd778aa0 tx=0x7f691c005f90 comp rx=0 tx=0).stop 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.496+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 msgr2=0x7f69341a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.496+0000 7f6939c00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 0x7f69341a5410 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f692c037040 tx=0x7f692c032ea0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 shutdown_connections 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69341097e0 0x7f69341a5410 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69180a1e20 0x7f69180a42e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6934108e20 0x7f693419e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 --2- 172.21.15.116:0/3398486889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6934107c30 0x7f693419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 >> 172.21.15.116:0/3398486889 conn(0x7f6934076be0 msgr2=0x7f693410c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 shutdown_connections 2024-03-04T07:48:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:00.497+0000 7f6939c00640 1 -- 172.21.15.116:0/3398486889 wait complete. 2024-03-04T07:48:01.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:00 smithi116 ceph-mon[32150]: pgmap v618: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:00 smithi152 ceph-mon[37154]: pgmap v618: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:00 smithi133 ceph-mon[36505]: pgmap v618: 97 pgs: 97 active+clean; 1.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:02 smithi116 ceph-mon[32150]: pgmap v619: 97 pgs: 97 active+clean; 1.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:48:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:02 smithi152 ceph-mon[37154]: pgmap v619: 97 pgs: 97 active+clean; 1.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:48:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:02 smithi133 ceph-mon[36505]: pgmap v619: 97 pgs: 97 active+clean; 1.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:48:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:04 smithi116 ceph-mon[32150]: pgmap v620: 97 pgs: 97 active+clean; 1.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:48:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:04 smithi152 ceph-mon[37154]: pgmap v620: 97 pgs: 97 active+clean; 1.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:48:05.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:04 smithi133 ceph-mon[36505]: pgmap v620: 97 pgs: 97 active+clean; 1.4 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:48:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:06 smithi116 ceph-mon[32150]: pgmap v621: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:48:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:06 smithi152 ceph-mon[37154]: pgmap v621: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:48:07.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:06 smithi133 ceph-mon[36505]: pgmap v621: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:48:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:08 smithi116 ceph-mon[32150]: pgmap v622: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:08 smithi152 ceph-mon[37154]: pgmap v622: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:09.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:08 smithi133 ceph-mon[36505]: pgmap v622: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:10 smithi116 ceph-mon[32150]: pgmap v623: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:10 smithi152 ceph-mon[37154]: pgmap v623: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:11.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:10 smithi133 ceph-mon[36505]: pgmap v623: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:12.615 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.615 INFO:tasks.workunit.client.0.smithi116.stdout:313/600: open fd = 3 2024-03-04T07:48:12.615 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.615 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.615 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.615 INFO:tasks.workunit.client.0.smithi116.stdout:314/600: open fd = 3 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:315/600: open fd = 3 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:316/600: open fd = 3 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.616 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:317/600: open fd = 3 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:318/600: open fd = 3 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:319/600: open fd = 3 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.617 INFO:tasks.workunit.client.0.smithi116.stdout:320/600: open fd = 3 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:321/600: open fd = 3 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:322/600: open fd = 3 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:323/600: open fd = 3 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:324/600: open fd = 3 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:325/600: open fd = 3 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.619 INFO:tasks.workunit.client.0.smithi116.stdout:326/600: open fd = 3 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:327/600: open fd = 3 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:328/600: open fd = 3 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.620 INFO:tasks.workunit.client.0.smithi116.stdout:329/600: open fd = 3 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:330/600: open fd = 3 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:331/600: open fd = 3 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.621 INFO:tasks.workunit.client.0.smithi116.stdout:332/600: open fd = 3 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:333/600: open fd = 3 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:334/600: open fd = 3 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:335/600: open fd = 3 2024-03-04T07:48:12.622 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:336/600: open fd = 3 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:337/600: open fd = 3 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:338/600: open fd = 3 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.623 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:339/600: open fd = 3 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:340/600: open fd = 3 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:341/600: open fd = 3 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.624 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:342/600: open fd = 3 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:343/600: open fd = 3 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:344/600: open fd = 3 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.625 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:345/600: open fd = 3 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:346/600: open fd = 3 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:347/600: open fd = 3 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.626 INFO:tasks.workunit.client.0.smithi116.stdout:348/600: open fd = 3 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:349/600: open fd = 3 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:350/600: open fd = 3 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:351/600: open fd = 3 2024-03-04T07:48:12.627 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:352/600: open fd = 3 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:353/600: open fd = 3 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.628 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:354/600: open fd = 3 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:355/600: open fd = 3 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:356/600: open fd = 3 2024-03-04T07:48:12.629 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:357/600: open fd = 3 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:358/600: open fd = 3 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:359/600: open fd = 3 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:360/600: open fd = 3 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:361/600: open fd = 3 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:362/600: open fd = 3 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.631 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:363/600: open fd = 3 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:364/600: open fd = 3 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:365/600: open fd = 3 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.632 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:366/600: open fd = 3 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:367/600: open fd = 3 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:368/600: open fd = 3 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.633 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:369/600: open fd = 3 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:370/600: open fd = 3 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:371/600: open fd = 3 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:372/600: open fd = 3 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:373/600: open fd = 3 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:374/600: open fd = 3 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:48:12.635 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:48:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:12 smithi116 ceph-mon[32150]: pgmap v624: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:12 smithi152 ceph-mon[37154]: pgmap v624: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:12 smithi133 ceph-mon[36505]: pgmap v624: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.639+0000 7f4ef3e00640 1 -- 172.21.15.116:0/2624745546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4101440 msgr2=0x7f4ef4101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.639+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/2624745546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4101440 0x7f4ef4101860 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009900 tx=0x7f4ee402bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.640+0000 7f4ef3e00640 1 -- 172.21.15.116:0/2624745546 shutdown_connections 2024-03-04T07:48:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.640+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/2624745546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef4102ff0 0x7f4ef4109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.640+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/2624745546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ef4102630 0x7f4ef4102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.640+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/2624745546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4101440 0x7f4ef4101860 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.640+0000 7f4ef3e00640 1 -- 172.21.15.116:0/2624745546 >> 172.21.15.116:0/2624745546 conn(0x7f4ef40fc9b0 msgr2=0x7f4ef40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.640+0000 7f4ef3e00640 1 -- 172.21.15.116:0/2624745546 shutdown_connections 2024-03-04T07:48:13.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.640+0000 7f4ef3e00640 1 -- 172.21.15.116:0/2624745546 wait complete. 2024-03-04T07:48:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 Processor -- start 2024-03-04T07:48:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 -- start start 2024-03-04T07:48:13.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ef4101440 0x7f4ef4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4102630 0x7f4ef4199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef4102ff0 0x7f4ef41a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4074210 con 0x7f4ef4102ff0 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4074090 con 0x7f4ef4102630 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.641+0000 7f4ef3e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4074390 con 0x7f4ef4101440 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4102630 0x7f4ef4199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4102630 0x7f4ef4199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55742/0 (socket says 172.21.15.116:55742) 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 -- 172.21.15.116:0/975167528 learned_addr learned my addr 172.21.15.116:0/975167528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4ef3400640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef4102ff0 0x7f4ef41a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ef4101440 0x7f4ef4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 -- 172.21.15.116:0/975167528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ef4101440 msgr2=0x7f4ef4199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ef4101440 0x7f4ef4199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 -- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef4102ff0 msgr2=0x7f4ef41a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef4102ff0 0x7f4ef41a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 -- 172.21.15.116:0/975167528 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef41a1580 con 0x7f4ef4102630 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.642+0000 7f4eebe00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4102630 0x7f4ef4199ea0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00e9f0 tx=0x7f4edc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.643+0000 7f4eeb400640 1 -- 172.21.15.116:0/975167528 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc00cd80 con 0x7f4ef4102630 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.643+0000 7f4eeb400640 1 -- 172.21.15.116:0/975167528 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4edc005500 con 0x7f4ef4102630 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.643+0000 7f4eeb400640 1 -- 172.21.15.116:0/975167528 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc010690 con 0x7f4ef4102630 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.643+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef41a1870 con 0x7f4ef4102630 2024-03-04T07:48:13.645 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.643+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef41a1d70 con 0x7f4ef4102630 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.647+0000 7f4efc5b9740 1 Processor -- start 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.647+0000 7f4efc5b9740 1 -- start start 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.647+0000 7f4eeb400640 1 -- 172.21.15.116:0/975167528 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4edc0046c0 con 0x7f4ef4102630 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56096724fa10 0x56096724fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967250400 0x560967254f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967257a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609671b5370 con 0x56096724fa10 2024-03-04T07:48:13.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56096715e550 con 0x560967250400 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609671b1410 con 0x560967255570 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56096724fa10 0x56096724fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967250400 0x560967254f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967250400 0x560967254f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55752/0 (socket says 172.21.15.116:55752) 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 -- 172.21.15.116:0/3701659863 learned_addr learned my addr 172.21.15.116:0/3701659863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56096724fa10 0x56096724fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37324/0 (socket says 172.21.15.116:37324) 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4eebe00640 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967257a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 -- 172.21.15.116:0/3701659863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 msgr2=0x560967257a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967257a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 -- 172.21.15.116:0/3701659863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56096724fa10 msgr2=0x56096724fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56096724fa10 0x56096724fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.649 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.648+0000 7f4ef2a00640 1 -- 172.21.15.116:0/3701659863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609671723c0 con 0x560967250400 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.649+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967250400 0x560967254f00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4000c00 tx=0x7f4ee4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3a67b64df47c3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.649+0000 7f4ee3e00640 1 -- 172.21.15.116:0/3701659863 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee402c980 con 0x560967250400 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.649+0000 7f4ee3e00640 1 -- 172.21.15.116:0/3701659863 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee402cb20 con 0x560967250400 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.649+0000 7f4ee3e00640 1 -- 172.21.15.116:0/3701659863 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40046c0 con 0x560967250400 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.649+0000 7f4eeb400640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4ed80a1df0 0x7f4ed80a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.649+0000 7f4eeb400640 1 -- 172.21.15.116:0/975167528 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4edc0ce140 con 0x7f4ef4102630 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.649+0000 7f4ef3400640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4ed80a1df0 0x7f4ed80a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 -- 172.21.15.116:0/3701659863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967250400 msgr2=0x560967254f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967250400 0x560967254f00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4000c00 tx=0x7f4ee4009560 comp rx=0 tx=0).stop 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 -- 172.21.15.116:0/3701659863 shutdown_connections 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967257a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967250400 0x560967254f00 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/3701659863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56096724fa10 0x56096724fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 -- 172.21.15.116:0/3701659863 >> 172.21.15.116:0/3701659863 conn(0x56096724af80 msgr2=0x56096724b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 -- 172.21.15.116:0/3701659863 shutdown_connections 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4ef3400640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4ed80a1df0 0x7f4ed80a42b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x560967250c60 tx=0x7f4eec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 -- 172.21.15.116:0/3701659863 wait complete. 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.650+0000 7f4efc5b9740 1 -- start start 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 1 Processor -- start 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x560967259cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967261270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609672584c0 con 0x560967250400 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609671719b0 con 0x560967255570 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560967258640 con 0x56096724fa10 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4efc5b9740 10 client.0 _ll_register_callbacks cb 0x5609671ad340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4eebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967261270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967261270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55766/0 (socket says 172.21.15.116:55766) 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4eebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37336/0 (socket says 172.21.15.116:37336) 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x560967259cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.651+0000 7f4ef3400640 1 -- 172.21.15.116:0/908866261 learned_addr learned my addr 172.21.15.116:0/908866261 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ef3400640 1 -- 172.21.15.116:0/908866261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 msgr2=0x560967259cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ef3400640 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x560967259cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ef3400640 1 -- 172.21.15.116:0/908866261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 msgr2=0x56096725a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ef3400640 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ef3400640 1 -- 172.21.15.116:0/908866261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560966eca210 con 0x560967255570 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4eebe00640 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ef3400640 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967261270 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4eec004070 tx=0x7f4eec004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eaf5fda5cbf4842 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ee0c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ee0c00640 1 -- 172.21.15.116:0/908866261 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec00f970 con 0x560967255570 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4ee0c00640 1 -- 172.21.15.116:0/908866261 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4eec00fb10 con 0x560967255570 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.652+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ee402cb20 con 0x560967255570 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4ee0c00640 1 -- 172.21.15.116:0/908866261 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4eec00fd10 con 0x560967255570 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 10 client.27128 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560966f96620 con 0x560967255570 2024-03-04T07:48:13.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4ee0c00640 1 -- 172.21.15.116:0/908866261 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee402cb20 con 0x560967255570 2024-03-04T07:48:13.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4ee0c00640 1 -- 172.21.15.116:0/908866261 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4eec026020 con 0x560967255570 2024-03-04T07:48:13.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 10 client.27128 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:13.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 10 client.27128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:13.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 20 client.27128 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:13.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560967265af0 0x560967267ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 4 client.27128 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:13.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609671c28b0 con 0x560967265af0 2024-03-04T07:48:13.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.653+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560967265af0 0x560967267ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.654+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560967265af0 0x560967267ee0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab05c071612671ed server_cookie=5d450c9a4b6a1a6e in_seq=0 out_seq=0 2024-03-04T07:48:13.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.654+0000 7f4ee0c00640 10 client.27128 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:13.660 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.654+0000 7f4ee0c00640 1 -- 172.21.15.116:0/908866261 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5609671c28b0 con 0x560967265af0 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.654+0000 7f4ee0c00640 10 client.27128 handle_command_reply: tid=0 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 client.27128 shutdown 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 msgr2=0x560967261270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967261270 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4eec004070 tx=0x7f4eec004ab0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 shutdown_connections 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560967265af0 0x560967267ee0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967261270 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4ee0c00640 0 client.27128 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/908866261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x560967259cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 >> 172.21.15.116:0/908866261 conn(0x56096724af80 msgr2=0x56096724a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:13.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 shutdown_connections 2024-03-04T07:48:13.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 1 -- 172.21.15.116:0/908866261 wait complete. 2024-03-04T07:48:13.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.655+0000 7f4efc5b9740 20 client.27128 trim_cache size 0 max 16384 2024-03-04T07:48:13.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 Processor -- start 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 -- start start 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56096724fa10 0x560967251780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x560967251cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967261230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41a8ac0 con 0x560967250400 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41ab0a0 con 0x56096724fa10 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.670+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41a90c0 con 0x560967255570 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56096724fa10 0x560967251780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56096724fa10 0x560967251780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55778/0 (socket says 172.21.15.116:55778) 2024-03-04T07:48:13.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 -- 172.21.15.116:0/851747101 learned_addr learned my addr 172.21.15.116:0/851747101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4eebe00640 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x560967251cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef3400640 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967261230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 -- 172.21.15.116:0/851747101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 msgr2=0x560967261230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967261230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 -- 172.21.15.116:0/851747101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 msgr2=0x560967251cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x560967251cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 -- 172.21.15.116:0/851747101 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56096724bcf0 con 0x56096724fa10 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef3400640 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967261230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4eebe00640 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x560967251cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56096724fa10 0x560967251780 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4ee402c7b0 tx=0x7f4ee402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8495d5777f8e2e96 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ee3400640 1 -- 172.21.15.116:0/851747101 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4051070 con 0x56096724fa10 2024-03-04T07:48:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.671+0000 7f4ee3400640 1 -- 172.21.15.116:0/851747101 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee4005520 con 0x56096724fa10 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4ee3400640 1 -- 172.21.15.116:0/851747101 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4034440 con 0x56096724fa10 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 -- 172.21.15.116:0/851747101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56096724fa10 msgr2=0x560967251780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56096724fa10 0x560967251780 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4ee402c7b0 tx=0x7f4ee402bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 -- 172.21.15.116:0/851747101 shutdown_connections 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560967255570 0x560967261230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x560967251cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/851747101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56096724fa10 0x560967251780 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 -- 172.21.15.116:0/851747101 >> 172.21.15.116:0/851747101 conn(0x56096724af80 msgr2=0x560967259cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 -- 172.21.15.116:0/851747101 shutdown_connections 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.672+0000 7f4efc5b9740 1 -- 172.21.15.116:0/851747101 wait complete. 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 -- start start 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 Processor -- start 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x56096725ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967260310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41a9720 con 0x560967250400 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41aaa40 con 0x560967255570 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41a9d80 con 0x56096724fa10 2024-03-04T07:48:13.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.673+0000 7f4efc5b9740 10 client.0 _ll_register_callbacks cb 0x7f4eec009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4eebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967260310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967260310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55792/0 (socket says 172.21.15.116:55792) 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4eebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37348/0 (socket says 172.21.15.116:37348) 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 -- 172.21.15.116:0/4132156236 learned_addr learned my addr 172.21.15.116:0/4132156236 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x56096725ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 -- 172.21.15.116:0/4132156236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 msgr2=0x56096725ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x56096725ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 -- 172.21.15.116:0/4132156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 msgr2=0x56096725f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 -- 172.21.15.116:0/4132156236 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560967260900 con 0x560967255570 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x56096725ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4eebe00640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ef3400640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967260310 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4eec018040 tx=0x7f4eec00fc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3475250f1c05590 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ee0c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ee0c00640 1 -- 172.21.15.116:0/4132156236 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec037020 con 0x560967255570 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ee4005520 con 0x560967255570 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.674+0000 7f4ee0c00640 1 -- 172.21.15.116:0/4132156236 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4eec011040 con 0x560967255570 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4ee0c00640 1 -- 172.21.15.116:0/4132156236 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4eec004900 con 0x560967255570 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 10 client.27134 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5609672c16d0 con 0x560967255570 2024-03-04T07:48:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4ee0c00640 1 -- 172.21.15.116:0/4132156236 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee4005520 con 0x560967255570 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4ee0c00640 1 -- 172.21.15.116:0/4132156236 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4eec026040 con 0x560967255570 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 10 client.27134 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 10 client.27134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 20 client.27134 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5609672c2000 0x5609672c4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 4 client.27134 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5609671bcc00 con 0x5609672c2000 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.675+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5609672c2000 0x5609672c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.676+0000 7f4ef2a00640 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5609672c2000 0x5609672c4420 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74340cdf39a3d68c server_cookie=6e59f6c5ef4a60ac in_seq=0 out_seq=0 2024-03-04T07:48:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.676+0000 7f4ee0c00640 10 client.27134 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.689+0000 7f4ee0c00640 1 -- 172.21.15.116:0/4132156236 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5609671bcc00 con 0x5609672c2000 2024-03-04T07:48:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.689+0000 7f4ee0c00640 10 client.27134 handle_command_reply: tid=0 2024-03-04T07:48:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.689+0000 7f4efc5b9740 1 client.27134 shutdown 2024-03-04T07:48:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 msgr2=0x560967260310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967260310 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4eec018040 tx=0x7f4eec00fc20 comp rx=0 tx=0).stop 2024-03-04T07:48:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 shutdown_connections 2024-03-04T07:48:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5609672c2000 0x5609672c4420 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560967255570 0x560967260310 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560967250400 0x56096725f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4ee0c00640 0 client.27134 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 --2- 172.21.15.116:0/4132156236 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56096724fa10 0x56096725ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 >> 172.21.15.116:0/4132156236 conn(0x56096724af80 msgr2=0x560967259cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 shutdown_connections 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 1 -- 172.21.15.116:0/4132156236 wait complete. 2024-03-04T07:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.690+0000 7f4efc5b9740 20 client.27134 trim_cache size 0 max 16384 2024-03-04T07:48:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:48:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:48:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:48:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4ed80a1df0 msgr2=0x7f4ed80a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4ed80a1df0 0x7f4ed80a42b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x560967250c60 tx=0x7f4eec009500 comp rx=0 tx=0).stop 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4102630 msgr2=0x7f4ef4199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4102630 0x7f4ef4199ea0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00e9f0 tx=0x7f4edc00edf0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 shutdown_connections 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef4102ff0 0x7f4ef41a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4ef4102630 0x7f4ef4199ea0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4ed80a1df0 0x7f4ed80a42b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 --2- 172.21.15.116:0/975167528 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ef4101440 0x7f4ef4199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 >> 172.21.15.116:0/975167528 conn(0x7f4ef40fc9b0 msgr2=0x7f4ef40fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 shutdown_connections 2024-03-04T07:48:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:13.691+0000 7f4ef3e00640 1 -- 172.21.15.116:0/975167528 wait complete. 2024-03-04T07:48:15.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:48:15.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:48:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:14 smithi116 ceph-mon[32150]: pgmap v625: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:48:15.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:14 smithi152 ceph-mon[37154]: pgmap v625: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:48:15.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:14 smithi133 ceph-mon[36505]: pgmap v625: 97 pgs: 97 active+clean; 1.5 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-04T07:48:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 -- 172.21.15.116:0/1058304991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0103810 msgr2=0x7f30e0103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 --2- 172.21.15.116:0/1058304991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0103810 0x7f30e0103c30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009ca0 tx=0x7f30d002c100 comp rx=0 tx=0).stop 2024-03-04T07:48:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 -- 172.21.15.116:0/1058304991 shutdown_connections 2024-03-04T07:48:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 --2- 172.21.15.116:0/1058304991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01053c0 0x7f30e010bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 --2- 172.21.15.116:0/1058304991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30e0104a00 0x7f30e0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 --2- 172.21.15.116:0/1058304991 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0103810 0x7f30e0103c30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 -- 172.21.15.116:0/1058304991 >> 172.21.15.116:0/1058304991 conn(0x7f30e00fed80 msgr2=0x7f30e01011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 -- 172.21.15.116:0/1058304991 shutdown_connections 2024-03-04T07:48:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.454+0000 7f30e5200640 1 -- 172.21.15.116:0/1058304991 wait complete. 2024-03-04T07:48:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.455+0000 7f30e5200640 1 Processor -- start 2024-03-04T07:48:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.455+0000 7f30e5200640 1 -- start start 2024-03-04T07:48:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30e5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e0103810 0x7f30e0199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30e5200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0104a00 0x7f30e019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30e5200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30e01053c0 0x7f30e01a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30e5200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e010f460 con 0x7f30e0103810 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30e5200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e010f2e0 con 0x7f30e0104a00 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30e5200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e010f5e0 con 0x7f30e01053c0 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e0103810 0x7f30e0199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e0103810 0x7f30e0199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37364/0 (socket says 172.21.15.116:37364) 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 -- 172.21.15.116:0/2515851841 learned_addr learned my addr 172.21.15.116:0/2515851841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30d5400640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0104a00 0x7f30e019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30de800640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30e01053c0 0x7f30e01a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 -- 172.21.15.116:0/2515851841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30e01053c0 msgr2=0x7f30e01a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30e01053c0 0x7f30e01a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 -- 172.21.15.116:0/2515851841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0104a00 msgr2=0x7f30e019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0104a00 0x7f30e019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.456+0000 7f30dde00640 1 -- 172.21.15.116:0/2515851841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e01a1760 con 0x7f30e0103810 2024-03-04T07:48:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.457+0000 7f30dde00640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e0103810 0x7f30e0199b40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f30d0000f80 tx=0x7f30d0035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.457+0000 7f30d7400640 1 -- 172.21.15.116:0/2515851841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d0039040 con 0x7f30e0103810 2024-03-04T07:48:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.457+0000 7f30d7400640 1 -- 172.21.15.116:0/2515851841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30d0044070 con 0x7f30e0103810 2024-03-04T07:48:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.457+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e01a19f0 con 0x7f30e0103810 2024-03-04T07:48:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.457+0000 7f30d7400640 1 -- 172.21.15.116:0/2515851841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d00044a0 con 0x7f30e0103810 2024-03-04T07:48:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.457+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e01a1eb0 con 0x7f30e0103810 2024-03-04T07:48:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.461+0000 7f30e7e3d740 1 Processor -- start 2024-03-04T07:48:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30e7e3d740 1 -- start start 2024-03-04T07:48:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30d7400640 1 -- 172.21.15.116:0/2515851841 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f30d0004640 con 0x7f30e0103810 2024-03-04T07:48:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c754e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c7553850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c7556350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c7473090 con 0x5589c754e360 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c74aad30 con 0x5589c7553ec0 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c7453a30 con 0x5589c754ed50 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30de800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c754e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30de800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c754e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37366/0 (socket says 172.21.15.116:37366) 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30de800640 1 -- 172.21.15.116:0/2212952179 learned_addr learned my addr 172.21.15.116:0/2212952179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30dde00640 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c7553850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.462+0000 7f30d5400640 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c7556350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30d5400640 1 -- 172.21.15.116:0/2212952179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 msgr2=0x5589c7553850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30d5400640 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c7553850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30d5400640 1 -- 172.21.15.116:0/2212952179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 msgr2=0x5589c754e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30d5400640 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c754e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30d5400640 1 -- 172.21.15.116:0/2212952179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589c72897d0 con 0x5589c7553ec0 2024-03-04T07:48:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30de800640 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c754e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30d5400640 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c7556350 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f30c800bbc0 tx=0x7f30c80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0579fe25028e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30cf400640 1 -- 172.21.15.116:0/2212952179 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8004030 con 0x5589c7553ec0 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30cf400640 1 -- 172.21.15.116:0/2212952179 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30c80041d0 con 0x5589c7553ec0 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30cf400640 1 -- 172.21.15.116:0/2212952179 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c80116a0 con 0x5589c7553ec0 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.463+0000 7f30d7400640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30c40a1d80 0x7f30c40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30d7400640 1 -- 172.21.15.116:0/2515851841 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f30d00f0430 con 0x7f30e0103810 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30de800640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30c40a1d80 0x7f30c40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2212952179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 msgr2=0x5589c7556350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c7556350 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f30c800bbc0 tx=0x7f30c80107a0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2212952179 shutdown_connections 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c7556350 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c7553850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2212952179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c754e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2212952179 >> 172.21.15.116:0/2212952179 conn(0x5589c75498d0 msgr2=0x5589c7549cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:15.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30de800640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30c40a1d80 0x7f30c40a4240 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x5589c754f5b0 tx=0x7f30d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2212952179 shutdown_connections 2024-03-04T07:48:15.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.464+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2212952179 wait complete. 2024-03-04T07:48:15.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 -- start start 2024-03-04T07:48:15.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 Processor -- start 2024-03-04T07:48:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754e360 0x5589c7558750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754ed50 0x5589c7558c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c74490c0 con 0x5589c754ed50 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c74a6dd0 con 0x5589c7553ec0 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.465+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c7556e70 con 0x5589c754e360 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30e7e3d740 10 client.0 _ll_register_callbacks cb 0x5589c7466860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30d5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754ed50 0x5589c7558c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55822/0 (socket says 172.21.15.116:55822) 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 -- 172.21.15.116:0/1464291597 learned_addr learned my addr 172.21.15.116:0/1464291597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30dde00640 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754e360 0x5589c7558750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 -- 172.21.15.116:0/1464291597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754e360 msgr2=0x5589c7558750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754e360 0x5589c7558750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 -- 172.21.15.116:0/1464291597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754ed50 msgr2=0x5589c7558c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754ed50 0x5589c7558c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 -- 172.21.15.116:0/1464291597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589c71bf3f0 con 0x5589c7553ec0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30dde00640 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754e360 0x5589c7558750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30de800640 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755fcd0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f30d80206d0 tx=0x7f30d8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82df51fe1b8854b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.466+0000 7f30c3e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30c3e00640 1 -- 172.21.15.116:0/1464291597 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d802d060 con 0x5589c7553ec0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30c3e00640 1 -- 172.21.15.116:0/1464291597 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f30d8018030 con 0x5589c7553ec0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30c80041d0 con 0x5589c7553ec0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30c3e00640 1 -- 172.21.15.116:0/1464291597 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30d8011890 con 0x5589c7553ec0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 10 client.27146 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5589c717f0f0 con 0x5589c7553ec0 2024-03-04T07:48:15.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30c3e00640 1 -- 172.21.15.116:0/1464291597 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30c80041d0 con 0x5589c7553ec0 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30c3e00640 1 -- 172.21.15.116:0/1464291597 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f30d8032070 con 0x5589c7553ec0 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 10 client.27146 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 10 client.27146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 20 client.27146 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c7564270 0x5589c7566690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 4 client.27146 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.467+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589c74b8270 con 0x5589c7564270 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.468+0000 7f30dde00640 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c7564270 0x5589c7566690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.468+0000 7f30dde00640 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c7564270 0x5589c7566690 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9011e2f84c5324a8 server_cookie=e8c638a7384d962d in_seq=0 out_seq=0 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.468+0000 7f30c3e00640 10 client.27146 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30c3e00640 1 -- 172.21.15.116:0/1464291597 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5589c74b8270 con 0x5589c7564270 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30c3e00640 10 client.27146 handle_command_reply: tid=0 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 client.27146 shutdown 2024-03-04T07:48:15.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 msgr2=0x5589c755fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755fcd0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f30d80206d0 tx=0x7f30d8004730 comp rx=0 tx=0).stop 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 shutdown_connections 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c7564270 0x5589c7566690 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755fcd0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754ed50 0x5589c7558c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/1464291597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754e360 0x5589c7558750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 >> 172.21.15.116:0/1464291597 conn(0x5589c75498d0 msgr2=0x5589c7540360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30c3e00640 0 client.27146 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 shutdown_connections 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 1 -- 172.21.15.116:0/1464291597 wait complete. 2024-03-04T07:48:15.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.469+0000 7f30e7e3d740 20 client.27146 trim_cache size 0 max 16384 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.486+0000 7f30e7e3d740 1 Processor -- start 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30e7e3d740 1 -- start start 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c756af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c756b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01a48e0 con 0x5589c754e360 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01a6ec0 con 0x5589c7553ec0 2024-03-04T07:48:15.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01a4ee0 con 0x5589c754ed50 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30dde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c756af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30de800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c756b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30de800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c756b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55836/0 (socket says 172.21.15.116:55836) 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.487+0000 7f30de800640 1 -- 172.21.15.116:0/2935129497 learned_addr learned my addr 172.21.15.116:0/2935129497 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30d5400640 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30de800640 1 -- 172.21.15.116:0/2935129497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 msgr2=0x5589c755fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30de800640 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30de800640 1 -- 172.21.15.116:0/2935129497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 msgr2=0x5589c756af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30de800640 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c756af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30de800640 1 -- 172.21.15.116:0/2935129497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589c7558540 con 0x5589c7553ec0 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30de800640 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c756b490 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f30d8020f60 tx=0x7f30d8011c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5532ac230ebd091b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30cea00640 1 -- 172.21.15.116:0/2935129497 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8038030 con 0x5589c7553ec0 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30cea00640 1 -- 172.21.15.116:0/2935129497 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30d8011e30 con 0x5589c7553ec0 2024-03-04T07:48:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.488+0000 7f30cea00640 1 -- 172.21.15.116:0/2935129497 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80035f0 con 0x5589c7553ec0 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2935129497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 msgr2=0x5589c756b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c756b490 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f30d8020f60 tx=0x7f30d8011c40 comp rx=0 tx=0).stop 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2935129497 shutdown_connections 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c756b490 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/2935129497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c756af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2935129497 >> 172.21.15.116:0/2935129497 conn(0x5589c75498d0 msgr2=0x5589c754d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2935129497 shutdown_connections 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 -- 172.21.15.116:0/2935129497 wait complete. 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.489+0000 7f30e7e3d740 1 -- start start 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 1 Processor -- start 2024-03-04T07:48:15.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c755f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01a5540 con 0x5589c754e360 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01a6860 con 0x5589c7553ec0 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01a5ba0 con 0x5589c754ed50 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30e7e3d740 10 client.0 _ll_register_callbacks cb 0x7f30d8020590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30dde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c755f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30dde00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c755f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37390/0 (socket says 172.21.15.116:37390) 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30dde00640 1 -- 172.21.15.116:0/3717721290 learned_addr learned my addr 172.21.15.116:0/3717721290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30de800640 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.490+0000 7f30d5400640 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30dde00640 1 -- 172.21.15.116:0/3717721290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 msgr2=0x5589c755df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30dde00640 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30dde00640 1 -- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 msgr2=0x5589c755e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30dde00640 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30dde00640 1 -- 172.21.15.116:0/3717721290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589c755ea00 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30dde00640 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c755f8f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f30d003d040 tx=0x7f30d0009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4cad517ede4292 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30c3e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30c3e00640 1 -- 172.21.15.116:0/3717721290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d00bcca0 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30c3e00640 1 -- 172.21.15.116:0/3717721290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f30d0032890 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30d8011e30 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30c3e00640 1 -- 172.21.15.116:0/3717721290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30d00bcac0 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30e7e3d740 10 client.17025 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.491+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5589c75c09b0 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30c3e00640 1 -- 172.21.15.116:0/3717721290 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30d8011e30 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30c3e00640 1 -- 172.21.15.116:0/3717721290 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f30d003c030 con 0x5589c754e360 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30e7e3d740 10 client.17025 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30e7e3d740 10 client.17025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30e7e3d740 20 client.17025 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c75c1290 0x5589c75c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30e7e3d740 4 client.17025 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5589c7553c30 con 0x5589c75c1290 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.492+0000 7f30d5400640 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c75c1290 0x5589c75c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30d5400640 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c75c1290 0x5589c75c36b0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c5f03a3d288eb server_cookie=614c2072149d6300 in_seq=0 out_seq=0 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30c3e00640 10 client.17025 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30c3e00640 1 -- 172.21.15.116:0/3717721290 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5589c7553c30 con 0x5589c75c1290 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30c3e00640 10 client.17025 handle_command_reply: tid=0 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 client.17025 shutdown 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 msgr2=0x5589c755f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c755f8f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f30d003d040 tx=0x7f30d0009a20 comp rx=0 tx=0).stop 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 shutdown_connections 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589c75c1290 0x5589c75c36b0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589c7553ec0 0x5589c755e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589c754ed50 0x5589c755df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30c3e00640 0 client.17025 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 --2- 172.21.15.116:0/3717721290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589c754e360 0x5589c755f8f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 >> 172.21.15.116:0/3717721290 conn(0x5589c75498d0 msgr2=0x5589c753fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 shutdown_connections 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.493+0000 7f30e7e3d740 1 -- 172.21.15.116:0/3717721290 wait complete. 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e7e3d740 20 client.17025 trim_cache size 0 max 16384 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:48:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:48:15.499 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30c40a1d80 msgr2=0x7f30c40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30c40a1d80 0x7f30c40a4240 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x5589c754f5b0 tx=0x7f30d8005f90 comp rx=0 tx=0).stop 2024-03-04T07:48:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e0103810 msgr2=0x7f30e0199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e0103810 0x7f30e0199b40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f30d0000f80 tx=0x7f30d0035d10 comp rx=0 tx=0).stop 2024-03-04T07:48:15.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 shutdown_connections 2024-03-04T07:48:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f30c40a1d80 0x7f30c40a4240 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30e01053c0 0x7f30e01a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f30e0104a00 0x7f30e019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 --2- 172.21.15.116:0/2515851841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e0103810 0x7f30e0199b40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 >> 172.21.15.116:0/2515851841 conn(0x7f30e00fed80 msgr2=0x7f30e0107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 shutdown_connections 2024-03-04T07:48:15.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.494+0000 7f30e5200640 1 -- 172.21.15.116:0/2515851841 wait complete. 2024-03-04T07:48:15.512 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:48:15.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:48:15.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 -- 172.21.15.116:0/2696702054 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 msgr2=0x7f153c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 --2- 172.21.15.116:0/2696702054 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c104e80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1528009900 tx=0x7f152802bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:15.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 -- 172.21.15.116:0/2696702054 shutdown_connections 2024-03-04T07:48:15.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 --2- 172.21.15.116:0/2696702054 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f153c1053c0 0x7f153c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 --2- 172.21.15.116:0/2696702054 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c104e80 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 --2- 172.21.15.116:0/2696702054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 0x7f153c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 -- 172.21.15.116:0/2696702054 >> 172.21.15.116:0/2696702054 conn(0x7f153c0fed80 msgr2=0x7f153c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:15.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 -- 172.21.15.116:0/2696702054 shutdown_connections 2024-03-04T07:48:15.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.770+0000 7f1541000640 1 -- 172.21.15.116:0/2696702054 wait complete. 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 Processor -- start 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 -- start start 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 0x7f153c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f153c1053c0 0x7f153c1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c10f4e0 con 0x7f153c103810 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c10f360 con 0x7f153c104a00 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.771+0000 7f1541000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c10f660 con 0x7f153c1053c0 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 0x7f153c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55862/0 (socket says 172.21.15.116:55862) 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 0x7f153c199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37396/0 (socket says 172.21.15.116:37396) 2024-03-04T07:48:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 -- 172.21.15.116:0/3909761722 learned_addr learned my addr 172.21.15.116:0/3909761722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f153a800640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f153c1053c0 0x7f153c1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 -- 172.21.15.116:0/3909761722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f153c1053c0 msgr2=0x7f153c1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f153c1053c0 0x7f153c1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 -- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 msgr2=0x7f153c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 0x7f153c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539400640 1 -- 172.21.15.116:0/3909761722 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c1a16f0 con 0x7f153c104a00 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.772+0000 7f1539e00640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 0x7f153c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.773+0000 7f1539400640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c19a010 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f1528005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.773+0000 7f1532a00640 1 -- 172.21.15.116:0/3909761722 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528043020 con 0x7f153c104a00 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.773+0000 7f1532a00640 1 -- 172.21.15.116:0/3909761722 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15280054e0 con 0x7f153c104a00 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.773+0000 7f1532a00640 1 -- 172.21.15.116:0/3909761722 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528003e60 con 0x7f153c104a00 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.773+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1a1980 con 0x7f153c104a00 2024-03-04T07:48:15.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.773+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c1a1ed0 con 0x7f153c104a00 2024-03-04T07:48:15.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.774+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c10aad0 con 0x7f153c104a00 2024-03-04T07:48:15.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.776+0000 7f1532a00640 1 -- 172.21.15.116:0/3909761722 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1528046020 con 0x7f153c104a00 2024-03-04T07:48:15.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.777+0000 7f1532a00640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f15200a1e00 0x7f15200a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:15.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.777+0000 7f1532a00640 1 -- 172.21.15.116:0/3909761722 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f15280f0050 con 0x7f153c104a00 2024-03-04T07:48:15.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.778+0000 7f1539e00640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f15200a1e00 0x7f15200a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:15.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.778+0000 7f1539e00640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f15200a1e00 0x7f15200a42c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f152c005fd0 tx=0x7f152c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:15.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:15.779+0000 7f1532a00640 1 -- 172.21.15.116:0/3909761722 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f152804d020 con 0x7f153c104a00 2024-03-04T07:48:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.016+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f153c02d1b0 con 0x7f153c104a00 2024-03-04T07:48:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.017+0000 7f1532a00640 1 -- 172.21.15.116:0/3909761722 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f15280bd480 con 0x7f153c104a00 2024-03-04T07:48:16.019 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:48:16.019 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:48:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:48:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f15200a1e00 msgr2=0x7f15200a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f15200a1e00 0x7f15200a42c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f152c005fd0 tx=0x7f152c00b410 comp rx=0 tx=0).stop 2024-03-04T07:48:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 msgr2=0x7f153c19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c19a010 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f1528005dd0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 shutdown_connections 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f15200a1e00 0x7f15200a42c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f153c1053c0 0x7f153c1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f153c104a00 0x7f153c19a010 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 --2- 172.21.15.116:0/3909761722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f153c103810 0x7f153c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 >> 172.21.15.116:0/3909761722 conn(0x7f153c0fed80 msgr2=0x7f153c107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.020+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 shutdown_connections 2024-03-04T07:48:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.021+0000 7f1541000640 1 -- 172.21.15.116:0/3909761722 wait complete. 2024-03-04T07:48:16.039 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:48:16.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:48:16.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 -- 172.21.15.116:0/1790091300 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd01027d0 msgr2=0x7f3bd0102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 --2- 172.21.15.116:0/1790091300 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd01027d0 0x7f3bd0102c50 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009860 tx=0x7f3bbc02bb70 comp rx=0 tx=0).stop 2024-03-04T07:48:16.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 -- 172.21.15.116:0/1790091300 shutdown_connections 2024-03-04T07:48:16.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 --2- 172.21.15.116:0/1790091300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd0103190 0x7f3bd0109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 --2- 172.21.15.116:0/1790091300 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd01027d0 0x7f3bd0102c50 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 --2- 172.21.15.116:0/1790091300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bd01015e0 0x7f3bd0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 -- 172.21.15.116:0/1790091300 >> 172.21.15.116:0/1790091300 conn(0x7f3bd00fcb70 msgr2=0x7f3bd00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:16.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 -- 172.21.15.116:0/1790091300 shutdown_connections 2024-03-04T07:48:16.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.248+0000 7f3bd8600640 1 -- 172.21.15.116:0/1790091300 wait complete. 2024-03-04T07:48:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.249+0000 7f3bd8600640 1 Processor -- start 2024-03-04T07:48:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.249+0000 7f3bd8600640 1 -- start start 2024-03-04T07:48:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd8600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd01015e0 0x7f3bd0199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd8600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bd01027d0 0x7f3bd019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd8600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd0103190 0x7f3bd01a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd8600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd010d260 con 0x7f3bd01015e0 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd8600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd010d0e0 con 0x7f3bd0103190 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd8600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd010d3e0 con 0x7f3bd01027d0 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd01015e0 0x7f3bd0199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd6600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd0103190 0x7f3bd01a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd01015e0 0x7f3bd0199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49642/0 (socket says 172.21.15.116:49642) 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd5c00640 1 -- 172.21.15.116:0/2016428449 learned_addr learned my addr 172.21.15.116:0/2016428449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd5200640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bd01027d0 0x7f3bd019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd6600640 1 -- 172.21.15.116:0/2016428449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bd01027d0 msgr2=0x7f3bd019a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd6600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bd01027d0 0x7f3bd019a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd6600640 1 -- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd01015e0 msgr2=0x7f3bd0199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd6600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd01015e0 0x7f3bd0199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd6600640 1 -- 172.21.15.116:0/2016428449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd01a1730 con 0x7f3bd0103190 2024-03-04T07:48:16.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.250+0000 7f3bd5200640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bd01027d0 0x7f3bd019a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:16.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.251+0000 7f3bd6600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd0103190 0x7f3bd01a10c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800ef00 tx=0x7f3bc800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:16.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.251+0000 7f3bce000640 1 -- 172.21.15.116:0/2016428449 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8004b20 con 0x7f3bd0103190 2024-03-04T07:48:16.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.251+0000 7f3bce000640 1 -- 172.21.15.116:0/2016428449 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc8003e50 con 0x7f3bd0103190 2024-03-04T07:48:16.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.251+0000 7f3bce000640 1 -- 172.21.15.116:0/2016428449 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc800b570 con 0x7f3bd0103190 2024-03-04T07:48:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.251+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd01a1a20 con 0x7f3bd0103190 2024-03-04T07:48:16.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.251+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd01a1ef0 con 0x7f3bd0103190 2024-03-04T07:48:16.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.254+0000 7f3bce000640 1 -- 172.21.15.116:0/2016428449 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3bc801d020 con 0x7f3bd0103190 2024-03-04T07:48:16.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.254+0000 7f3bdb246740 1 Processor -- start 2024-03-04T07:48:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.254+0000 7f3bdb246740 1 -- start start 2024-03-04T07:48:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f410c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f46190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f48c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bdb246740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558505ea5d60 con 0x558505f46800 2024-03-04T07:48:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bdb246740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558505e4ecf0 con 0x558505f41690 2024-03-04T07:48:16.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bdb246740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558505ea1e00 con 0x558505f40ca0 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f46190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f46190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38010/0 (socket says 172.21.15.116:38010) 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f48c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 -- 172.21.15.116:0/3601613368 learned_addr learned my addr 172.21.15.116:0/3601613368 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5200640 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 -- 172.21.15.116:0/3601613368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 msgr2=0x558505f410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 -- 172.21.15.116:0/3601613368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 msgr2=0x558505f48c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f48c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5c00640 1 -- 172.21.15.116:0/3601613368 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558505bba3f0 con 0x558505f41690 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.255+0000 7f3bd5200640 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f48c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.256+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f46190 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4009d70 tx=0x7f3bc4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac33fa786047034 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.256+0000 7f3bce000640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3bb80a1d80 0x7f3bb80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.256+0000 7f3bc2a00640 1 -- 172.21.15.116:0/3601613368 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4004220 con 0x558505f41690 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.256+0000 7f3bc2a00640 1 -- 172.21.15.116:0/3601613368 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc40043c0 con 0x558505f41690 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.256+0000 7f3bce000640 1 -- 172.21.15.116:0/2016428449 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3bc80cdc00 con 0x7f3bd0103190 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.256+0000 7f3bc2a00640 1 -- 172.21.15.116:0/3601613368 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4019950 con 0x558505f41690 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.256+0000 7f3bd5200640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3bb80a1d80 0x7f3bb80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 -- 172.21.15.116:0/3601613368 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 msgr2=0x558505f46190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f46190 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4009d70 tx=0x7f3bc4011810 comp rx=0 tx=0).stop 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bd5200640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3bb80a1d80 0x7f3bb80a4240 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009830 tx=0x7f3bbc035f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 -- 172.21.15.116:0/3601613368 shutdown_connections 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f48c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f46190 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3601613368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 -- 172.21.15.116:0/3601613368 >> 172.21.15.116:0/3601613368 conn(0x558505f3c210 msgr2=0x558505f3c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 -- 172.21.15.116:0/3601613368 shutdown_connections 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 -- 172.21.15.116:0/3601613368 wait complete. 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.257+0000 7f3bdb246740 1 -- start start 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 1 Processor -- start 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f40ca0 0x558505f4b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f41690 0x558505f4b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558505f497b0 con 0x558505f46800 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558505e44220 con 0x558505f40ca0 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558505f49930 con 0x558505f41690 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bdb246740 10 client.0 _ll_register_callbacks cb 0x558505bb9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:16.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f40ca0 0x558505f4b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f40ca0 0x558505f4b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38020/0 (socket says 172.21.15.116:38020) 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bd6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bd5c00640 1 -- 172.21.15.116:0/2850049003 learned_addr learned my addr 172.21.15.116:0/2850049003 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.258+0000 7f3bd5200640 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f41690 0x558505f4b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bd5c00640 1 -- 172.21.15.116:0/2850049003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f41690 msgr2=0x558505f4b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f41690 0x558505f4b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bd5c00640 1 -- 172.21.15.116:0/2850049003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 msgr2=0x558505f526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bd5c00640 1 -- 172.21.15.116:0/2850049003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558505c847d0 con 0x558505f40ca0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f40ca0 0x558505f4b160 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3bc401b040 tx=0x7f3bc4003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45af1a7bbf2c2846 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bb7400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bb7400640 1 -- 172.21.15.116:0/2850049003 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4028070 con 0x558505f40ca0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bb7400640 1 -- 172.21.15.116:0/2850049003 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3bc4005040 con 0x558505f40ca0 2024-03-04T07:48:16.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bc40043c0 con 0x558505f40ca0 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bb7400640 1 -- 172.21.15.116:0/2850049003 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bc4034050 con 0x558505f40ca0 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.259+0000 7f3bdb246740 10 client.27170 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558505f55a50 con 0x558505f40ca0 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bb7400640 1 -- 172.21.15.116:0/2850049003 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc40043c0 con 0x558505f40ca0 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bb7400640 1 -- 172.21.15.116:0/2850049003 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3bc4004c50 con 0x558505f40ca0 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bdb246740 10 client.27170 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bdb246740 10 client.27170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:16.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bdb246740 20 client.27170 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bdb246740 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505f56e70 0x558505f592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bdb246740 4 client.27170 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558505eb32a0 con 0x558505f56e70 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.260+0000 7f3bd5200640 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505f56e70 0x558505f592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.261+0000 7f3bd5200640 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505f56e70 0x558505f592d0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dd7e64bfb42cc6a server_cookie=4c21ff4f677ad02c in_seq=0 out_seq=0 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.261+0000 7f3bb7400640 10 client.27170 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bb7400640 1 -- 172.21.15.116:0/2850049003 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558505eb32a0 con 0x558505f56e70 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bb7400640 10 client.27170 handle_command_reply: tid=0 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 client.27170 shutdown 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f40ca0 msgr2=0x558505f4b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f40ca0 0x558505f4b160 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3bc401b040 tx=0x7f3bc4003820 comp rx=0 tx=0).stop 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 shutdown_connections 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505f56e70 0x558505f592d0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f41690 0x558505f4b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bb7400640 0 client.27170 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:16.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 --2- 172.21.15.116:0/2850049003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f40ca0 0x558505f4b160 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 >> 172.21.15.116:0/2850049003 conn(0x558505f3c210 msgr2=0x558505f3b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:16.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 shutdown_connections 2024-03-04T07:48:16.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 1 -- 172.21.15.116:0/2850049003 wait complete. 2024-03-04T07:48:16.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.262+0000 7f3bdb246740 20 client.27170 trim_cache size 0 max 16384 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 Processor -- start 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 -- start start 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f5db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f41690 0x558505f526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f46800 0x558505f5e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01a8be0 con 0x558505f41690 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01ab1c0 con 0x558505f46800 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bdb246740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01a91e0 con 0x558505f40ca0 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bd5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f41690 0x558505f526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f5db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bd6600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f46800 0x558505f5e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f5db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:45628/0 (socket says 172.21.15.116:45628) 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.276+0000 7f3bd5c00640 1 -- 172.21.15.116:0/1333616905 learned_addr learned my addr 172.21.15.116:0/1333616905 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bd6600640 1 -- 172.21.15.116:0/1333616905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 msgr2=0x558505f5db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bd6600640 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f5db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bd6600640 1 -- 172.21.15.116:0/1333616905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f41690 msgr2=0x558505f526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bd6600640 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f41690 0x558505f526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bd6600640 1 -- 172.21.15.116:0/1333616905 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558505f49690 con 0x558505f46800 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f5db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bd6600640 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f46800 0x558505f5e0d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800cc40 tx=0x7f3bc8096bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3617929d32d7b4d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bc2000640 1 -- 172.21.15.116:0/1333616905 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc80b3070 con 0x558505f46800 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bc2000640 1 -- 172.21.15.116:0/1333616905 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc80a30d0 con 0x558505f46800 2024-03-04T07:48:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.277+0000 7f3bc2000640 1 -- 172.21.15.116:0/1333616905 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc801a040 con 0x558505f46800 2024-03-04T07:48:16.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 -- 172.21.15.116:0/1333616905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f46800 msgr2=0x558505f5e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f46800 0x558505f5e0d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800cc40 tx=0x7f3bc8096bd0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 -- 172.21.15.116:0/1333616905 shutdown_connections 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f46800 0x558505f5e0d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f41690 0x558505f526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 --2- 172.21.15.116:0/1333616905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f5db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 -- 172.21.15.116:0/1333616905 >> 172.21.15.116:0/1333616905 conn(0x558505f3c210 msgr2=0x558505f3fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 -- 172.21.15.116:0/1333616905 shutdown_connections 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 -- 172.21.15.116:0/1333616905 wait complete. 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 -- start start 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.278+0000 7f3bdb246740 1 Processor -- start 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f521c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f50820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bdb246740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f50d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bdb246740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01a9840 con 0x558505f46800 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bdb246740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01aab60 con 0x558505f41690 2024-03-04T07:48:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bdb246740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01a9ea0 con 0x558505f40ca0 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bdb246740 10 client.0 _ll_register_callbacks cb 0x7f3bc4003e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f50d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f50820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f50820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38040/0 (socket says 172.21.15.116:38040) 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 -- 172.21.15.116:0/3026322881 learned_addr learned my addr 172.21.15.116:0/3026322881 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd6600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f50d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49668/0 (socket says 172.21.15.116:49668) 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 -- 172.21.15.116:0/3026322881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 msgr2=0x558505f521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 -- 172.21.15.116:0/3026322881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 msgr2=0x558505f50d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f50d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd5200640 1 -- 172.21.15.116:0/3026322881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558505f512a0 con 0x558505f41690 2024-03-04T07:48:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.279+0000 7f3bd6600640 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f50d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bd5200640 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f50820 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009860 tx=0x7f3bbc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf7d725c8bcb777f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bb7400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bb7400640 1 -- 172.21.15.116:0/3026322881 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc039040 con 0x558505f41690 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bb7400640 1 -- 172.21.15.116:0/3026322881 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3bbc02c960 con 0x558505f41690 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3bc80a30d0 con 0x558505f41690 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bb7400640 1 -- 172.21.15.116:0/3026322881 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3bbc02cb60 con 0x558505f41690 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bdb246740 10 client.27176 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.280+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558505fb3420 con 0x558505f41690 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bb7400640 1 -- 172.21.15.116:0/3026322881 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc80a30d0 con 0x558505f41690 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bb7400640 1 -- 172.21.15.116:0/3026322881 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3bbc04d040 con 0x558505f41690 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bdb246740 10 client.27176 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bdb246740 10 client.27176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bdb246740 20 client.27176 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505fb3d60 0x558505fb6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bdb246740 4 client.27176 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558505ead5f0 con 0x558505fb3d60 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505fb3d60 0x558505fb6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bd5c00640 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505fb3d60 0x558505fb6180 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60278349039bf8fa server_cookie=ed256cf5b396e4da in_seq=0 out_seq=0 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.281+0000 7f3bb7400640 10 client.27176 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bb7400640 1 -- 172.21.15.116:0/3026322881 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558505ead5f0 con 0x558505fb3d60 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bb7400640 10 client.27176 handle_command_reply: tid=0 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 client.27176 shutdown 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 msgr2=0x558505f50820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f50820 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009860 tx=0x7f3bbc005f90 comp rx=0 tx=0).stop 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 shutdown_connections 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558505fb3d60 0x558505fb6180 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558505f46800 0x558505f50d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558505f41690 0x558505f50820 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bb7400640 0 client.27176 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 --2- 172.21.15.116:0/3026322881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558505f40ca0 0x558505f521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 >> 172.21.15.116:0/3026322881 conn(0x558505f3c210 msgr2=0x558505f3ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 shutdown_connections 2024-03-04T07:48:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 1 -- 172.21.15.116:0/3026322881 wait complete. 2024-03-04T07:48:16.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.282+0000 7f3bdb246740 20 client.27176 trim_cache size 0 max 16384 2024-03-04T07:48:16.287 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:48:16.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3bb80a1d80 msgr2=0x7f3bb80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3bb80a1d80 0x7f3bb80a4240 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009830 tx=0x7f3bbc035f70 comp rx=0 tx=0).stop 2024-03-04T07:48:16.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd0103190 msgr2=0x7f3bd01a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:16.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd0103190 0x7f3bd01a10c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800ef00 tx=0x7f3bc800c740 comp rx=0 tx=0).stop 2024-03-04T07:48:16.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 shutdown_connections 2024-03-04T07:48:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3bd0103190 0x7f3bd01a10c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3bb80a1d80 0x7f3bb80a4240 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3bd01027d0 0x7f3bd019a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 --2- 172.21.15.116:0/2016428449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd01015e0 0x7f3bd0199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 >> 172.21.15.116:0/2016428449 conn(0x7f3bd00fcb70 msgr2=0x7f3bd0073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 shutdown_connections 2024-03-04T07:48:16.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:16.283+0000 7f3bd8600640 1 -- 172.21.15.116:0/2016428449 wait complete. 2024-03-04T07:48:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:16 smithi116 ceph-mon[32150]: pgmap v626: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:48:17.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:16 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3909761722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:16 smithi152 ceph-mon[37154]: pgmap v626: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:48:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:16 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3909761722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:17.300 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:48:17.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:48:17.301 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:48:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:16 smithi133 ceph-mon[36505]: pgmap v626: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:48:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:16 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3909761722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 -- 172.21.15.116:0/2526475376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441027d0 msgr2=0x7f9f44102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2526475376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441027d0 0x7f9f44102c50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9f30009860 tx=0x7f9f3002bb70 comp rx=0 tx=0).stop 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 -- 172.21.15.116:0/2526475376 shutdown_connections 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2526475376 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 0x7f9f44109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2526475376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441027d0 0x7f9f44102c50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2526475376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f441015e0 0x7f9f44101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 -- 172.21.15.116:0/2526475376 >> 172.21.15.116:0/2526475376 conn(0x7f9f440fcb70 msgr2=0x7f9f440fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.528+0000 7f9f49a00640 1 -- 172.21.15.116:0/2526475376 shutdown_connections 2024-03-04T07:48:17.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.529+0000 7f9f49a00640 1 -- 172.21.15.116:0/2526475376 wait complete. 2024-03-04T07:48:17.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 Processor -- start 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 -- start start 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441015e0 0x7f9f44199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f441027d0 0x7f9f4419a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 0x7f9f441a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4410d260 con 0x7f9f441027d0 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4410d0e0 con 0x7f9f44103190 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.530+0000 7f9f49a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4410d3e0 con 0x7f9f441015e0 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f42000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f441027d0 0x7f9f4419a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f43400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 0x7f9f441a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f43400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 0x7f9f441a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38056/0 (socket says 172.21.15.116:38056) 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f43400640 1 -- 172.21.15.116:0/2600732543 learned_addr learned my addr 172.21.15.116:0/2600732543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f42a00640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441015e0 0x7f9f44199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f42000640 1 -- 172.21.15.116:0/2600732543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441015e0 msgr2=0x7f9f44199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f42000640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441015e0 0x7f9f44199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f42000640 1 -- 172.21.15.116:0/2600732543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 msgr2=0x7f9f441a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f42000640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 0x7f9f441a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f42000640 1 -- 172.21.15.116:0/2600732543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f441a1760 con 0x7f9f441027d0 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.531+0000 7f9f43400640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 0x7f9f441a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.532+0000 7f9f42000640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f441027d0 0x7f9f4419a080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9f3002c0b0 tx=0x7f9f30032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.532+0000 7f9f3b400640 1 -- 172.21.15.116:0/2600732543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30041020 con 0x7f9f441027d0 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.532+0000 7f9f3b400640 1 -- 172.21.15.116:0/2600732543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f3002c9f0 con 0x7f9f441027d0 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.532+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f441a19f0 con 0x7f9f441027d0 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.532+0000 7f9f3b400640 1 -- 172.21.15.116:0/2600732543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30039040 con 0x7f9f441027d0 2024-03-04T07:48:17.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.532+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f441a1ee0 con 0x7f9f441027d0 2024-03-04T07:48:17.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.535+0000 7f9f3b400640 1 -- 172.21.15.116:0/2600732543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f9f30044020 con 0x7f9f441027d0 2024-03-04T07:48:17.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.535+0000 7f9f4c770740 1 Processor -- start 2024-03-04T07:48:17.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.535+0000 7f9f4c770740 1 -- start start 2024-03-04T07:48:17.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bed4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bf50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f4c770740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c75b00e20 con 0x561c75bed0d0 2024-03-04T07:48:17.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f4c770740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c75b52a20 con 0x561c75bedac0 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f4c770740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c75afbb00 con 0x561c75bf2c30 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f42a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f42a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38070/0 (socket says 172.21.15.116:38070) 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f42a00640 1 -- 172.21.15.116:0/2955746974 learned_addr learned my addr 172.21.15.116:0/2955746974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f43400640 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bed4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.536+0000 7f9f42000640 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bf50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f42a00640 1 -- 172.21.15.116:0/2955746974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 msgr2=0x561c75bf50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f42a00640 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bf50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f42a00640 1 -- 172.21.15.116:0/2955746974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 msgr2=0x561c75bed4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f42a00640 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bed4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f42a00640 1 -- 172.21.15.116:0/2955746974 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c75b10e10 con 0x561c75bedac0 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f42a00640 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf25c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009ca0 tx=0x7f9f34011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9301a44455bc283 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f3b400640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9f280a1d80 0x7f9f280a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f2fe00640 1 -- 172.21.15.116:0/2955746974 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34004220 con 0x561c75bedac0 2024-03-04T07:48:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f2fe00640 1 -- 172.21.15.116:0/2955746974 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f340043c0 con 0x561c75bedac0 2024-03-04T07:48:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f2fe00640 1 -- 172.21.15.116:0/2955746974 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34019930 con 0x561c75bedac0 2024-03-04T07:48:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f3b400640 1 -- 172.21.15.116:0/2600732543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9f300ef5f0 con 0x7f9f441027d0 2024-03-04T07:48:17.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.537+0000 7f9f43400640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9f280a1d80 0x7f9f280a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 -- 172.21.15.116:0/2955746974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 msgr2=0x561c75bf25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf25c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009ca0 tx=0x7f9f34011810 comp rx=0 tx=0).stop 2024-03-04T07:48:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f43400640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9f280a1d80 0x7f9f280a4240 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x561c75bee320 tx=0x7f9f3c0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 -- 172.21.15.116:0/2955746974 shutdown_connections 2024-03-04T07:48:17.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bf50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf25c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 --2- 172.21.15.116:0/2955746974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bed4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 -- 172.21.15.116:0/2955746974 >> 172.21.15.116:0/2955746974 conn(0x561c75be8640 msgr2=0x561c75be8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 -- 172.21.15.116:0/2955746974 shutdown_connections 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.538+0000 7f9f4c770740 1 -- 172.21.15.116:0/2955746974 wait complete. 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 -- start start 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 Processor -- start 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bf7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bfea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c75b0ef40 con 0x561c75bed0d0 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c75b4eac0 con 0x561c75bedac0 2024-03-04T07:48:17.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c75bf5be0 con 0x561c75bf2c30 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.539+0000 7f9f4c770740 10 client.0 _ll_register_callbacks cb 0x561c75b4daa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bf7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38086/0 (socket says 172.21.15.116:38086) 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f43400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bfea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 -- 172.21.15.116:0/3942458301 learned_addr learned my addr 172.21.15.116:0/3942458301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 -- 172.21.15.116:0/3942458301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 msgr2=0x561c75bfea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bfea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 -- 172.21.15.116:0/3942458301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 msgr2=0x561c75bf7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bf7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 -- 172.21.15.116:0/3942458301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c7592f7e0 con 0x561c75bedac0 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f42000640 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf79d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9f300bc270 tx=0x7f9f300097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db0406a45e938a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:17.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.540+0000 7f9f2cc00640 1 -- 172.21.15.116:0/3942458301 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f300bbe20 con 0x561c75bedac0 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f2cc00640 1 -- 172.21.15.116:0/3942458301 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9f300b8950 con 0x561c75bedac0 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f340043c0 con 0x561c75bedac0 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f2cc00640 1 -- 172.21.15.116:0/3942458301 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f300bbc40 con 0x561c75bedac0 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 10 client.27188 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c7598ff70 con 0x561c75bedac0 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f2cc00640 1 -- 172.21.15.116:0/3942458301 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f340043c0 con 0x561c75bedac0 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f2cc00640 1 -- 172.21.15.116:0/3942458301 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9f3003c030 con 0x561c75bedac0 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 10 client.27188 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 10 client.27188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 20 client.27188 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c031d0 0x561c75c05670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 4 client.27188 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.541+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c75b5ff60 con 0x561c75c031d0 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.542+0000 7f9f42a00640 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c031d0 0x561c75c05670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.542+0000 7f9f42a00640 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c031d0 0x561c75c05670 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9b9fee4bd3556d6 server_cookie=a48f623970b8871c in_seq=0 out_seq=0 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.542+0000 7f9f2cc00640 10 client.27188 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f2cc00640 1 -- 172.21.15.116:0/3942458301 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c75b5ff60 con 0x561c75c031d0 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f2cc00640 10 client.27188 handle_command_reply: tid=0 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 client.27188 shutdown 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 msgr2=0x561c75bf79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf79d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9f300bc270 tx=0x7f9f300097b0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 shutdown_connections 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c031d0 0x561c75c05670 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bf2c30 0x561c75bfea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bedac0 0x561c75bf79d0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3942458301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bf7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 >> 172.21.15.116:0/3942458301 conn(0x561c75be8640 msgr2=0x561c75be9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.543+0000 7f9f2cc00640 0 client.27188 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.544+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 shutdown_connections 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.544+0000 7f9f4c770740 1 -- 172.21.15.116:0/3942458301 wait complete. 2024-03-04T07:48:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.544+0000 7f9f4c770740 20 client.27188 trim_cache size 0 max 16384 2024-03-04T07:48:17.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.561+0000 7f9f4c770740 1 Processor -- start 2024-03-04T07:48:17.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.561+0000 7f9f4c770740 1 -- start start 2024-03-04T07:48:17.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.561+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bed0d0 0x561c75be71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.561+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.561+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bf2c30 0x561c75be7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f4c770740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f441a8d00 con 0x561c75bf2c30 2024-03-04T07:48:17.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f4c770740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f441ab2e0 con 0x561c75bed0d0 2024-03-04T07:48:17.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f4c770740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f441a9300 con 0x561c75bedac0 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f43400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bf2c30 0x561c75be7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bed0d0 0x561c75be71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bed0d0 0x561c75be71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38100/0 (socket says 172.21.15.116:38100) 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 -- 172.21.15.116:0/3813941037 learned_addr learned my addr 172.21.15.116:0/3813941037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42000640 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 -- 172.21.15.116:0/3813941037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 msgr2=0x561c75bfe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 -- 172.21.15.116:0/3813941037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bf2c30 msgr2=0x561c75be7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bf2c30 0x561c75be7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42a00640 1 -- 172.21.15.116:0/3813941037 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c75827170 con 0x561c75bed0d0 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.562+0000 7f9f42000640 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f42a00640 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bed0d0 0x561c75be71d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009230 tx=0x7f9f34011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fde93887bd9eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f2f400640 1 -- 172.21.15.116:0/3813941037 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34005040 con 0x561c75bed0d0 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f2f400640 1 -- 172.21.15.116:0/3813941037 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f340035f0 con 0x561c75bed0d0 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f2f400640 1 -- 172.21.15.116:0/3813941037 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3402b520 con 0x561c75bed0d0 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f4c770740 1 -- 172.21.15.116:0/3813941037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bed0d0 msgr2=0x561c75be71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bed0d0 0x561c75be71d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009230 tx=0x7f9f34011810 comp rx=0 tx=0).stop 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f4c770740 1 -- 172.21.15.116:0/3813941037 shutdown_connections 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.563+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bf2c30 0x561c75be7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.564+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.564+0000 7f9f4c770740 1 --2- 172.21.15.116:0/3813941037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bed0d0 0x561c75be71d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.564+0000 7f9f4c770740 1 -- 172.21.15.116:0/3813941037 >> 172.21.15.116:0/3813941037 conn(0x561c75be8640 msgr2=0x561c75beefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.564+0000 7f9f4c770740 1 -- 172.21.15.116:0/3813941037 shutdown_connections 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.564+0000 7f9f4c770740 1 -- 172.21.15.116:0/3813941037 wait complete. 2024-03-04T07:48:17.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.564+0000 7f9f4c770740 1 -- start start 2024-03-04T07:48:17.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.564+0000 7f9f4c770740 1 Processor -- start 2024-03-04T07:48:17.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bfc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f4c770740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bf2c30 0x561c75bfde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f4c770740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f441a9960 con 0x561c75bed0d0 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f4c770740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f441aac80 con 0x561c75bf2c30 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f4c770740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f441a9fc0 con 0x561c75bedac0 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f4c770740 10 client.0 _ll_register_callbacks cb 0x7f9f300f2d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bfc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f43400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bf2c30 0x561c75bfde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bfc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49724/0 (socket says 172.21.15.116:49724) 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 -- 172.21.15.116:0/1730886620 learned_addr learned my addr 172.21.15.116:0/1730886620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42000640 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 -- 172.21.15.116:0/1730886620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 msgr2=0x561c75bfce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 -- 172.21.15.116:0/1730886620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bf2c30 msgr2=0x561c75bfde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bf2c30 0x561c75bfde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.565+0000 7f9f42a00640 1 -- 172.21.15.116:0/1730886620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c75bfe3f0 con 0x561c75bed0d0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f42a00640 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bfc900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9f340057c0 tx=0x7f9f340093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b446c4cc532c4eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f2cc00640 1 -- 172.21.15.116:0/1730886620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f340039b0 con 0x561c75bed0d0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f340035f0 con 0x561c75bed0d0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f2cc00640 1 -- 172.21.15.116:0/1730886620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9f340050b0 con 0x561c75bed0d0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f2cc00640 1 -- 172.21.15.116:0/1730886620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f340052b0 con 0x561c75bed0d0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f4c770740 10 client.17067 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561c75c5f2a0 con 0x561c75bed0d0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.566+0000 7f9f2cc00640 1 -- 172.21.15.116:0/1730886620 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f340035f0 con 0x561c75bed0d0 2024-03-04T07:48:17.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f2cc00640 1 -- 172.21.15.116:0/1730886620 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9f34033070 con 0x561c75bed0d0 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f4c770740 10 client.17067 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f4c770740 10 client.17067 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f4c770740 20 client.17067 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f4c770740 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c5fb10 0x561c75c61f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f4c770740 4 client.17067 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561c75b5a2b0 con 0x561c75c5fb10 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f42000640 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c5fb10 0x561c75c61f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f42000640 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c5fb10 0x561c75c61f30 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aabf2765ac6d102c server_cookie=cb7b70b57c5ed62d in_seq=0 out_seq=0 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.567+0000 7f9f2cc00640 10 client.17067 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.568+0000 7f9f2cc00640 1 -- 172.21.15.116:0/1730886620 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561c75b5a2b0 con 0x561c75c5fb10 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.568+0000 7f9f2cc00640 10 client.17067 handle_command_reply: tid=0 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 client.17067 shutdown 2024-03-04T07:48:17.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 msgr2=0x561c75bfc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bfc900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9f340057c0 tx=0x7f9f340093a0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 shutdown_connections 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561c75c5fb10 0x561c75c61f30 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561c75bf2c30 0x561c75bfde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561c75bedac0 0x561c75bfce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 --2- 172.21.15.116:0/1730886620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c75bed0d0 0x561c75bfc900 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 >> 172.21.15.116:0/1730886620 conn(0x561c75be8640 msgr2=0x561c75be71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f2cc00640 0 client.17067 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 shutdown_connections 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 1 -- 172.21.15.116:0/1730886620 wait complete. 2024-03-04T07:48:17.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.569+0000 7f9f4c770740 20 client.17067 trim_cache size 0 max 16384 2024-03-04T07:48:17.575 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:48:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "9edf0454-c901-4332-a68b-fb5172050ffb", 2024-03-04T07:48:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:48:17.575 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9f280a1d80 msgr2=0x7f9f280a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9f280a1d80 0x7f9f280a4240 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x561c75bee320 tx=0x7f9f3c0095c0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f441027d0 msgr2=0x7f9f4419a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f441027d0 0x7f9f4419a080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9f3002c0b0 tx=0x7f9f30032430 comp rx=0 tx=0).stop 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 shutdown_connections 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f44103190 0x7f9f441a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f441027d0 0x7f9f4419a080 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f9f280a1d80 0x7f9f280a4240 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 --2- 172.21.15.116:0/2600732543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9f441015e0 0x7f9f44199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 >> 172.21.15.116:0/2600732543 conn(0x7f9f440fcb70 msgr2=0x7f9f44073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 shutdown_connections 2024-03-04T07:48:17.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.570+0000 7f9f49a00640 1 -- 172.21.15.116:0/2600732543 wait complete. 2024-03-04T07:48:17.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:48:17.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 -- 172.21.15.116:0/1289423332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81036f0 msgr2=0x7fcfa8103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 --2- 172.21.15.116:0/1289423332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81036f0 0x7fcfa8103b10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009900 tx=0x7fcf9802bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 -- 172.21.15.116:0/1289423332 shutdown_connections 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 --2- 172.21.15.116:0/1289423332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa81052a0 0x7fcfa810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 --2- 172.21.15.116:0/1289423332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81048e0 0x7fcfa8104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 --2- 172.21.15.116:0/1289423332 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81036f0 0x7fcfa8103b10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 -- 172.21.15.116:0/1289423332 >> 172.21.15.116:0/1289423332 conn(0x7fcfa80fec80 msgr2=0x7fcfa81010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.795+0000 7fcfaca00640 1 -- 172.21.15.116:0/1289423332 shutdown_connections 2024-03-04T07:48:17.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.796+0000 7fcfaca00640 1 -- 172.21.15.116:0/1289423332 wait complete. 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 Processor -- start 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 -- start start 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81036f0 0x7fcfa81999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa81048e0 0x7fcfa8199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81052a0 0x7fcfa81a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa810f390 con 0x7fcfa81036f0 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa810f210 con 0x7fcfa81052a0 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfaca00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa810f510 con 0x7fcfa81048e0 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.797+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81036f0 0x7fcfa81999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81036f0 0x7fcfa81999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49740/0 (socket says 172.21.15.116:49740) 2024-03-04T07:48:17.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 -- 172.21.15.116:0/3528699577 learned_addr learned my addr 172.21.15.116:0/3528699577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81052a0 0x7fcfa81a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa81048e0 0x7fcfa8199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 -- 172.21.15.116:0/3528699577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa81048e0 msgr2=0x7fcfa8199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa81048e0 0x7fcfa8199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 -- 172.21.15.116:0/3528699577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81052a0 msgr2=0x7fcfa81a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81052a0 0x7fcfa81a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 -- 172.21.15.116:0/3528699577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa81a1600 con 0x7fcfa81036f0 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7e00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81052a0 0x7fcfa81a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.798+0000 7fcfa7400640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81036f0 0x7fcfa81999e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcf98000c00 tx=0x7fcf9802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.799+0000 7fcf9fe00640 1 -- 172.21.15.116:0/3528699577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98043020 con 0x7fcfa81036f0 2024-03-04T07:48:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.799+0000 7fcf9fe00640 1 -- 172.21.15.116:0/3528699577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf980054e0 con 0x7fcfa81036f0 2024-03-04T07:48:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.799+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa81a1890 con 0x7fcfa81036f0 2024-03-04T07:48:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.799+0000 7fcf9fe00640 1 -- 172.21.15.116:0/3528699577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98003c40 con 0x7fcfa81036f0 2024-03-04T07:48:17.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.799+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa81a1de0 con 0x7fcfa81036f0 2024-03-04T07:48:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 Processor -- start 2024-03-04T07:48:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 -- start start 2024-03-04T07:48:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcf9fe00640 1 -- 172.21.15.116:0/3528699577 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fcf98046020 con 0x7fcfa81036f0 2024-03-04T07:48:17.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d2f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2f700 0x5566d3d34200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d36d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3c5c090 con 0x5566d3d34870 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3c93d30 con 0x5566d3d2f700 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.802+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3c3ca30 con 0x5566d3d2ed10 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2f700 0x5566d3d34200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d36d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2f700 0x5566d3d34200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38136/0 (socket says 172.21.15.116:38136) 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 -- 172.21.15.116:0/3976245712 learned_addr learned my addr 172.21.15.116:0/3976245712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7e00640 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d2f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 -- 172.21.15.116:0/3976245712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 msgr2=0x5566d3d2f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d2f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 -- 172.21.15.116:0/3976245712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 msgr2=0x5566d3d36d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d36d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 -- 172.21.15.116:0/3976245712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566d3a727d0 con 0x5566d3d2f700 2024-03-04T07:48:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.803+0000 7fcfa7400640 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2f700 0x5566d3d34200 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fcf980f3d40 tx=0x7fcf980f3e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6dad0e8b42b1ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcf93e00640 1 -- 172.21.15.116:0/3976245712 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980ff070 con 0x5566d3d2f700 2024-03-04T07:48:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcf93e00640 1 -- 172.21.15.116:0/3976245712 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf980f5400 con 0x5566d3d2f700 2024-03-04T07:48:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcf93e00640 1 -- 172.21.15.116:0/3976245712 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980f45e0 con 0x5566d3d2f700 2024-03-04T07:48:17.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcf9fe00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcf9fe00640 1 -- 172.21.15.116:0/3528699577 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcf980efd60 con 0x7fcfa81036f0 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcfa7e00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcfb0e03740 1 -- 172.21.15.116:0/3976245712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2f700 msgr2=0x5566d3d34200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.804+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2f700 0x5566d3d34200 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fcf980f3d40 tx=0x7fcf980f3e30 comp rx=0 tx=0).stop 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 -- 172.21.15.116:0/3976245712 shutdown_connections 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d36d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2f700 0x5566d3d34200 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/3976245712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d2f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 -- 172.21.15.116:0/3976245712 >> 172.21.15.116:0/3976245712 conn(0x5566d3d2a280 msgr2=0x5566d3d2a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfa7e00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x5566d3d2ff60 tx=0x7fcfa0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 -- 172.21.15.116:0/3976245712 shutdown_connections 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 -- 172.21.15.116:0/3976245712 wait complete. 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 -- start start 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.805+0000 7fcfb0e03740 1 Processor -- start 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2ed10 0x5566d3d38d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2f700 0x5566d3d392d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d40310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3c320c0 con 0x5566d3d2ed10 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3c8fdd0 con 0x5566d3d34870 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3d375e0 con 0x5566d3d2f700 2024-03-04T07:48:17.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfb0e03740 10 client.0 _ll_register_callbacks cb 0x5566d3c4f860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2ed10 0x5566d3d38d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d40310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2ed10 0x5566d3d38d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49766/0 (socket says 172.21.15.116:49766) 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 -- 172.21.15.116:0/2834588254 learned_addr learned my addr 172.21.15.116:0/2834588254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2f700 0x5566d3d392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 -- 172.21.15.116:0/2834588254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2f700 msgr2=0x5566d3d392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2f700 0x5566d3d392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 -- 172.21.15.116:0/2834588254 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 msgr2=0x5566d3d40310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d40310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.806+0000 7fcfa7400640 1 -- 172.21.15.116:0/2834588254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566d39a83f0 con 0x5566d3d2ed10 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcfa7400640 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2ed10 0x5566d3d38d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcf980bca20 tx=0x7fcf9802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fe2078f414a7a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcf90c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcf90c00640 1 -- 172.21.15.116:0/2834588254 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980ff070 con 0x5566d3d2ed10 2024-03-04T07:48:17.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcf90c00640 1 -- 172.21.15.116:0/2834588254 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcf980bad90 con 0x5566d3d2ed10 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf980f5400 con 0x5566d3d2ed10 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcf90c00640 1 -- 172.21.15.116:0/2834588254 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf980b8d30 con 0x5566d3d2ed10 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcfb0e03740 10 client.17079 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5566d39680f0 con 0x5566d3d2ed10 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.807+0000 7fcf90c00640 1 -- 172.21.15.116:0/2834588254 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf980f5400 con 0x5566d3d2ed10 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcf90c00640 1 -- 172.21.15.116:0/2834588254 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcf980bccf0 con 0x5566d3d2ed10 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcfb0e03740 10 client.17079 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcfb0e03740 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcfb0e03740 20 client.17079 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3d44470 0x5566d3d46860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcfb0e03740 4 client.17079 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566d3ca1270 con 0x5566d3d44470 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.808+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3d44470 0x5566d3d46860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.809+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3d44470 0x5566d3d46860 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83c4575f0f79312a server_cookie=a71ede551ab3f8f in_seq=0 out_seq=0 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.809+0000 7fcf90c00640 10 client.17079 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.809+0000 7fcf90c00640 1 -- 172.21.15.116:0/2834588254 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5566d3ca1270 con 0x5566d3d44470 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.809+0000 7fcf90c00640 10 client.17079 handle_command_reply: tid=0 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.809+0000 7fcfb0e03740 1 client.17079 shutdown 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.809+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2ed10 msgr2=0x5566d3d38d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.809+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2ed10 0x5566d3d38d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcf980bca20 tx=0x7fcf9802bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 shutdown_connections 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3d44470 0x5566d3d46860 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d40310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2f700 0x5566d3d392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/2834588254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2ed10 0x5566d3d38d90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcf90c00640 0 client.17079 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 >> 172.21.15.116:0/2834588254 conn(0x5566d3d2a280 msgr2=0x5566d3d294c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 shutdown_connections 2024-03-04T07:48:17.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 1 -- 172.21.15.116:0/2834588254 wait complete. 2024-03-04T07:48:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.810+0000 7fcfb0e03740 20 client.17079 trim_cache size 0 max 16384 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 Processor -- start 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 -- start start 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d28c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2f700 0x5566d3d402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d29170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81a8ae0 con 0x5566d3d2f700 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81ab0c0 con 0x5566d3d34870 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.827+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81a90e0 con 0x5566d3d2ed10 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2f700 0x5566d3d402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d29170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2f700 0x5566d3d402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49782/0 (socket says 172.21.15.116:49782) 2024-03-04T07:48:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 -- 172.21.15.116:0/4003428198 learned_addr learned my addr 172.21.15.116:0/4003428198 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa7400640 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d28c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 -- 172.21.15.116:0/4003428198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 msgr2=0x5566d3d28c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d28c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 -- 172.21.15.116:0/4003428198 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 msgr2=0x5566d3d29170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d29170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 -- 172.21.15.116:0/4003428198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566d3d46bd0 con 0x5566d3d2f700 2024-03-04T07:48:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.828+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2f700 0x5566d3d402a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x5566d3d470b0 tx=0x7fcf94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f7277774a6f7ee server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcf93400640 1 -- 172.21.15.116:0/4003428198 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94004030 con 0x5566d3d2f700 2024-03-04T07:48:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcf93400640 1 -- 172.21.15.116:0/4003428198 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf94011df0 con 0x5566d3d2f700 2024-03-04T07:48:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcf93400640 1 -- 172.21.15.116:0/4003428198 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94003ab0 con 0x5566d3d2f700 2024-03-04T07:48:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 -- 172.21.15.116:0/4003428198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2f700 msgr2=0x5566d3d402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2f700 0x5566d3d402a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x5566d3d470b0 tx=0x7fcf94005f90 comp rx=0 tx=0).stop 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 -- 172.21.15.116:0/4003428198 shutdown_connections 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d34870 0x5566d3d29170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d2f700 0x5566d3d402a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/4003428198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d2ed10 0x5566d3d28c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 -- 172.21.15.116:0/4003428198 >> 172.21.15.116:0/4003428198 conn(0x5566d3d2a280 msgr2=0x5566d3d389a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 -- 172.21.15.116:0/4003428198 shutdown_connections 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.829+0000 7fcfb0e03740 1 -- 172.21.15.116:0/4003428198 wait complete. 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 -- start start 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 Processor -- start 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2ed10 0x5566d3d3e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d3e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d3f820 0x5566d3d3fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81a9740 con 0x5566d3d34870 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81aaa60 con 0x5566d3d2ed10 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81a9da0 con 0x5566d3d3f820 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.830+0000 7fcfb0e03740 10 client.0 _ll_register_callbacks cb 0x7fcf980f3b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2ed10 0x5566d3d3e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2ed10 0x5566d3d3e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38158/0 (socket says 172.21.15.116:38158) 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d3e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 -- 172.21.15.116:0/1348914143 learned_addr learned my addr 172.21.15.116:0/1348914143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7e00640 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d3f820 0x5566d3d3fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 -- 172.21.15.116:0/1348914143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d3f820 msgr2=0x5566d3d3fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d3f820 0x5566d3d3fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 -- 172.21.15.116:0/1348914143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 msgr2=0x5566d3d3e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d3e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 -- 172.21.15.116:0/1348914143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566d3d401e0 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfa7400640 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2ed10 0x5566d3d3e2d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcf9802c170 tx=0x7fcf980f3390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=624b37d2e750876a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcf90c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcf90c00640 1 -- 172.21.15.116:0/1348914143 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980ff070 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf94011df0 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.831+0000 7fcf90c00640 1 -- 172.21.15.116:0/1348914143 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcf980bad90 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcf90c00640 1 -- 172.21.15.116:0/1348914143 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf980bc3f0 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 10 client.27212 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5566d3d401e0 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcf90c00640 1 -- 172.21.15.116:0/1348914143 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf94011df0 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcf90c00640 1 -- 172.21.15.116:0/1348914143 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcf980bccf0 con 0x5566d3d2ed10 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 10 client.27212 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 10 client.27212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 20 client.27212 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:17.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3da0f20 0x5566d3da3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 4 client.27212 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:48:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5566d3c9b5c0 con 0x5566d3da0f20 2024-03-04T07:48:17.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.832+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3da0f20 0x5566d3da3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcf90c00640 10 client.27212 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfa6a00640 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3da0f20 0x5566d3da3340 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76329b31100cb0a6 server_cookie=8d5bc8edd45c3b57 in_seq=0 out_seq=0 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcf90c00640 1 -- 172.21.15.116:0/1348914143 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5566d3c9b5c0 con 0x5566d3da0f20 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcf90c00640 10 client.27212 handle_command_reply: tid=0 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 client.27212 shutdown 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2ed10 msgr2=0x5566d3d3e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2ed10 0x5566d3d3e2d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcf9802c170 tx=0x7fcf980f3390 comp rx=0 tx=0).stop 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 shutdown_connections 2024-03-04T07:48:17.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5566d3da0f20 0x5566d3da3340 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5566d3d3f820 0x5566d3d3fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566d3d34870 0x5566d3d3e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcf90c00640 0 client.27212 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 --2- 172.21.15.116:0/1348914143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5566d3d2ed10 0x5566d3d3e2d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 >> 172.21.15.116:0/1348914143 conn(0x5566d3d2a280 msgr2=0x5566d3d2d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.833+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 shutdown_connections 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfb0e03740 1 -- 172.21.15.116:0/1348914143 wait complete. 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfb0e03740 20 client.27212 trim_cache size 0 max 16384 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:17.838 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:48:17.839 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:48:17.839 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:17.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf8c0a1d80 msgr2=0x7fcf8c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x5566d3d2ff60 tx=0x7fcfa0009500 comp rx=0 tx=0).stop 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81036f0 msgr2=0x7fcfa81999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81036f0 0x7fcfa81999e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcf98000c00 tx=0x7fcf9802c650 comp rx=0 tx=0).stop 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 shutdown_connections 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcfa81052a0 0x7fcfa81a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfa81048e0 0x7fcfa8199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 --2- 172.21.15.116:0/3528699577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfa81036f0 0x7fcfa81999e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 >> 172.21.15.116:0/3528699577 conn(0x7fcfa80fec80 msgr2=0x7fcfa8107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.834+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 shutdown_connections 2024-03-04T07:48:17.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:17.835+0000 7fcfaca00640 1 -- 172.21.15.116:0/3528699577 wait complete. 2024-03-04T07:48:17.851 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:48:17.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:48:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 -- 172.21.15.116:0/2983080513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360077b70 msgr2=0x7fe360078410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 --2- 172.21.15.116:0/2983080513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360077b70 0x7fe360078410 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe358009ca0 tx=0x7fe35802c100 comp rx=0 tx=0).stop 2024-03-04T07:48:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 -- 172.21.15.116:0/2983080513 shutdown_connections 2024-03-04T07:48:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 --2- 172.21.15.116:0/2983080513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360077b70 0x7fe360078410 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 --2- 172.21.15.116:0/2983080513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3600771d0 0x7fe360077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 --2- 172.21.15.116:0/2983080513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe360075f90 0x7fe3600763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 -- 172.21.15.116:0/2983080513 >> 172.21.15.116:0/2983080513 conn(0x7fe3600fed80 msgr2=0x7fe3601011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 -- 172.21.15.116:0/2983080513 shutdown_connections 2024-03-04T07:48:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.046+0000 7fe367400640 1 -- 172.21.15.116:0/2983080513 wait complete. 2024-03-04T07:48:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.047+0000 7fe367400640 1 Processor -- start 2024-03-04T07:48:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.047+0000 7fe367400640 1 -- start start 2024-03-04T07:48:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe367400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360075f90 0x7fe36019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe367400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3600771d0 0x7fe36019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe367400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe360077b70 0x7fe3601a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe367400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3601138a0 con 0x7fe3600771d0 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe367400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360113720 con 0x7fe360075f90 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe367400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360113a20 con 0x7fe360077b70 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe364a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360075f90 0x7fe36019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe35fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3600771d0 0x7fe36019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe364a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360075f90 0x7fe36019ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38170/0 (socket says 172.21.15.116:38170) 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe364a00640 1 -- 172.21.15.116:0/1867994559 learned_addr learned my addr 172.21.15.116:0/1867994559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe365400640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe360077b70 0x7fe3601a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe364a00640 1 -- 172.21.15.116:0/1867994559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe360077b70 msgr2=0x7fe3601a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe364a00640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe360077b70 0x7fe3601a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe364a00640 1 -- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3600771d0 msgr2=0x7fe36019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.048+0000 7fe364a00640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3600771d0 0x7fe36019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.049+0000 7fe364a00640 1 -- 172.21.15.116:0/1867994559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3601a5af0 con 0x7fe360075f90 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.049+0000 7fe364a00640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360075f90 0x7fe36019ded0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe35400d950 tx=0x7fe35400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.049+0000 7fe35d600640 1 -- 172.21.15.116:0/1867994559 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354017070 con 0x7fe360075f90 2024-03-04T07:48:18.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.049+0000 7fe35d600640 1 -- 172.21.15.116:0/1867994559 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe354009c30 con 0x7fe360075f90 2024-03-04T07:48:18.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.049+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3601a5de0 con 0x7fe360075f90 2024-03-04T07:48:18.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.049+0000 7fe35d600640 1 -- 172.21.15.116:0/1867994559 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400f6b0 con 0x7fe360075f90 2024-03-04T07:48:18.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.049+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3601a6240 con 0x7fe360075f90 2024-03-04T07:48:18.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.050+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe360110310 con 0x7fe360075f90 2024-03-04T07:48:18.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.052+0000 7fe35d600640 1 -- 172.21.15.116:0/1867994559 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fe354010460 con 0x7fe360075f90 2024-03-04T07:48:18.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.053+0000 7fe35d600640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3480a1d80 0x7fe3480a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.053+0000 7fe35d600640 1 -- 172.21.15.116:0/1867994559 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe3540cdc70 con 0x7fe360075f90 2024-03-04T07:48:18.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.053+0000 7fe35fe00640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3480a1d80 0x7fe3480a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.054+0000 7fe35fe00640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3480a1d80 0x7fe3480a4240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe36019f3e0 tx=0x7fe35000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.057+0000 7fe35d600640 1 -- 172.21.15.116:0/1867994559 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe354009de0 con 0x7fe360075f90 2024-03-04T07:48:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:17 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:48:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:17 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:17 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:48:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:17 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:18.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:17 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:48:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:17 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:48:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:17 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:17 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:48:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:17 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:17 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:48:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.296+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe360062270 con 0x7fe360075f90 2024-03-04T07:48:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.297+0000 7fe35d600640 1 -- 172.21.15.116:0/1867994559 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fe35409aa50 con 0x7fe360075f90 2024-03-04T07:48:18.297 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:48:18.297 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:48:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:48:18.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.299+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3480a1d80 msgr2=0x7fe3480a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.299+0000 7fe367400640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3480a1d80 0x7fe3480a4240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe36019f3e0 tx=0x7fe35000b390 comp rx=0 tx=0).stop 2024-03-04T07:48:18.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.299+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360075f90 msgr2=0x7fe36019ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.299+0000 7fe367400640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360075f90 0x7fe36019ded0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe35400d950 tx=0x7fe35400de10 comp rx=0 tx=0).stop 2024-03-04T07:48:18.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.299+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 shutdown_connections 2024-03-04T07:48:18.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.300+0000 7fe367400640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fe3480a1d80 0x7fe3480a4240 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.300+0000 7fe367400640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe360077b70 0x7fe3601a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.300+0000 7fe367400640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3600771d0 0x7fe36019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.300+0000 7fe367400640 1 --2- 172.21.15.116:0/1867994559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe360075f90 0x7fe36019ded0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.300+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 >> 172.21.15.116:0/1867994559 conn(0x7fe3600fed80 msgr2=0x7fe360100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.300+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 shutdown_connections 2024-03-04T07:48:18.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.300+0000 7fe367400640 1 -- 172.21.15.116:0/1867994559 wait complete. 2024-03-04T07:48:18.318 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:48:18.318 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:48:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:17 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:48:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:17 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:17 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:48:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:17 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:17 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:48:18.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.529+0000 7f445f000640 1 -- 172.21.15.116:0/4290638986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f44580ff8a0 msgr2=0x7f44580ffcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.529+0000 7f445f000640 1 --2- 172.21.15.116:0/4290638986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f44580ff8a0 0x7f44580ffcc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4448009900 tx=0x7f444802bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.529+0000 7f445f000640 1 -- 172.21.15.116:0/4290638986 shutdown_connections 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.529+0000 7f445f000640 1 --2- 172.21.15.116:0/4290638986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4458103230 0x7f4458109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.529+0000 7f445f000640 1 --2- 172.21.15.116:0/4290638986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4458102870 0x7f4458102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.529+0000 7f445f000640 1 --2- 172.21.15.116:0/4290638986 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f44580ff8a0 0x7f44580ffcc0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.529+0000 7f445f000640 1 -- 172.21.15.116:0/4290638986 >> 172.21.15.116:0/4290638986 conn(0x7f44580fb1f0 msgr2=0x7f44580fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.530+0000 7f445f000640 1 -- 172.21.15.116:0/4290638986 shutdown_connections 2024-03-04T07:48:18.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.530+0000 7f445f000640 1 -- 172.21.15.116:0/4290638986 wait complete. 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 Processor -- start 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 -- start start 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44580ff8a0 0x7f4458199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4458102870 0x7f445819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4458103230 0x7f44581a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445810d2d0 con 0x7f44580ff8a0 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445810d150 con 0x7f4458103230 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.531+0000 7f445f000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445810d450 con 0x7f4458102870 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44580ff8a0 0x7f4458199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44580ff8a0 0x7f4458199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49818/0 (socket says 172.21.15.116:49818) 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 -- 172.21.15.116:0/894812143 learned_addr learned my addr 172.21.15.116:0/894812143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f445d000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4458103230 0x7f44581a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457400640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4458102870 0x7f445819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 -- 172.21.15.116:0/894812143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4458102870 msgr2=0x7f445819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4458102870 0x7f445819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 -- 172.21.15.116:0/894812143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4458103230 msgr2=0x7f44581a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4458103230 0x7f44581a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f4457e00640 1 -- 172.21.15.116:0/894812143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44581a1740 con 0x7f44580ff8a0 2024-03-04T07:48:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.532+0000 7f445d000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4458103230 0x7f44581a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:18.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.533+0000 7f4457e00640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44580ff8a0 0x7f4458199b20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f444802efd0 tx=0x7f444802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.533+0000 7f4454c00640 1 -- 172.21.15.116:0/894812143 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4448041070 con 0x7f44580ff8a0 2024-03-04T07:48:18.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.533+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44581a19d0 con 0x7f44580ff8a0 2024-03-04T07:48:18.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.533+0000 7f4454c00640 1 -- 172.21.15.116:0/894812143 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4448004450 con 0x7f44580ff8a0 2024-03-04T07:48:18.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.533+0000 7f4454c00640 1 -- 172.21.15.116:0/894812143 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444804b060 con 0x7f44580ff8a0 2024-03-04T07:48:18.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.533+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44581a1ee0 con 0x7f44580ff8a0 2024-03-04T07:48:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.537+0000 7f4461d7a740 1 Processor -- start 2024-03-04T07:48:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.537+0000 7f4461d7a740 1 -- start start 2024-03-04T07:48:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.537+0000 7f4454c00640 1 -- 172.21.15.116:0/894812143 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4448044070 con 0x7f44580ff8a0 2024-03-04T07:48:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0c7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c7800 0x56139e0c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4461d7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56139e026d60 con 0x56139e0c2690 2024-03-04T07:48:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4461d7a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56139dfcfcf0 con 0x56139e0c1ca0 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4461d7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56139e022e00 con 0x56139e0c7800 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4457e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0c20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38200/0 (socket says 172.21.15.116:38200) 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 -- 172.21.15.116:0/3014958887 learned_addr learned my addr 172.21.15.116:0/3014958887 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f4457400640 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c7800 0x56139e0c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 -- 172.21.15.116:0/3014958887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c7800 msgr2=0x56139e0c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c7800 0x56139e0c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 -- 172.21.15.116:0/3014958887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 msgr2=0x56139e0c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.538+0000 7f445d000640 1 -- 172.21.15.116:0/3014958887 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56139dd3b3f0 con 0x56139e0c1ca0 2024-03-04T07:48:18.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.539+0000 7f445d000640 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0c20c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f44500117e0 tx=0x7f4450011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4a7510256d57f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.539+0000 7f444d600640 1 -- 172.21.15.116:0/3014958887 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011310 con 0x56139e0c1ca0 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.539+0000 7f444d600640 1 -- 172.21.15.116:0/3014958887 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4450003e50 con 0x56139e0c1ca0 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.539+0000 7f444d600640 1 -- 172.21.15.116:0/3014958887 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44500045e0 con 0x56139e0c1ca0 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.539+0000 7f4454c00640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f443c0a1d80 0x7f443c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.539+0000 7f4454c00640 1 -- 172.21.15.116:0/894812143 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f44480efc30 con 0x7f44580ff8a0 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.539+0000 7f4457e00640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f443c0a1d80 0x7f443c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 -- 172.21.15.116:0/3014958887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 msgr2=0x56139e0c20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0c20c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f44500117e0 tx=0x7f4450011ca0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 -- 172.21.15.116:0/3014958887 shutdown_connections 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c7800 0x56139e0c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 --2- 172.21.15.116:0/3014958887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0c20c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 -- 172.21.15.116:0/3014958887 >> 172.21.15.116:0/3014958887 conn(0x56139e0bd210 msgr2=0x56139e0bd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4457e00640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f443c0a1d80 0x7f443c0a4240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f44480c4b90 tx=0x7f44480f3670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 -- 172.21.15.116:0/3014958887 shutdown_connections 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 -- 172.21.15.116:0/3014958887 wait complete. 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.540+0000 7f4461d7a740 1 -- start start 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 1 Processor -- start 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c7800 0x56139e0d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56139e0ca7b0 con 0x56139e0c2690 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56139dfc5220 con 0x56139e0c7800 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56139e0ca930 con 0x56139e0c1ca0 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4461d7a740 10 client.0 _ll_register_callbacks cb 0x56139dd3abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4457400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f445d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c7800 0x56139e0d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4457400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49846/0 (socket says 172.21.15.116:49846) 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f4457400640 1 -- 172.21.15.116:0/4138580070 learned_addr learned my addr 172.21.15.116:0/4138580070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.541+0000 7f445d000640 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4457400640 1 -- 172.21.15.116:0/4138580070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 msgr2=0x56139e0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4457400640 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4457400640 1 -- 172.21.15.116:0/4138580070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c7800 msgr2=0x56139e0d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4457400640 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c7800 0x56139e0d3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4457400640 1 -- 172.21.15.116:0/4138580070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56139de057d0 con 0x56139e0c2690 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4457400640 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0cc710 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4444012700 tx=0x7f4444012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b80e71c09ae919 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4442000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4442000640 1 -- 172.21.15.116:0/4138580070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444400f8c0 con 0x56139e0c2690 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4450003e50 con 0x56139e0c2690 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4442000640 1 -- 172.21.15.116:0/4138580070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f444400fa60 con 0x56139e0c2690 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.542+0000 7f4442000640 1 -- 172.21.15.116:0/4138580070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f444400fc60 con 0x56139e0c2690 2024-03-04T07:48:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 10 client.17103 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56139e0d6be0 con 0x56139e0c2690 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4442000640 1 -- 172.21.15.116:0/4138580070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4450003e50 con 0x56139e0c2690 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4442000640 1 -- 172.21.15.116:0/4138580070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f444401c420 con 0x56139e0c2690 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 10 client.17103 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 10 client.17103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 20 client.17103 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e0d8250 0x56139e0da670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 4 client.17103 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56139e0342a0 con 0x56139e0d8250 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.543+0000 7f445d000640 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e0d8250 0x56139e0da670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.544+0000 7f445d000640 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e0d8250 0x56139e0da670 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a67547f821c0817 server_cookie=1d41c5869d178db0 in_seq=0 out_seq=0 2024-03-04T07:48:18.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.544+0000 7f4442000640 10 client.17103 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:18.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.544+0000 7f4442000640 1 -- 172.21.15.116:0/4138580070 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56139e0342a0 con 0x56139e0d8250 2024-03-04T07:48:18.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4442000640 10 client.17103 handle_command_reply: tid=0 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 client.17103 shutdown 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 msgr2=0x56139e0cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0cc710 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4444012700 tx=0x7f4444012bc0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 shutdown_connections 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e0d8250 0x56139e0da670 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c7800 0x56139e0d3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4442000640 0 client.17103 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 --2- 172.21.15.116:0/4138580070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 >> 172.21.15.116:0/4138580070 conn(0x56139e0bd210 msgr2=0x56139e0bc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 shutdown_connections 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 1 -- 172.21.15.116:0/4138580070 wait complete. 2024-03-04T07:48:18.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.545+0000 7f4461d7a740 20 client.17103 trim_cache size 0 max 16384 2024-03-04T07:48:18.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.558+0000 7f4461d7a740 1 Processor -- start 2024-03-04T07:48:18.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.558+0000 7f4461d7a740 1 -- start start 2024-03-04T07:48:18.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4461d7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581a8c20 con 0x56139e0c7800 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4461d7a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581ab200 con 0x56139e0c2690 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4461d7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581a9220 con 0x56139e0c1ca0 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38218/0 (socket says 172.21.15.116:38218) 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 -- 172.21.15.116:0/946662365 learned_addr learned my addr 172.21.15.116:0/946662365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f445d000640 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f445d000640 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 -- 172.21.15.116:0/946662365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 msgr2=0x56139e0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 -- 172.21.15.116:0/946662365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 msgr2=0x56139e0d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f4457400640 1 -- 172.21.15.116:0/946662365 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56139e0d6a00 con 0x56139e0c2690 2024-03-04T07:48:18.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.559+0000 7f445d000640 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.560+0000 7f4457400640 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c2690 0x56139e0cc710 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4444000c00 tx=0x7f444400ff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cddcb02ceff557a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.560+0000 7f444cc00640 1 -- 172.21.15.116:0/946662365 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444402b030 con 0x56139e0c2690 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.560+0000 7f444cc00640 1 -- 172.21.15.116:0/946662365 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44440055f0 con 0x56139e0c2690 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.560+0000 7f444cc00640 1 -- 172.21.15.116:0/946662365 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444400f9b0 con 0x56139e0c2690 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.560+0000 7f4461d7a740 1 -- 172.21.15.116:0/946662365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c2690 msgr2=0x56139e0cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.560+0000 7f4461d7a740 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c2690 0x56139e0cc710 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4444000c00 tx=0x7f444400ff90 comp rx=0 tx=0).stop 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 -- 172.21.15.116:0/946662365 shutdown_connections 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c2690 0x56139e0cc710 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 --2- 172.21.15.116:0/946662365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c1ca0 0x56139e0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 -- 172.21.15.116:0/946662365 >> 172.21.15.116:0/946662365 conn(0x56139e0bd210 msgr2=0x56139e0c3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 -- 172.21.15.116:0/946662365 shutdown_connections 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 -- 172.21.15.116:0/946662365 wait complete. 2024-03-04T07:48:18.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 -- start start 2024-03-04T07:48:18.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.561+0000 7f4461d7a740 1 Processor -- start 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c2690 0x56139e0d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4461d7a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4461d7a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581a9880 con 0x56139e0c7800 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4461d7a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581aaba0 con 0x56139e0c1ca0 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4461d7a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581a9ee0 con 0x56139e0c2690 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4461d7a740 10 client.0 _ll_register_callbacks cb 0x7f4444009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38220/0 (socket says 172.21.15.116:38220) 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 -- 172.21.15.116:0/56908087 learned_addr learned my addr 172.21.15.116:0/56908087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4457400640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c2690 0x56139e0d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 -- 172.21.15.116:0/56908087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c2690 msgr2=0x56139e0d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c2690 0x56139e0d1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 -- 172.21.15.116:0/56908087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 msgr2=0x56139e0d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f445d000640 1 -- 172.21.15.116:0/56908087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56139e0d30b0 con 0x56139e0c1ca0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.562+0000 7f4457400640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c2690 0x56139e0d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f445d000640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0d1570 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f445002f480 tx=0x7f4450009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14a663951d0e5ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4442000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4442000640 1 -- 172.21.15.116:0/56908087 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450025040 con 0x56139e0c1ca0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4442000640 1 -- 172.21.15.116:0/56908087 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f445002f250 con 0x56139e0c1ca0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44440055f0 con 0x56139e0c1ca0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4442000640 1 -- 172.21.15.116:0/56908087 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4450011050 con 0x56139e0c1ca0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4461d7a740 10 client.27236 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56139e134160 con 0x56139e0c1ca0 2024-03-04T07:48:18.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4442000640 1 -- 172.21.15.116:0/56908087 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44440055f0 con 0x56139e0c1ca0 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4442000640 1 -- 172.21.15.116:0/56908087 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4450018020 con 0x56139e0c1ca0 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4461d7a740 10 client.27236 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4461d7a740 10 client.27236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.563+0000 7f4461d7a740 20 client.27236 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4461d7a740 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e134910 0x56139e136d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4461d7a740 4 client.27236 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56139e02e5f0 con 0x56139e134910 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4457400640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e134910 0x56139e136d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4457400640 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e134910 0x56139e136d30 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55bdde0771816337 server_cookie=176f498ab8cdbcdf in_seq=0 out_seq=0 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4442000640 10 client.27236 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4442000640 1 -- 172.21.15.116:0/56908087 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56139e02e5f0 con 0x56139e134910 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4442000640 10 client.27236 handle_command_reply: tid=0 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.564+0000 7f4461d7a740 1 client.27236 shutdown 2024-03-04T07:48:18.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 msgr2=0x56139e0d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0d1570 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f445002f480 tx=0x7f4450009500 comp rx=0 tx=0).stop 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 shutdown_connections 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56139e134910 0x56139e136d30 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56139e0c7800 0x56139e0d2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56139e0c2690 0x56139e0d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 --2- 172.21.15.116:0/56908087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56139e0c1ca0 0x56139e0d1570 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 >> 172.21.15.116:0/56908087 conn(0x56139e0bd210 msgr2=0x56139e0c0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4442000640 0 client.27236 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 shutdown_connections 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 1 -- 172.21.15.116:0/56908087 wait complete. 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f4461d7a740 20 client.27236 trim_cache size 0 max 16384 2024-03-04T07:48:18.571 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:48:18.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f443c0a1d80 msgr2=0x7f443c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f445f000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f443c0a1d80 0x7f443c0a4240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f44480c4b90 tx=0x7f44480f3670 comp rx=0 tx=0).stop 2024-03-04T07:48:18.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44580ff8a0 msgr2=0x7f4458199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:18.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.565+0000 7f445f000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44580ff8a0 0x7f4458199b20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f444802efd0 tx=0x7f444802c6c0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 shutdown_connections 2024-03-04T07:48:18.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4458103230 0x7f44581a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f443c0a1d80 0x7f443c0a4240 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4458102870 0x7f445819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 --2- 172.21.15.116:0/894812143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44580ff8a0 0x7f4458199b20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:18.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 >> 172.21.15.116:0/894812143 conn(0x7f44580fb1f0 msgr2=0x7f4458100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:18.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 shutdown_connections 2024-03-04T07:48:18.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:18.566+0000 7f445f000640 1 -- 172.21.15.116:0/894812143 wait complete. 2024-03-04T07:48:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:18 smithi116 ceph-mon[32150]: pgmap v627: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:18 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1867994559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:18 smithi152 ceph-mon[37154]: pgmap v627: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:18 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1867994559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:19.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:18 smithi133 ceph-mon[36505]: pgmap v627: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:19.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:18 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1867994559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:19.582 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:48:19.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:48:19.583 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 -- 172.21.15.116:0/4257296355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820107bf0 msgr2=0x7fd820108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 --2- 172.21.15.116:0/4257296355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820107bf0 0x7fd820108010 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd814009900 tx=0x7fd81402bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 -- 172.21.15.116:0/4257296355 shutdown_connections 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 --2- 172.21.15.116:0/4257296355 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8201097a0 0x7fd820110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 --2- 172.21.15.116:0/4257296355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd820108de0 0x7fd820109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 --2- 172.21.15.116:0/4257296355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd820107bf0 0x7fd820108010 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 -- 172.21.15.116:0/4257296355 >> 172.21.15.116:0/4257296355 conn(0x7fd820076b90 msgr2=0x7fd820078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:19.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 -- 172.21.15.116:0/4257296355 shutdown_connections 2024-03-04T07:48:19.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.810+0000 7fd827e00640 1 -- 172.21.15.116:0/4257296355 wait complete. 2024-03-04T07:48:19.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 Processor -- start 2024-03-04T07:48:19.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 -- start start 2024-03-04T07:48:19.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd820107bf0 0x7fd82019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd820108de0 0x7fd82019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8201097a0 0x7fd8201a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201138b0 con 0x7fd8201097a0 2024-03-04T07:48:19.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd820113730 con 0x7fd820108de0 2024-03-04T07:48:19.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.811+0000 7fd827e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd820113a30 con 0x7fd820107bf0 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8201097a0 0x7fd8201a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd824a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd820108de0 0x7fd82019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd820107bf0 0x7fd82019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8201097a0 0x7fd8201a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49876/0 (socket says 172.21.15.116:49876) 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd824a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd820108de0 0x7fd82019e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38246/0 (socket says 172.21.15.116:38246) 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825e00640 1 -- 172.21.15.116:0/1366035306 learned_addr learned my addr 172.21.15.116:0/1366035306 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825400640 1 -- 172.21.15.116:0/1366035306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd820108de0 msgr2=0x7fd82019e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825400640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd820108de0 0x7fd82019e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825400640 1 -- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8201097a0 msgr2=0x7fd8201a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825400640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8201097a0 0x7fd8201a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825400640 1 -- 172.21.15.116:0/1366035306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8201a5b20 con 0x7fd820107bf0 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8201097a0 0x7fd8201a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd824a00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd820108de0 0x7fd82019e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.812+0000 7fd825400640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd820107bf0 0x7fd82019df00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd814009230 tx=0x7fd81402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.813+0000 7fd81e000640 1 -- 172.21.15.116:0/1366035306 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814043020 con 0x7fd820107bf0 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.813+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8201a5db0 con 0x7fd820107bf0 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.813+0000 7fd81e000640 1 -- 172.21.15.116:0/1366035306 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8140054e0 con 0x7fd820107bf0 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.813+0000 7fd81e000640 1 -- 172.21.15.116:0/1366035306 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814003b60 con 0x7fd820107bf0 2024-03-04T07:48:19.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.813+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8201a6210 con 0x7fd820107bf0 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 Processor -- start 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 -- start start 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d39f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300d2b2e20 con 0x56300d39f0d0 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300d304a20 con 0x56300d3a4c30 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd82aa84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300d2adb00 con 0x56300d39fac0 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.816+0000 7fd81e000640 1 -- 172.21.15.116:0/1366035306 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd814046020 con 0x7fd820107bf0 2024-03-04T07:48:19.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38258/0 (socket says 172.21.15.116:38258) 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 -- 172.21.15.116:0/1317846247 learned_addr learned my addr 172.21.15.116:0/1317846247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd825e00640 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d39f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd825400640 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 -- 172.21.15.116:0/1317846247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 msgr2=0x56300d3a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 -- 172.21.15.116:0/1317846247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 msgr2=0x56300d39f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d39f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 -- 172.21.15.116:0/1317846247 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56300d2c2e10 con 0x56300d3a4c30 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd825400640 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd824a00640 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a70c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0114f0 tx=0x7fd80c0118f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c6f0109859f8ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.817+0000 7fd812600640 1 -- 172.21.15.116:0/1317846247 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c004020 con 0x56300d3a4c30 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd812600640 1 -- 172.21.15.116:0/1317846247 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd80c0041c0 con 0x56300d3a4c30 2024-03-04T07:48:19.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd812600640 1 -- 172.21.15.116:0/1317846247 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c00f850 con 0x56300d3a4c30 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd81e000640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8080a1d80 0x7fd8080a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd81e000640 1 -- 172.21.15.116:0/1366035306 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd8140efda0 con 0x7fd820107bf0 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 -- 172.21.15.116:0/1317846247 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 msgr2=0x56300d3a70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd825e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8080a1d80 0x7fd8080a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a70c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0114f0 tx=0x7fd80c0118f0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 -- 172.21.15.116:0/1317846247 shutdown_connections 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a70c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 --2- 172.21.15.116:0/1317846247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d39f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 -- 172.21.15.116:0/1317846247 >> 172.21.15.116:0/1317846247 conn(0x56300d39a640 msgr2=0x56300d39aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 -- 172.21.15.116:0/1317846247 shutdown_connections 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.818+0000 7fd82aa84740 1 -- 172.21.15.116:0/1317846247 wait complete. 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.819+0000 7fd825e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8080a1d80 0x7fd8080a4240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x56300d3a0320 tx=0x7fd818009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:19.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.819+0000 7fd82aa84740 1 -- start start 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.819+0000 7fd82aa84740 1 Processor -- start 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.819+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39f0d0 0x56300d3a9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.819+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d3a4c30 0x56300d3b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd82aa84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300d2c0f40 con 0x56300d3a4c30 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd82aa84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300d300ac0 con 0x56300d39f0d0 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd82aa84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56300d3a7be0 con 0x56300d39fac0 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd82aa84740 10 client.0 _ll_register_callbacks cb 0x56300d2ffaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39f0d0 0x56300d3a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39f0d0 0x56300d3a9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38270/0 (socket says 172.21.15.116:38270) 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 -- 172.21.15.116:0/4057549832 learned_addr learned my addr 172.21.15.116:0/4057549832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825e00640 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d3a4c30 0x56300d3b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd824a00640 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 -- 172.21.15.116:0/4057549832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 msgr2=0x56300d3a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 -- 172.21.15.116:0/4057549832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d3a4c30 msgr2=0x56300d3b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d3a4c30 0x56300d3b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 -- 172.21.15.116:0/4057549832 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56300d0e17e0 con 0x56300d39f0d0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd825400640 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39f0d0 0x56300d3a9430 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd814009900 tx=0x7fd8140bc5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a58d402c22adc7b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.820+0000 7fd806a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd806a00640 1 -- 172.21.15.116:0/4057549832 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8140bd190 con 0x56300d39f0d0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd806a00640 1 -- 172.21.15.116:0/4057549832 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd814003d00 con 0x56300d39f0d0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd80c0041c0 con 0x56300d39f0d0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd806a00640 1 -- 172.21.15.116:0/4057549832 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8140bca90 con 0x56300d39f0d0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 10 client.27248 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56300d141f70 con 0x56300d39f0d0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd806a00640 1 -- 172.21.15.116:0/4057549832 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd80c0041c0 con 0x56300d39f0d0 2024-03-04T07:48:19.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd806a00640 1 -- 172.21.15.116:0/4057549832 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd8140c4e30 con 0x56300d39f0d0 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 10 client.27248 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 10 client.27248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 20 client.27248 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d3b52e0 0x56300d3b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 4 client.27248 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.821+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56300d311f60 con 0x56300d3b52e0 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.822+0000 7fd824a00640 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d3b52e0 0x56300d3b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.822+0000 7fd824a00640 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d3b52e0 0x56300d3b7730 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43dc010bb2227c07 server_cookie=76852aa2e7517def in_seq=0 out_seq=0 2024-03-04T07:48:19.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.822+0000 7fd806a00640 10 client.27248 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:19.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd806a00640 1 -- 172.21.15.116:0/4057549832 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56300d311f60 con 0x56300d3b52e0 2024-03-04T07:48:19.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd806a00640 10 client.27248 handle_command_reply: tid=0 2024-03-04T07:48:19.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 client.27248 shutdown 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39f0d0 msgr2=0x56300d3a9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39f0d0 0x56300d3a9430 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd814009900 tx=0x7fd8140bc5d0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 shutdown_connections 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d3b52e0 0x56300d3b7730 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d3a4c30 0x56300d3b09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 --2- 172.21.15.116:0/4057549832 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39f0d0 0x56300d3a9430 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd806a00640 0 client.27248 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 >> 172.21.15.116:0/4057549832 conn(0x56300d39a640 msgr2=0x56300d39b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 shutdown_connections 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.823+0000 7fd82aa84740 1 -- 172.21.15.116:0/4057549832 wait complete. 2024-03-04T07:48:19.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.824+0000 7fd82aa84740 20 client.27248 trim_cache size 0 max 16384 2024-03-04T07:48:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 Processor -- start 2024-03-04T07:48:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 -- start start 2024-03-04T07:48:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3a9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201acf50 con 0x56300d39f0d0 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201af530 con 0x56300d3a4c30 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.841+0000 7fd82aa84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201ad550 con 0x56300d39fac0 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38286/0 (socket says 172.21.15.116:38286) 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 -- 172.21.15.116:0/3954646309 learned_addr learned my addr 172.21.15.116:0/3954646309 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825400640 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3a9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd824a00640 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 -- 172.21.15.116:0/3954646309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 msgr2=0x56300d3b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 -- 172.21.15.116:0/3954646309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 msgr2=0x56300d3a9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3a9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 -- 172.21.15.116:0/3954646309 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56300cfd9170 con 0x56300d3a4c30 2024-03-04T07:48:19.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd824a00640 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:19.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825400640 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3a9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:19.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.842+0000 7fd825e00640 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a9580 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd81800fa40 tx=0x7fd81801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba9a0438c0b13013 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:19.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd811c00640 1 -- 172.21.15.116:0/3954646309 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818004b20 con 0x56300d3a4c30 2024-03-04T07:48:19.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd811c00640 1 -- 172.21.15.116:0/3954646309 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd818004cc0 con 0x56300d3a4c30 2024-03-04T07:48:19.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd811c00640 1 -- 172.21.15.116:0/3954646309 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8180058e0 con 0x56300d3a4c30 2024-03-04T07:48:19.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 -- 172.21.15.116:0/3954646309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 msgr2=0x56300d3a9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a9580 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd81800fa40 tx=0x7fd81801a000 comp rx=0 tx=0).stop 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 -- 172.21.15.116:0/3954646309 shutdown_connections 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d3a4c30 0x56300d3a9580 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d39fac0 0x56300d3b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 --2- 172.21.15.116:0/3954646309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3a9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 -- 172.21.15.116:0/3954646309 >> 172.21.15.116:0/3954646309 conn(0x56300d39a640 msgr2=0x56300d3a0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.843+0000 7fd82aa84740 1 -- 172.21.15.116:0/3954646309 shutdown_connections 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.844+0000 7fd82aa84740 1 -- 172.21.15.116:0/3954646309 wait complete. 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.844+0000 7fd82aa84740 1 -- start start 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.844+0000 7fd82aa84740 1 Processor -- start 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.844+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39fac0 0x56300d3aea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd82aa84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d3a4c30 0x56300d3af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd82aa84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201adbb0 con 0x56300d39f0d0 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd82aa84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201aeed0 con 0x56300d39fac0 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd82aa84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201ae210 con 0x56300d3a4c30 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd82aa84740 10 client.0 _ll_register_callbacks cb 0x7fd814004250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49920/0 (socket says 172.21.15.116:49920) 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd824a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39fac0 0x56300d3aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 -- 172.21.15.116:0/2507585275 learned_addr learned my addr 172.21.15.116:0/2507585275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825e00640 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d3a4c30 0x56300d3af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 -- 172.21.15.116:0/2507585275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d3a4c30 msgr2=0x56300d3af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d3a4c30 0x56300d3af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 -- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39fac0 msgr2=0x56300d3aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39fac0 0x56300d3aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 -- 172.21.15.116:0/2507585275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56300d3affd0 con 0x56300d39f0d0 2024-03-04T07:48:19.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd824a00640 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39fac0 0x56300d3aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd825400640 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3ae530 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd8140b9460 tx=0x7fd8140bc470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d97e8b549fe84f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.845+0000 7fd806a00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd806a00640 1 -- 172.21.15.116:0/2507585275 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8140c0be0 con 0x56300d39f0d0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd806a00640 1 -- 172.21.15.116:0/2507585275 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd8140c4da0 con 0x56300d39f0d0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd818004cc0 con 0x56300d39f0d0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd806a00640 1 -- 172.21.15.116:0/2507585275 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8140c4fa0 con 0x56300d39f0d0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 10 client.17124 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56300d4110e0 con 0x56300d39f0d0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd806a00640 1 -- 172.21.15.116:0/2507585275 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd818004cc0 con 0x56300d39f0d0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd806a00640 1 -- 172.21.15.116:0/2507585275 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd8140b8870 con 0x56300d39f0d0 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 10 client.17124 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 10 client.17124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 20 client.17124 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:19.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d411840 0x56300d413c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 4 client.17124 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:48:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.846+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56300d30c2b0 con 0x56300d411840 2024-03-04T07:48:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.847+0000 7fd824a00640 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d411840 0x56300d413c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.847+0000 7fd824a00640 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d411840 0x56300d413c60 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31efd7202a6a45c5 server_cookie=5564cab242dd4d4a in_seq=0 out_seq=0 2024-03-04T07:48:19.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.847+0000 7fd806a00640 10 client.17124 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd806a00640 1 -- 172.21.15.116:0/2507585275 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56300d30c2b0 con 0x56300d411840 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd806a00640 10 client.17124 handle_command_reply: tid=0 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 client.17124 shutdown 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 msgr2=0x56300d3ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3ae530 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd8140b9460 tx=0x7fd8140bc470 comp rx=0 tx=0).stop 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 shutdown_connections 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56300d411840 0x56300d413c60 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56300d3a4c30 0x56300d3af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56300d39fac0 0x56300d3aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 --2- 172.21.15.116:0/2507585275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56300d39f0d0 0x56300d3ae530 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd806a00640 0 client.17124 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 >> 172.21.15.116:0/2507585275 conn(0x56300d39a640 msgr2=0x56300d3a9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 shutdown_connections 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.848+0000 7fd82aa84740 1 -- 172.21.15.116:0/2507585275 wait complete. 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd82aa84740 20 client.17124 trim_cache size 0 max 16384 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "02296187-4cb1-4d0e-9075-cf4e6c1ef1ec", 2024-03-04T07:48:19.853 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:48:19.854 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8080a1d80 msgr2=0x7fd8080a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8080a1d80 0x7fd8080a4240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x56300d3a0320 tx=0x7fd818009500 comp rx=0 tx=0).stop 2024-03-04T07:48:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd820107bf0 msgr2=0x7fd82019df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:19.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd820107bf0 0x7fd82019df00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd814009230 tx=0x7fd81402c5d0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 shutdown_connections 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8201097a0 0x7fd8201a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd820108de0 0x7fd82019e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd8080a1d80 0x7fd8080a4240 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 --2- 172.21.15.116:0/1366035306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd820107bf0 0x7fd82019df00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.849+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 >> 172.21.15.116:0/1366035306 conn(0x7fd820076b90 msgr2=0x7fd82010c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.850+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 shutdown_connections 2024-03-04T07:48:19.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:19.850+0000 7fd827e00640 1 -- 172.21.15.116:0/1366035306 wait complete. 2024-03-04T07:48:19.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:48:19.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:48:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.080+0000 7f49b4600640 1 -- 172.21.15.116:0/1340464914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac1097a0 msgr2=0x7f49ac110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.080+0000 7f49b4600640 1 --2- 172.21.15.116:0/1340464914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac1097a0 0x7f49ac110030 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009ca0 tx=0x7f49a402c100 comp rx=0 tx=0).stop 2024-03-04T07:48:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.081+0000 7f49b4600640 1 -- 172.21.15.116:0/1340464914 shutdown_connections 2024-03-04T07:48:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.081+0000 7f49b4600640 1 --2- 172.21.15.116:0/1340464914 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac1097a0 0x7f49ac110030 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.081+0000 7f49b4600640 1 --2- 172.21.15.116:0/1340464914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49ac108de0 0x7f49ac109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.081+0000 7f49b4600640 1 --2- 172.21.15.116:0/1340464914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac107bf0 0x7f49ac108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.081+0000 7f49b4600640 1 -- 172.21.15.116:0/1340464914 >> 172.21.15.116:0/1340464914 conn(0x7f49ac076b90 msgr2=0x7f49ac078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.081+0000 7f49b4600640 1 -- 172.21.15.116:0/1340464914 shutdown_connections 2024-03-04T07:48:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.081+0000 7f49b4600640 1 -- 172.21.15.116:0/1340464914 wait complete. 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 Processor -- start 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 -- start start 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac107bf0 0x7f49ac19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac108de0 0x7f49ac19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49ac1097a0 0x7f49ac1a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac113850 con 0x7f49ac108de0 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1136d0 con 0x7f49ac107bf0 2024-03-04T07:48:20.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.082+0000 7f49b4600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1139d0 con 0x7f49ac1097a0 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac107bf0 0x7f49ac19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac108de0 0x7f49ac19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac107bf0 0x7f49ac19de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38322/0 (socket says 172.21.15.116:38322) 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 -- 172.21.15.116:0/1407961145 learned_addr learned my addr 172.21.15.116:0/1407961145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b2600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49ac1097a0 0x7f49ac1a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 -- 172.21.15.116:0/1407961145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49ac1097a0 msgr2=0x7f49ac1a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49ac1097a0 0x7f49ac1a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 -- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac108de0 msgr2=0x7f49ac19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac108de0 0x7f49ac19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 -- 172.21.15.116:0/1407961145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac1a5ab0 con 0x7f49ac107bf0 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b2600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49ac1097a0 0x7f49ac1a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49b1c00640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac107bf0 0x7f49ac19de90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f49a000e9c0 tx=0x7f49a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49aa000640 1 -- 172.21.15.116:0/1407961145 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000cd80 con 0x7f49ac107bf0 2024-03-04T07:48:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.083+0000 7f49aa000640 1 -- 172.21.15.116:0/1407961145 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49a00035f0 con 0x7f49ac107bf0 2024-03-04T07:48:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.084+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac1a5da0 con 0x7f49ac107bf0 2024-03-04T07:48:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.084+0000 7f49aa000640 1 -- 172.21.15.116:0/1407961145 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0010690 con 0x7f49ac107bf0 2024-03-04T07:48:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.084+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac1a6210 con 0x7f49ac107bf0 2024-03-04T07:48:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.086+0000 7f49aa000640 1 -- 172.21.15.116:0/1407961145 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f49a00037a0 con 0x7f49ac107bf0 2024-03-04T07:48:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 Processor -- start 2024-03-04T07:48:20.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 -- start start 2024-03-04T07:48:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906db50 0x564d0906df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d0906e540 0x564d09073040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d090736b0 0x564d09075b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d08f9b090 con 0x564d0906db50 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d08fd2d30 con 0x564d0906e540 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b741b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d08f7ba30 con 0x564d090736b0 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906db50 0x564d0906df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.087+0000 7f49b1c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d0906e540 0x564d09073040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b2600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906db50 0x564d0906df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49944/0 (socket says 172.21.15.116:49944) 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b2600640 1 -- 172.21.15.116:0/3614122924 learned_addr learned my addr 172.21.15.116:0/3614122924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1200640 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d090736b0 0x564d09075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49aa000640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49940a1d80 0x7f49940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1c00640 1 -- 172.21.15.116:0/3614122924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d090736b0 msgr2=0x564d09075b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1c00640 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d090736b0 0x564d09075b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1c00640 1 -- 172.21.15.116:0/3614122924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906db50 msgr2=0x564d0906df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1c00640 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906db50 0x564d0906df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49aa000640 1 -- 172.21.15.116:0/1407961145 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f49a00ce220 con 0x7f49ac107bf0 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1c00640 1 -- 172.21.15.116:0/3614122924 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d08db17d0 con 0x564d0906e540 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1200640 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d090736b0 0x564d09075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b2600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49940a1d80 0x7f49940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.088+0000 7f49b1c00640 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d0906e540 0x564d09073040 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f49a00cbc80 tx=0x7f49a0003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3f380f3df638c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f499aa00640 1 -- 172.21.15.116:0/3614122924 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a009a870 con 0x564d0906e540 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f499aa00640 1 -- 172.21.15.116:0/3614122924 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49a009b610 con 0x564d0906e540 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f499aa00640 1 -- 172.21.15.116:0/3614122924 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a00d1d50 con 0x564d0906e540 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 -- 172.21.15.116:0/3614122924 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d0906e540 msgr2=0x564d09073040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d0906e540 0x564d09073040 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f49a00cbc80 tx=0x7f49a0003a90 comp rx=0 tx=0).stop 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b2600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49940a1d80 0x7f49940a4240 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f49a4032420 tx=0x7f49a402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 -- 172.21.15.116:0/3614122924 shutdown_connections 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d090736b0 0x564d09075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d0906e540 0x564d09073040 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 --2- 172.21.15.116:0/3614122924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906db50 0x564d0906df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 -- 172.21.15.116:0/3614122924 >> 172.21.15.116:0/3614122924 conn(0x564d090690c0 msgr2=0x564d090694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 -- 172.21.15.116:0/3614122924 shutdown_connections 2024-03-04T07:48:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.089+0000 7f49b741b740 1 -- 172.21.15.116:0/3614122924 wait complete. 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 -- start start 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 Processor -- start 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d09077e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d090783a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d08f710c0 con 0x564d0906e540 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d08fcedd0 con 0x564d090736b0 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.090+0000 7f49b741b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d090765a0 con 0x564d0906db50 2024-03-04T07:48:20.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b741b740 10 client.0 _ll_register_callbacks cb 0x564d08f8e860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d090783a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38350/0 (socket says 172.21.15.116:38350) 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d090783a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49948/0 (socket says 172.21.15.116:49948) 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 -- 172.21.15.116:0/230630637 learned_addr learned my addr 172.21.15.116:0/230630637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b1c00640 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d09077e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 -- 172.21.15.116:0/230630637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 msgr2=0x564d09077e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d09077e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 -- 172.21.15.116:0/230630637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 msgr2=0x564d090783a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d090783a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 -- 172.21.15.116:0/230630637 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d08ce73f0 con 0x564d090736b0 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f49b2600640 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907f370 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f49a40324c0 tx=0x7f49a4005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6cd3bb0bce14ace server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:20.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.091+0000 7f498f400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f498f400640 1 -- 172.21.15.116:0/230630637 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4051060 con 0x564d090736b0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f498f400640 1 -- 172.21.15.116:0/230630637 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f49a4034b90 con 0x564d090736b0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49a009b610 con 0x564d090736b0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f498f400640 1 -- 172.21.15.116:0/230630637 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49a4039040 con 0x564d090736b0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f49b741b740 10 client.27266 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564d08ca70f0 con 0x564d090736b0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f498f400640 1 -- 172.21.15.116:0/230630637 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49a009b610 con 0x564d090736b0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f498f400640 1 -- 172.21.15.116:0/230630637 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f49a403f040 con 0x564d090736b0 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f49b741b740 10 client.27266 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.092+0000 7f49b741b740 10 client.27266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.093+0000 7f49b741b740 20 client.27266 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.093+0000 7f49b741b740 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090838b0 0x564d09085ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.093+0000 7f49b741b740 4 client.27266 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:20.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.093+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d08fe0270 con 0x564d090838b0 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.093+0000 7f49b1c00640 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090838b0 0x564d09085ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.093+0000 7f49b1c00640 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090838b0 0x564d09085ca0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fc294e560712f21 server_cookie=96212dd6e00eaab0 in_seq=0 out_seq=0 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.093+0000 7f498f400640 10 client.27266 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f498f400640 1 -- 172.21.15.116:0/230630637 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564d08fe0270 con 0x564d090838b0 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f498f400640 10 client.27266 handle_command_reply: tid=0 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 client.27266 shutdown 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 msgr2=0x564d0907f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907f370 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f49a40324c0 tx=0x7f49a4005ef0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 shutdown_connections 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090838b0 0x564d09085ca0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907f370 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d090783a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 --2- 172.21.15.116:0/230630637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d09077e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f498f400640 0 client.27266 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:20.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.094+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 >> 172.21.15.116:0/230630637 conn(0x564d090690c0 msgr2=0x564d09068360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:20.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.095+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 shutdown_connections 2024-03-04T07:48:20.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.095+0000 7f49b741b740 1 -- 172.21.15.116:0/230630637 wait complete. 2024-03-04T07:48:20.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.095+0000 7f49b741b740 20 client.27266 trim_cache size 0 max 16384 2024-03-04T07:48:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 Processor -- start 2024-03-04T07:48:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 -- start start 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0906fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0906ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1ad150 con 0x564d0906e540 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1af730 con 0x564d090736b0 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.112+0000 7f49b741b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1ad750 con 0x564d0906db50 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0906ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0906ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38362/0 (socket says 172.21.15.116:38362) 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b1c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0906fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 -- 172.21.15.116:0/3101719980 learned_addr learned my addr 172.21.15.116:0/3101719980 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 -- 172.21.15.116:0/3101719980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 msgr2=0x564d0906fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 --2- 172.21.15.116:0/3101719980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0906fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 -- 172.21.15.116:0/3101719980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 msgr2=0x564d0907f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 --2- 172.21.15.116:0/3101719980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 -- 172.21.15.116:0/3101719980 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d09076250 con 0x564d090736b0 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f49b2600640 1 --2- 172.21.15.116:0/3101719980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0906ca30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009430 tx=0x7f49a40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9def2dc78b93015c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f499a000640 1 -- 172.21.15.116:0/3101719980 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a405b040 con 0x564d090736b0 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f499a000640 1 -- 172.21.15.116:0/3101719980 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49a4045030 con 0x564d090736b0 2024-03-04T07:48:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.113+0000 7f499a000640 1 -- 172.21.15.116:0/3101719980 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4051040 con 0x564d090736b0 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 -- 172.21.15.116:0/3101719980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 msgr2=0x564d0906ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 --2- 172.21.15.116:0/3101719980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0906ca30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009430 tx=0x7f49a40095c0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 -- 172.21.15.116:0/3101719980 shutdown_connections 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 --2- 172.21.15.116:0/3101719980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0906ca30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 --2- 172.21.15.116:0/3101719980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 --2- 172.21.15.116:0/3101719980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0906fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 -- 172.21.15.116:0/3101719980 >> 172.21.15.116:0/3101719980 conn(0x564d090690c0 msgr2=0x564d0906c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 -- 172.21.15.116:0/3101719980 shutdown_connections 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.114+0000 7f49b741b740 1 -- 172.21.15.116:0/3101719980 wait complete. 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 -- start start 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 Processor -- start 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0907ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1addb0 con 0x564d0906e540 2024-03-04T07:48:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1af0d0 con 0x564d090736b0 2024-03-04T07:48:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1ae410 con 0x564d0906db50 2024-03-04T07:48:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b741b740 10 client.0 _ll_register_callbacks cb 0x7f49a4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b2600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38364/0 (socket says 172.21.15.116:38364) 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b2600640 1 -- 172.21.15.116:0/720397820 learned_addr learned my addr 172.21.15.116:0/720397820 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.115+0000 7f49b1c00640 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0907ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b2600640 1 -- 172.21.15.116:0/720397820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 msgr2=0x564d0907ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b2600640 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0907ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b2600640 1 -- 172.21.15.116:0/720397820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 msgr2=0x564d0907d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b2600640 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b2600640 1 -- 172.21.15.116:0/720397820 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d0907dfa0 con 0x564d090736b0 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b1200640 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b2600640 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907da60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f49a40324c0 tx=0x7f49a4040d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2c64661237a232 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f498f400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f498f400640 1 -- 172.21.15.116:0/720397820 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4051070 con 0x564d090736b0 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f498f400640 1 -- 172.21.15.116:0/720397820 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f49a4004730 con 0x564d090736b0 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.116+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49a4045030 con 0x564d090736b0 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f498f400640 1 -- 172.21.15.116:0/720397820 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49a406f050 con 0x564d090736b0 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 10 client.27272 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564d090dfd80 con 0x564d090736b0 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f498f400640 1 -- 172.21.15.116:0/720397820 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49a4045030 con 0x564d090736b0 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f498f400640 1 -- 172.21.15.116:0/720397820 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f49a403f070 con 0x564d090736b0 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 10 client.27272 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 10 client.27272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 20 client.27272 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090e0650 0x564d090e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 4 client.27272 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:48:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d08fda5c0 con 0x564d090e0650 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.117+0000 7f49b1c00640 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090e0650 0x564d090e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f498f400640 10 client.27272 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b1c00640 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090e0650 0x564d090e2a70 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0913038b5a78bb9 server_cookie=f5907cbe6b9d5897 in_seq=0 out_seq=0 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f498f400640 1 -- 172.21.15.116:0/720397820 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564d08fda5c0 con 0x564d090e0650 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f498f400640 10 client.27272 handle_command_reply: tid=0 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 client.27272 shutdown 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 msgr2=0x564d0907da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907da60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f49a40324c0 tx=0x7f49a4040d40 comp rx=0 tx=0).stop 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 shutdown_connections 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x564d090e0650 0x564d090e2a70 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564d090736b0 0x564d0907da60 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d0906e540 0x564d0907d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f498f400640 0 client.27272 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 --2- 172.21.15.116:0/720397820 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d0906db50 0x564d0907ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 >> 172.21.15.116:0/720397820 conn(0x564d090690c0 msgr2=0x564d0906cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 shutdown_connections 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 1 -- 172.21.15.116:0/720397820 wait complete. 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.118+0000 7f49b741b740 20 client.27272 trim_cache size 0 max 16384 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:20.127 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout: "02296187-4cb1-4d0e-9075-cf4e6c1ef1ec": { 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "02296187-4cb1-4d0e-9075-cf4e6c1ef1ec", 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:48:20.128 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49940a1d80 msgr2=0x7f49940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49940a1d80 0x7f49940a4240 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f49a4032420 tx=0x7f49a402e040 comp rx=0 tx=0).stop 2024-03-04T07:48:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac107bf0 msgr2=0x7f49ac19de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac107bf0 0x7f49ac19de90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f49a000e9c0 tx=0x7f49a000ee80 comp rx=0 tx=0).stop 2024-03-04T07:48:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 shutdown_connections 2024-03-04T07:48:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f49940a1d80 0x7f49940a4240 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49ac1097a0 0x7f49ac1a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac108de0 0x7f49ac19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 --2- 172.21.15.116:0/1407961145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f49ac107bf0 0x7f49ac19de90 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 >> 172.21.15.116:0/1407961145 conn(0x7f49ac076b90 msgr2=0x7f49ac10c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 shutdown_connections 2024-03-04T07:48:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:20.119+0000 7f49b4600640 1 -- 172.21.15.116:0/1407961145 wait complete. 2024-03-04T07:48:20.136 INFO:tasks.cephfs.filesystem:scrub status for tag:02296187-4cb1-4d0e-9075-cf4e6c1ef1ec - {'path': '/', 'tag': '02296187-4cb1-4d0e-9075-cf4e6c1ef1ec', 'options': 'recursive,force'} 2024-03-04T07:48:20.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:19 smithi116 ceph-mon[32150]: pgmap v628: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:20.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:19 smithi152 ceph-mon[37154]: pgmap v628: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:20.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:19 smithi133 ceph-mon[36505]: pgmap v628: 97 pgs: 97 active+clean; 1.6 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:20 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:48:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:20 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:20 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:48:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:20 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:20 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:48:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:20 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:20 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:48:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:20 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:20 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:48:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:20 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:20 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:48:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:20 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:22.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:21 smithi116 ceph-mon[32150]: pgmap v629: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:48:22.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:21 smithi152 ceph-mon[37154]: pgmap v629: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:48:22.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:21 smithi133 ceph-mon[36505]: pgmap v629: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:48:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:22 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:48:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:22 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:48:23.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:22 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:48:24.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:23 smithi116 ceph-mon[32150]: pgmap v630: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:48:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:23 smithi152 ceph-mon[37154]: pgmap v630: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:48:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:23 smithi133 ceph-mon[36505]: pgmap v630: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:48:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 -- 172.21.15.116:0/3246710594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 msgr2=0x7fc85c077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 --2- 172.21.15.116:0/3246710594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c077630 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009900 tx=0x7fc84c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 -- 172.21.15.116:0/3246710594 shutdown_connections 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 --2- 172.21.15.116:0/3246710594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c077b70 0x7fc85c078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 --2- 172.21.15.116:0/3246710594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c077630 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 --2- 172.21.15.116:0/3246710594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc85c075f90 0x7fc85c0763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 -- 172.21.15.116:0/3246710594 >> 172.21.15.116:0/3246710594 conn(0x7fc85c0fed90 msgr2=0x7fc85c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 -- 172.21.15.116:0/3246710594 shutdown_connections 2024-03-04T07:48:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.834+0000 7fc863600640 1 -- 172.21.15.116:0/3246710594 wait complete. 2024-03-04T07:48:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.835+0000 7fc863600640 1 Processor -- start 2024-03-04T07:48:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.835+0000 7fc863600640 1 -- start start 2024-03-04T07:48:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc863600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c075f90 0x7fc85c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc863600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c19e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc863600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc85c077b70 0x7fc85c1a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc863600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c1138f0 con 0x7fc85c075f90 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc863600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c113770 con 0x7fc85c0771d0 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc863600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c113a70 con 0x7fc85c077b70 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc860c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c075f90 0x7fc85c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c19e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c19e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58562/0 (socket says 172.21.15.116:58562) 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc860c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c075f90 0x7fc85c19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51798/0 (socket says 172.21.15.116:51798) 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 -- 172.21.15.116:0/304382557 learned_addr learned my addr 172.21.15.116:0/304382557 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc861600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc85c077b70 0x7fc85c1a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 -- 172.21.15.116:0/304382557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc85c077b70 msgr2=0x7fc85c1a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc85c077b70 0x7fc85c1a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 -- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c075f90 msgr2=0x7fc85c19de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c075f90 0x7fc85c19de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.836+0000 7fc85be00640 1 -- 172.21.15.116:0/304382557 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c1a5a70 con 0x7fc85c0771d0 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.837+0000 7fc85be00640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c19e390 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc84c000c00 tx=0x7fc84c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.837+0000 7fc859600640 1 -- 172.21.15.116:0/304382557 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c043020 con 0x7fc85c0771d0 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.837+0000 7fc859600640 1 -- 172.21.15.116:0/304382557 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc84c0054e0 con 0x7fc85c0771d0 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.837+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c1a5d00 con 0x7fc85c0771d0 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.837+0000 7fc859600640 1 -- 172.21.15.116:0/304382557 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c003c40 con 0x7fc85c0771d0 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.837+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c1a6240 con 0x7fc85c0771d0 2024-03-04T07:48:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.840+0000 7fc859600640 1 -- 172.21.15.116:0/304382557 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fc84c046020 con 0x7fc85c0771d0 2024-03-04T07:48:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.840+0000 7fc86642e740 1 Processor -- start 2024-03-04T07:48:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.840+0000 7fc86642e740 1 -- start start 2024-03-04T07:48:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc86642e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d933c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc86642e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d938d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc86642e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d93b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc86642e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55795d899370 con 0x55795d9393b0 2024-03-04T07:48:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc86642e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55795d842550 con 0x55795d934240 2024-03-04T07:48:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc86642e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55795d895410 con 0x55795d933850 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc85be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d93b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc860c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d938d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc861600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d933c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc860c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d938d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58574/0 (socket says 172.21.15.116:58574) 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc860c00640 1 -- 172.21.15.116:0/3966605223 learned_addr learned my addr 172.21.15.116:0/3966605223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc860c00640 1 -- 172.21.15.116:0/3966605223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 msgr2=0x55795d933c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc860c00640 1 --2- 172.21.15.116:0/3966605223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d933c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc859600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc8400a1ea0 0x7fc8400a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc860c00640 1 -- 172.21.15.116:0/3966605223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 msgr2=0x55795d93b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.841+0000 7fc860c00640 1 --2- 172.21.15.116:0/3966605223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d93b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.842+0000 7fc860c00640 1 -- 172.21.15.116:0/3966605223 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55795d8563c0 con 0x55795d934240 2024-03-04T07:48:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.842+0000 7fc859600640 1 -- 172.21.15.116:0/304382557 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc84c0efda0 con 0x7fc85c0771d0 2024-03-04T07:48:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.842+0000 7fc861600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc8400a1ea0 0x7fc8400a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.842+0000 7fc860c00640 1 --2- 172.21.15.116:0/3966605223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d938d40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc848009d20 tx=0x7fc848011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e62debe509001d70 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.842+0000 7fc851c00640 1 -- 172.21.15.116:0/3966605223 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84801e070 con 0x55795d934240 2024-03-04T07:48:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.842+0000 7fc851c00640 1 -- 172.21.15.116:0/3966605223 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc848019b10 con 0x55795d934240 2024-03-04T07:48:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.842+0000 7fc851c00640 1 -- 172.21.15.116:0/3966605223 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84800ebf0 con 0x55795d934240 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 -- 172.21.15.116:0/3966605223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 msgr2=0x55795d938d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc861600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc8400a1ea0 0x7fc8400a4360 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55795d934aa0 tx=0x7fc854009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 --2- 172.21.15.116:0/3966605223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d938d40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc848009d20 tx=0x7fc848011810 comp rx=0 tx=0).stop 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 -- 172.21.15.116:0/3966605223 shutdown_connections 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 --2- 172.21.15.116:0/3966605223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d93b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 --2- 172.21.15.116:0/3966605223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d938d40 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 --2- 172.21.15.116:0/3966605223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d933c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 -- 172.21.15.116:0/3966605223 >> 172.21.15.116:0/3966605223 conn(0x55795d92edc0 msgr2=0x55795d92f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 -- 172.21.15.116:0/3966605223 shutdown_connections 2024-03-04T07:48:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.843+0000 7fc86642e740 1 -- 172.21.15.116:0/3966605223 wait complete. 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 -- start start 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 Processor -- start 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d93dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d93e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d945140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55795d93c360 con 0x55795d9393b0 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55795d8559b0 con 0x55795d934240 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55795d93c4e0 con 0x55795d933850 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc86642e740 10 client.0 _ll_register_callbacks cb 0x55795d891340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc85be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d93e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc860c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d93dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc861600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d945140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc85be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d93e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58584/0 (socket says 172.21.15.116:58584) 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc85be00640 1 -- 172.21.15.116:0/501991662 learned_addr learned my addr 172.21.15.116:0/501991662 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.844+0000 7fc860c00640 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d93dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:56002/0 (socket says 172.21.15.116:56002) 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc85be00640 1 -- 172.21.15.116:0/501991662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 msgr2=0x55795d93dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc85be00640 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d93dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc85be00640 1 -- 172.21.15.116:0/501991662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 msgr2=0x55795d945140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc85be00640 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d945140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc85be00640 1 -- 172.21.15.116:0/501991662 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55795d5ae210 con 0x55795d934240 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc85be00640 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d93e100 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc84c006010 tx=0x7fc84c009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3552aefab9a60fe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc846000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc846000640 1 -- 172.21.15.116:0/501991662 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0bc3f0 con 0x55795d934240 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc846000640 1 -- 172.21.15.116:0/501991662 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc84c0ed970 con 0x55795d934240 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.845+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc848019b10 con 0x55795d934240 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc846000640 1 -- 172.21.15.116:0/501991662 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc84c004250 con 0x55795d934240 2024-03-04T07:48:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 10 client.27284 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55795d67a620 con 0x55795d934240 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc846000640 1 -- 172.21.15.116:0/501991662 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc848019b10 con 0x55795d934240 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc846000640 1 -- 172.21.15.116:0/501991662 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc84c0bccf0 con 0x55795d934240 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 10 client.27284 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 10 client.27284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 20 client.27284 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9499c0 0x55795d94bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 4 client.27284 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55795d8a68b0 con 0x55795d9499c0 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.846+0000 7fc860c00640 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9499c0 0x55795d94bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.847+0000 7fc860c00640 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9499c0 0x55795d94bdb0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eab5f5fef5a0fa3 server_cookie=64f62fe229ac2d22 in_seq=0 out_seq=0 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.847+0000 7fc846000640 10 client.27284 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc846000640 1 -- 172.21.15.116:0/501991662 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55795d8a68b0 con 0x55795d9499c0 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc846000640 10 client.27284 handle_command_reply: tid=0 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 client.27284 shutdown 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 msgr2=0x55795d93e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d93e100 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc84c006010 tx=0x7fc84c009620 comp rx=0 tx=0).stop 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 shutdown_connections 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9499c0 0x55795d94bdb0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d945140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d93e100 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 --2- 172.21.15.116:0/501991662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d933850 0x55795d93dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc846000640 0 client.27284 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 >> 172.21.15.116:0/501991662 conn(0x55795d92edc0 msgr2=0x55795d92dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 shutdown_connections 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 1 -- 172.21.15.116:0/501991662 wait complete. 2024-03-04T07:48:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.848+0000 7fc86642e740 20 client.27284 trim_cache size 0 max 16384 2024-03-04T07:48:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.863+0000 7fc86642e740 1 Processor -- start 2024-03-04T07:48:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.863+0000 7fc86642e740 1 -- start start 2024-03-04T07:48:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.863+0000 7fc86642e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d933850 0x55795d9355c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc86642e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d934240 0x55795d945100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc86642e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d935b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc86642e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c1acf00 con 0x55795d9393b0 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc86642e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c1af4e0 con 0x55795d933850 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc86642e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c1ad500 con 0x55795d934240 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d934240 0x55795d945100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d934240 0x55795d945100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:56006/0 (socket says 172.21.15.116:56006) 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 -- 172.21.15.116:0/3659918415 learned_addr learned my addr 172.21.15.116:0/3659918415 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc860c00640 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d933850 0x55795d9355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc861600640 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d935b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 -- 172.21.15.116:0/3659918415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d933850 msgr2=0x55795d9355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d933850 0x55795d9355c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 -- 172.21.15.116:0/3659918415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 msgr2=0x55795d935b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d935b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 -- 172.21.15.116:0/3659918415 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55795d938840 con 0x55795d934240 2024-03-04T07:48:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc860c00640 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d933850 0x55795d9355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc861600640 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d935b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.864+0000 7fc85be00640 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d934240 0x55795d945100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc84c02ceb0 tx=0x7fc84c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f1eb69c9654367e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc851200640 1 -- 172.21.15.116:0/3659918415 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0bc3f0 con 0x55795d934240 2024-03-04T07:48:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc851200640 1 -- 172.21.15.116:0/3659918415 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc84c0035f0 con 0x55795d934240 2024-03-04T07:48:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc851200640 1 -- 172.21.15.116:0/3659918415 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0bccf0 con 0x55795d934240 2024-03-04T07:48:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 -- 172.21.15.116:0/3659918415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d934240 msgr2=0x55795d945100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d934240 0x55795d945100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc84c02ceb0 tx=0x7fc84c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 -- 172.21.15.116:0/3659918415 shutdown_connections 2024-03-04T07:48:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d9393b0 0x55795d935b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d934240 0x55795d945100 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 --2- 172.21.15.116:0/3659918415 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d933850 0x55795d9355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 -- 172.21.15.116:0/3659918415 >> 172.21.15.116:0/3659918415 conn(0x55795d92edc0 msgr2=0x55795d93dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 -- 172.21.15.116:0/3659918415 shutdown_connections 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.865+0000 7fc86642e740 1 -- 172.21.15.116:0/3659918415 wait complete. 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 -- start start 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 Processor -- start 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d933850 0x55795d942ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d942ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d9393b0 0x55795d943fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c1adb60 con 0x55795d933850 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c1aee80 con 0x55795d934240 2024-03-04T07:48:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c1ae1c0 con 0x55795d9393b0 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.866+0000 7fc86642e740 10 client.0 _ll_register_callbacks cb 0x7fc84c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc860c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d933850 0x55795d942ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d942ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d942ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58598/0 (socket says 172.21.15.116:58598) 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 -- 172.21.15.116:0/1899176717 learned_addr learned my addr 172.21.15.116:0/1899176717 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc861600640 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d9393b0 0x55795d943fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 -- 172.21.15.116:0/1899176717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d9393b0 msgr2=0x55795d943fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d9393b0 0x55795d943fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 -- 172.21.15.116:0/1899176717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d933850 msgr2=0x55795d942ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d933850 0x55795d942ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 -- 172.21.15.116:0/1899176717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55795d9445a0 con 0x55795d934240 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc860c00640 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d933850 0x55795d942ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc85be00640 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d942ff0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc84c02c170 tx=0x7fc84c0bafa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a68b996e13e05bd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:26.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.867+0000 7fc846000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc846000640 1 -- 172.21.15.116:0/1899176717 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0a1070 con 0x55795d934240 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc846000640 1 -- 172.21.15.116:0/1899176717 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc84c0c4da0 con 0x55795d934240 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55795d8a0c00 con 0x55795d934240 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc846000640 1 -- 172.21.15.116:0/1899176717 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc84c0bc3f0 con 0x55795d934240 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 10 client.27290 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55795d9445a0 con 0x55795d934240 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc846000640 1 -- 172.21.15.116:0/1899176717 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55795d8a0c00 con 0x55795d934240 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc846000640 1 -- 172.21.15.116:0/1899176717 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc84c0bbf70 con 0x55795d934240 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 10 client.27290 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 10 client.27290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 20 client.27290 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9a5960 0x55795d9a7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 4 client.27290 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:48:26.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55795d81f5d0 con 0x55795d9a5960 2024-03-04T07:48:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.868+0000 7fc860c00640 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9a5960 0x55795d9a7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.869+0000 7fc860c00640 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9a5960 0x55795d9a7d80 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2af6176459548c4 server_cookie=f57d22e37f0a4e35 in_seq=0 out_seq=0 2024-03-04T07:48:26.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.869+0000 7fc846000640 10 client.27290 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc846000640 1 -- 172.21.15.116:0/1899176717 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55795d81f5d0 con 0x55795d9a5960 2024-03-04T07:48:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc846000640 10 client.27290 handle_command_reply: tid=0 2024-03-04T07:48:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 client.27290 shutdown 2024-03-04T07:48:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 msgr2=0x55795d942ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d942ff0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc84c02c170 tx=0x7fc84c0bafa0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 shutdown_connections 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55795d9a5960 0x55795d9a7d80 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55795d9393b0 0x55795d943fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc846000640 0 client.27290 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55795d934240 0x55795d942ff0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 --2- 172.21.15.116:0/1899176717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55795d933850 0x55795d942ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 >> 172.21.15.116:0/1899176717 conn(0x55795d92edc0 msgr2=0x55795d9328b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 shutdown_connections 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.884+0000 7fc86642e740 1 -- 172.21.15.116:0/1899176717 wait complete. 2024-03-04T07:48:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.885+0000 7fc86642e740 20 client.27290 trim_cache size 0 max 16384 2024-03-04T07:48:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:48:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:48:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:48:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:48:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.885+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc8400a1ea0 msgr2=0x7fc8400a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.885+0000 7fc863600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc8400a1ea0 0x7fc8400a4360 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55795d934aa0 tx=0x7fc854009500 comp rx=0 tx=0).stop 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.885+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 msgr2=0x7fc85c19e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.885+0000 7fc863600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c19e390 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc84c000c00 tx=0x7fc84c02c650 comp rx=0 tx=0).stop 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 shutdown_connections 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc8400a1ea0 0x7fc8400a4360 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc85c077b70 0x7fc85c1a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc85c0771d0 0x7fc85c19e390 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 --2- 172.21.15.116:0/304382557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c075f90 0x7fc85c19de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 >> 172.21.15.116:0/304382557 conn(0x7fc85c0fed90 msgr2=0x7fc85c100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 shutdown_connections 2024-03-04T07:48:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:26.886+0000 7fc863600640 1 -- 172.21.15.116:0/304382557 wait complete. 2024-03-04T07:48:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:26 smithi133 ceph-mon[36505]: pgmap v631: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-04T07:48:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:26 smithi116 ceph-mon[32150]: pgmap v631: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-04T07:48:27.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:26 smithi152 ceph-mon[37154]: pgmap v631: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-04T07:48:29.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:28 smithi133 ceph-mon[36505]: pgmap v632: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:48:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:28 smithi116 ceph-mon[32150]: pgmap v632: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:48:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:28 smithi152 ceph-mon[37154]: pgmap v632: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:48:31.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:30 smithi133 ceph-mon[36505]: pgmap v633: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:48:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:30 smithi116 ceph-mon[32150]: pgmap v633: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:48:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:30 smithi152 ceph-mon[37154]: pgmap v633: 97 pgs: 97 active+clean; 1.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-03-04T07:48:33.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:32 smithi133 ceph-mon[36505]: pgmap v634: 97 pgs: 97 active+clean; 1.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:48:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:32 smithi116 ceph-mon[32150]: pgmap v634: 97 pgs: 97 active+clean; 1.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:48:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:32 smithi152 ceph-mon[37154]: pgmap v634: 97 pgs: 97 active+clean; 1.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:48:34.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:33 smithi116 ceph-mon[32150]: pgmap v635: 97 pgs: 97 active+clean; 1.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:34.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:33 smithi152 ceph-mon[37154]: pgmap v635: 97 pgs: 97 active+clean; 1.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:34.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:33 smithi133 ceph-mon[36505]: pgmap v635: 97 pgs: 97 active+clean; 1.8 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:37.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:36 smithi133 ceph-mon[36505]: pgmap v636: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:48:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:36 smithi116 ceph-mon[32150]: pgmap v636: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:48:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:36 smithi152 ceph-mon[37154]: pgmap v636: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:48:39.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:38 smithi133 ceph-mon[36505]: pgmap v637: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:38 smithi116 ceph-mon[32150]: pgmap v637: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:38 smithi152 ceph-mon[37154]: pgmap v637: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:48:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 -- 172.21.15.116:0/2466246948 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d4107c30 msgr2=0x7fb7d4108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 --2- 172.21.15.116:0/2466246948 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d4107c30 0x7fb7d4108050 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009860 tx=0x7fb7c402bb70 comp rx=0 tx=0).stop 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 -- 172.21.15.116:0/2466246948 shutdown_connections 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 --2- 172.21.15.116:0/2466246948 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7d41097e0 0x7fb7d4110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 --2- 172.21.15.116:0/2466246948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 0x7fb7d41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 --2- 172.21.15.116:0/2466246948 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d4107c30 0x7fb7d4108050 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 -- 172.21.15.116:0/2466246948 >> 172.21.15.116:0/2466246948 conn(0x7fb7d4076be0 msgr2=0x7fb7d4079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 -- 172.21.15.116:0/2466246948 shutdown_connections 2024-03-04T07:48:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.026+0000 7fb7da000640 1 -- 172.21.15.116:0/2466246948 wait complete. 2024-03-04T07:48:40.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.027+0000 7fb7da000640 1 Processor -- start 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7da000640 1 -- start start 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7da000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7d4107c30 0x7fb7d419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7da000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 0x7fb7d419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7da000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d41097e0 0x7fb7d41a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7da000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4113880 con 0x7fb7d4108e20 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7da000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4113700 con 0x7fb7d41097e0 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7da000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4113a00 con 0x7fb7d4107c30 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 0x7fb7d419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7d3400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7d4107c30 0x7fb7d419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 0x7fb7d419e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43350/0 (socket says 172.21.15.116:43350) 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7d2a00640 1 -- 172.21.15.116:0/429505624 learned_addr learned my addr 172.21.15.116:0/429505624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.028+0000 7fb7d3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d41097e0 0x7fb7d41a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d2a00640 1 -- 172.21.15.116:0/429505624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7d4107c30 msgr2=0x7fb7d419de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7d4107c30 0x7fb7d419de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d2a00640 1 -- 172.21.15.116:0/429505624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d41097e0 msgr2=0x7fb7d41a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d41097e0 0x7fb7d41a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d2a00640 1 -- 172.21.15.116:0/429505624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d41a5a80 con 0x7fb7d4108e20 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d3400640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7d4107c30 0x7fb7d419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d41097e0 0x7fb7d41a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 0x7fb7d419e3a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000e9c0 tx=0x7fb7c000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7cbe00640 1 -- 172.21.15.116:0/429505624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000cde0 con 0x7fb7d4108e20 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7cbe00640 1 -- 172.21.15.116:0/429505624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c00035f0 con 0x7fb7d4108e20 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d41a5d70 con 0x7fb7d4108e20 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.029+0000 7fb7cbe00640 1 -- 172.21.15.116:0/429505624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0010650 con 0x7fb7d4108e20 2024-03-04T07:48:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.030+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d41a6260 con 0x7fb7d4108e20 2024-03-04T07:48:40.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 Processor -- start 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 -- start start 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7cbe00640 1 -- 172.21.15.116:0/429505624 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb7c00037a0 con 0x7fb7d4108e20 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f75305c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f7530ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f7530d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7526b370 con 0x558f75306240 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f75214550 con 0x558f7530b3b0 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f75267410 con 0x558f75305850 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7d3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f7530ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.033+0000 7fb7d3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f7530ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43358/0 (socket says 172.21.15.116:43358) 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3400640 1 -- 172.21.15.116:0/2878907301 learned_addr learned my addr 172.21.15.116:0/2878907301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:40.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f75305c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f7530d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3e00640 1 -- 172.21.15.116:0/2878907301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 msgr2=0x558f7530d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f7530d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3e00640 1 -- 172.21.15.116:0/2878907301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 msgr2=0x558f7530ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f7530ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3e00640 1 -- 172.21.15.116:0/2878907301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f752283c0 con 0x558f75305850 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3400640 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f7530ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.034+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f75305c70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc010690 tx=0x7fb7cc010b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eec73752f6f444f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7cbe00640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7b80a1e20 0x7fb7b80a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7bfe00640 1 -- 172.21.15.116:0/2878907301 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc004b20 con 0x558f75305850 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7bfe00640 1 -- 172.21.15.116:0/2878907301 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc00edb0 con 0x558f75305850 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7bfe00640 1 -- 172.21.15.116:0/2878907301 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc005850 con 0x558f75305850 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7cbe00640 1 -- 172.21.15.116:0/429505624 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb7c00ce030 con 0x7fb7d4108e20 2024-03-04T07:48:40.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7d3400640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7b80a1e20 0x7fb7b80a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2878907301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 msgr2=0x558f75305c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.035+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f75305c70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc010690 tx=0x7fb7cc010b50 comp rx=0 tx=0).stop 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2878907301 shutdown_connections 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f7530d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f7530ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2878907301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f75305c70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2878907301 >> 172.21.15.116:0/2878907301 conn(0x558f75300dc0 msgr2=0x558f753011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7d3400640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7b80a1e20 0x7fb7b80a42e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb7c402c180 tx=0x7fb7c402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2878907301 shutdown_connections 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2878907301 wait complete. 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 -- start start 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.036+0000 7fb7dcdba740 1 Processor -- start 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f7530fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f75310090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f753170d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7530e360 con 0x558f75306240 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f752279b0 con 0x558f7530b3b0 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7530e4e0 con 0x558f75305850 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7dcdba740 10 client.0 _ll_register_callbacks cb 0x558f75263340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f75310090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f75310090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43370/0 (socket says 172.21.15.116:43370) 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7d2a00640 1 -- 172.21.15.116:0/2546239768 learned_addr learned my addr 172.21.15.116:0/2546239768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7d3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f753170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.037+0000 7fb7d3400640 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f7530fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7d3e00640 1 -- 172.21.15.116:0/2546239768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 msgr2=0x558f7530fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f7530fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7d3e00640 1 -- 172.21.15.116:0/2546239768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 msgr2=0x558f75310090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f75310090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7d3e00640 1 -- 172.21.15.116:0/2546239768 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f74f80210 con 0x558f7530b3b0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f75310090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f753170d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e6c0 tx=0x7fb7cc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb9b3570a73a19e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7bcc00640 1 -- 172.21.15.116:0/2546239768 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc02b060 con 0x558f7530b3b0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7bcc00640 1 -- 172.21.15.116:0/2546239768 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb7cc005860 con 0x558f7530b3b0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7cc00edb0 con 0x558f7530b3b0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7bcc00640 1 -- 172.21.15.116:0/2546239768 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7cc004a40 con 0x558f7530b3b0 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7dcdba740 10 client.27302 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.038+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558f7504c620 con 0x558f7530b3b0 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7bcc00640 1 -- 172.21.15.116:0/2546239768 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc00edb0 con 0x558f7530b3b0 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7bcc00640 1 -- 172.21.15.116:0/2546239768 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb7cc030070 con 0x558f7530b3b0 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7dcdba740 10 client.27302 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7dcdba740 10 client.27302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7dcdba740 20 client.27302 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f7531b670 0x558f7531da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7dcdba740 4 client.27302 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f752788b0 con 0x558f7531b670 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.039+0000 7fb7d3400640 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f7531b670 0x558f7531da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.040+0000 7fb7d3400640 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f7531b670 0x558f7531da60 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed0f67316851b9f server_cookie=78944215b8ae204a in_seq=0 out_seq=0 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.040+0000 7fb7bcc00640 10 client.27302 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.040+0000 7fb7bcc00640 1 -- 172.21.15.116:0/2546239768 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558f752788b0 con 0x558f7531b670 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.040+0000 7fb7bcc00640 10 client.27302 handle_command_reply: tid=0 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.040+0000 7fb7dcdba740 1 client.27302 shutdown 2024-03-04T07:48:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 msgr2=0x558f753170d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f753170d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e6c0 tx=0x7fb7cc00ed00 comp rx=0 tx=0).stop 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 shutdown_connections 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f7531b670 0x558f7531da60 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f7530b3b0 0x558f753170d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7bcc00640 0 client.27302 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75306240 0x558f75310090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/2546239768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f75305850 0x558f7530fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 >> 172.21.15.116:0/2546239768 conn(0x558f75300dc0 msgr2=0x558f752fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 shutdown_connections 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 1 -- 172.21.15.116:0/2546239768 wait complete. 2024-03-04T07:48:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.041+0000 7fb7dcdba740 20 client.27302 trim_cache size 0 max 16384 2024-03-04T07:48:40.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 Processor -- start 2024-03-04T07:48:40.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 -- start start 2024-03-04T07:48:40.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f7530ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75317090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75307560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41ad090 con 0x558f75305850 2024-03-04T07:48:40.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41af670 con 0x558f75306240 2024-03-04T07:48:40.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.056+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41ad690 con 0x558f7530b3b0 2024-03-04T07:48:40.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75317090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75317090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:33700/0 (socket says 172.21.15.116:33700) 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 -- 172.21.15.116:0/4190528348 learned_addr learned my addr 172.21.15.116:0/4190528348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d3400640 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f7530ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75307560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 -- 172.21.15.116:0/4190528348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 msgr2=0x558f75307560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75307560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 -- 172.21.15.116:0/4190528348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 msgr2=0x558f7530ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f7530ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 -- 172.21.15.116:0/4190528348 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f75300710 con 0x558f75306240 2024-03-04T07:48:40.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d3400640 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f7530ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75317090 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb7c009ad30 tx=0x7fb7c00cbb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=777aa86ecda73fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7bf400640 1 -- 172.21.15.116:0/4190528348 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c009e020 con 0x558f75306240 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7bf400640 1 -- 172.21.15.116:0/4190528348 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c009b4a0 con 0x558f75306240 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.057+0000 7fb7bf400640 1 -- 172.21.15.116:0/4190528348 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0097410 con 0x558f75306240 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 -- 172.21.15.116:0/4190528348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 msgr2=0x558f75317090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75317090 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb7c009ad30 tx=0x7fb7c00cbb60 comp rx=0 tx=0).stop 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 -- 172.21.15.116:0/4190528348 shutdown_connections 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75307560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75317090 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/4190528348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f7530ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 -- 172.21.15.116:0/4190528348 >> 172.21.15.116:0/4190528348 conn(0x558f75300dc0 msgr2=0x558f753011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 -- 172.21.15.116:0/4190528348 shutdown_connections 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.058+0000 7fb7dcdba740 1 -- 172.21.15.116:0/4190528348 wait complete. 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 -- start start 2024-03-04T07:48:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 Processor -- start 2024-03-04T07:48:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f753146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75314c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75315c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41adcf0 con 0x558f75305850 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41af010 con 0x558f75306240 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d41ae350 con 0x558f7530b3b0 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7dcdba740 10 client.0 _ll_register_callbacks cb 0x7fb7cc016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7d3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f753146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.059+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75314c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75314c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:33704/0 (socket says 172.21.15.116:33704) 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 -- 172.21.15.116:0/1101911979 learned_addr learned my addr 172.21.15.116:0/1101911979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d3e00640 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75315c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 -- 172.21.15.116:0/1101911979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 msgr2=0x558f75315c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75315c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 -- 172.21.15.116:0/1101911979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 msgr2=0x558f753146c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f753146c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 -- 172.21.15.116:0/1101911979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f75316200 con 0x558f75306240 2024-03-04T07:48:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7d2a00640 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75314c00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009230 tx=0x7fb7c009ae30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b42d20b52ce6081 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7bcc00640 1 -- 172.21.15.116:0/1101911979 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c009e030 con 0x558f75306240 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7bcc00640 1 -- 172.21.15.116:0/1101911979 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb7c00d00f0 con 0x558f75306240 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7c009b4a0 con 0x558f75306240 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.060+0000 7fb7bcc00640 1 -- 172.21.15.116:0/1101911979 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7c009eef0 con 0x558f75306240 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 10 client.27308 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558f75376df0 con 0x558f75306240 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7bcc00640 1 -- 172.21.15.116:0/1101911979 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c009b4a0 con 0x558f75306240 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7bcc00640 1 -- 172.21.15.116:0/1101911979 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb7c00b3ad0 con 0x558f75306240 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 10 client.27308 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 10 client.27308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 20 client.27308 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f75377570 0x558f75379a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 4 client.27308 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:48:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558f75272c00 con 0x558f75377570 2024-03-04T07:48:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.061+0000 7fb7d3400640 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f75377570 0x558f75379a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.062+0000 7fb7d3400640 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f75377570 0x558f75379a10 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=179bd316900fe1df server_cookie=dcb4351b03479dae in_seq=0 out_seq=0 2024-03-04T07:48:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.062+0000 7fb7bcc00640 10 client.27308 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7bcc00640 1 -- 172.21.15.116:0/1101911979 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558f75272c00 con 0x558f75377570 2024-03-04T07:48:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7bcc00640 10 client.27308 handle_command_reply: tid=0 2024-03-04T07:48:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 client.27308 shutdown 2024-03-04T07:48:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 msgr2=0x558f75314c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75314c00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009230 tx=0x7fb7c009ae30 comp rx=0 tx=0).stop 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 shutdown_connections 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558f75377570 0x558f75379a10 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f7530b3b0 0x558f75315c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558f75306240 0x558f75314c00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 --2- 172.21.15.116:0/1101911979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558f75305850 0x558f753146c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7bcc00640 0 client.27308 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 >> 172.21.15.116:0/1101911979 conn(0x558f75300dc0 msgr2=0x558f7530ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 shutdown_connections 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.076+0000 7fb7dcdba740 1 -- 172.21.15.116:0/1101911979 wait complete. 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7dcdba740 20 client.27308 trim_cache size 0 max 16384 2024-03-04T07:48:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:48:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:48:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:48:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:48:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7b80a1e20 msgr2=0x7fb7b80a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7b80a1e20 0x7fb7b80a42e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb7c402c180 tx=0x7fb7c402e040 comp rx=0 tx=0).stop 2024-03-04T07:48:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 msgr2=0x7fb7d419e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 0x7fb7d419e3a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000e9c0 tx=0x7fb7c000ee80 comp rx=0 tx=0).stop 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 shutdown_connections 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7d41097e0 0x7fb7d41a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7d4108e20 0x7fb7d419e3a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb7b80a1e20 0x7fb7b80a42e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 --2- 172.21.15.116:0/429505624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7d4107c30 0x7fb7d419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.077+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 >> 172.21.15.116:0/429505624 conn(0x7fb7d4076be0 msgr2=0x7fb7d410c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.078+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 shutdown_connections 2024-03-04T07:48:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:40.078+0000 7fb7da000640 1 -- 172.21.15.116:0/429505624 wait complete. 2024-03-04T07:48:41.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:40 smithi133 ceph-mon[36505]: pgmap v638: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:40 smithi116 ceph-mon[32150]: pgmap v638: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:40 smithi152 ceph-mon[37154]: pgmap v638: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:43.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:42 smithi133 ceph-mon[36505]: pgmap v639: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:48:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:42 smithi116 ceph-mon[32150]: pgmap v639: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:48:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:42 smithi152 ceph-mon[37154]: pgmap v639: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:48:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:44 smithi133 ceph-mon[36505]: pgmap v640: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:45.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:44 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:48:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:44 smithi116 ceph-mon[32150]: pgmap v640: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:45.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:44 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:48:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:44 smithi152 ceph-mon[37154]: pgmap v640: 97 pgs: 97 active+clean; 1.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:48:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:44 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:48:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:48:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:48:46.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:48:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:48:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:48:46.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:48:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:48:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:48:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:48:47.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:46 smithi133 ceph-mon[36505]: pgmap v641: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:46 smithi116 ceph-mon[32150]: pgmap v641: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:46 smithi152 ceph-mon[37154]: pgmap v641: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:48 smithi116 ceph-mon[32150]: pgmap v642: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:48 smithi152 ceph-mon[37154]: pgmap v642: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:49.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:48 smithi133 ceph-mon[36505]: pgmap v642: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:50.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:48:50.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:48:50.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 -- 172.21.15.116:0/3420662388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 msgr2=0x7fb674103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 --2- 172.21.15.116:0/3420662388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674103c10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb668009900 tx=0x7fb66802bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 -- 172.21.15.116:0/3420662388 shutdown_connections 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 --2- 172.21.15.116:0/3420662388 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6741053a0 0x7fb67410bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 --2- 172.21.15.116:0/3420662388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6741049e0 0x7fb674104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 --2- 172.21.15.116:0/3420662388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674103c10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 -- 172.21.15.116:0/3420662388 >> 172.21.15.116:0/3420662388 conn(0x7fb6740fed80 msgr2=0x7fb6741011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 -- 172.21.15.116:0/3420662388 shutdown_connections 2024-03-04T07:48:50.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.356+0000 7fb67be00640 1 -- 172.21.15.116:0/3420662388 wait complete. 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.357+0000 7fb67be00640 1 Processor -- start 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.357+0000 7fb67be00640 1 -- start start 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb67be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb67be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6741049e0 0x7fb67419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb67be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6741053a0 0x7fb6741a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb67be00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67410f4b0 con 0x7fb6741037f0 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb67be00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67410f330 con 0x7fb6741053a0 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb67be00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67410f630 con 0x7fb6741049e0 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32976/0 (socket says 172.21.15.116:32976) 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 -- 172.21.15.116:0/2682461638 learned_addr learned my addr 172.21.15.116:0/2682461638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679e00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6741053a0 0x7fb6741a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb678a00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6741049e0 0x7fb67419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 -- 172.21.15.116:0/2682461638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6741049e0 msgr2=0x7fb67419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6741049e0 0x7fb67419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 -- 172.21.15.116:0/2682461638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6741053a0 msgr2=0x7fb6741a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6741053a0 0x7fb6741a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679400640 1 -- 172.21.15.116:0/2682461638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6741a1710 con 0x7fb6741037f0 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.358+0000 7fb679e00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6741053a0 0x7fb6741a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.359+0000 7fb679400640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674199af0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb668000c00 tx=0x7fb66802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.359+0000 7fb672000640 1 -- 172.21.15.116:0/2682461638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668043020 con 0x7fb6741037f0 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.359+0000 7fb672000640 1 -- 172.21.15.116:0/2682461638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6680054e0 con 0x7fb6741037f0 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.359+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6741a19a0 con 0x7fb6741037f0 2024-03-04T07:48:50.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.359+0000 7fb672000640 1 -- 172.21.15.116:0/2682461638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668003ce0 con 0x7fb6741037f0 2024-03-04T07:48:50.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.359+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6741a1ec0 con 0x7fb6741037f0 2024-03-04T07:48:50.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.363+0000 7fb67eb9a740 1 Processor -- start 2024-03-04T07:48:50.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.363+0000 7fb67eb9a740 1 -- start start 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb672000640 1 -- 172.21.15.116:0/2682461638 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb668046020 con 0x7fb6741037f0 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817eab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a81710090 con 0x558a817e3540 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a81747d30 con 0x558a817e2b50 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a816f0a30 con 0x558a817e86b0 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60966/0 (socket says 172.21.15.116:60966) 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679e00640 1 -- 172.21.15.116:0/1492351519 learned_addr learned my addr 172.21.15.116:0/1492351519 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:50.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb678a00640 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817eab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679e00640 1 -- 172.21.15.116:0/1492351519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 msgr2=0x558a817eab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679e00640 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817eab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679e00640 1 -- 172.21.15.116:0/1492351519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 msgr2=0x558a817e8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.364+0000 7fb679e00640 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb679e00640 1 -- 172.21.15.116:0/1492351519 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a815267d0 con 0x558a817e2b50 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb679e00640 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e2f70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0117e0 tx=0x7fb66c011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d40854af0dc17191 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb666600640 1 -- 172.21.15.116:0/1492351519 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c011310 con 0x558a817e2b50 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb666600640 1 -- 172.21.15.116:0/1492351519 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c003e50 con 0x558a817e2b50 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb666600640 1 -- 172.21.15.116:0/1492351519 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c004620 con 0x558a817e2b50 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb672000640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb65c0a1d80 0x7fb65c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb672000640 1 -- 172.21.15.116:0/2682461638 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb6680efe30 con 0x7fb6741037f0 2024-03-04T07:48:50.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.365+0000 7fb679400640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb65c0a1d80 0x7fb65c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1492351519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 msgr2=0x558a817e2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e2f70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0117e0 tx=0x7fb66c011ca0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1492351519 shutdown_connections 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1492351519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e2f70 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1492351519 >> 172.21.15.116:0/1492351519 conn(0x558a817de0c0 msgr2=0x558a817de4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1492351519 shutdown_connections 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1492351519 wait complete. 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.366+0000 7fb679400640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb65c0a1d80 0x7fb65c0a4240 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb6680b9530 tx=0x7fb6680f38b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 -- start start 2024-03-04T07:48:50.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 Processor -- start 2024-03-04T07:48:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817ecf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817ed460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a816e60c0 con 0x558a817e3540 2024-03-04T07:48:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a81743dd0 con 0x558a817e86b0 2024-03-04T07:48:50.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a817eb660 con 0x558a817e2b50 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb67eb9a740 10 client.0 _ll_register_callbacks cb 0x558a81703860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.367+0000 7fb678a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817ed460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb678a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817ed460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33002/0 (socket says 172.21.15.116:33002) 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb678a00640 1 -- 172.21.15.116:0/3883961092 learned_addr learned my addr 172.21.15.116:0/3883961092 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817ecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 -- 172.21.15.116:0/3883961092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 msgr2=0x558a817ecf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817ecf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 -- 172.21.15.116:0/3883961092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 msgr2=0x558a817ed460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817ed460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 -- 172.21.15.116:0/3883961092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a8145c3f0 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb678a00640 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817ed460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb679e00640 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f44a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb66c000f80 tx=0x7fb66c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a405c97cdfc9a8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb65aa00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.368+0000 7fb65aa00640 1 -- 172.21.15.116:0/3883961092 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00fbb0 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb65aa00640 1 -- 172.21.15.116:0/3883961092 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb66c004a40 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66c003e50 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb65aa00640 1 -- 172.21.15.116:0/3883961092 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb66c00fd50 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 10 client.27320 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558a8141c0f0 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb65aa00640 1 -- 172.21.15.116:0/3883961092 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c003e50 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb65aa00640 1 -- 172.21.15.116:0/3883961092 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb66c028040 con 0x558a817e86b0 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 10 client.27320 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 10 client.27320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 20 client.27320 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:50.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a817f86b0 0x558a817faaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 4 client.27320 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.369+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a81755270 con 0x558a817f86b0 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.370+0000 7fb678a00640 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a817f86b0 0x558a817faaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.370+0000 7fb65aa00640 10 client.27320 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.370+0000 7fb678a00640 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a817f86b0 0x558a817faaa0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83d5a8c7f1237496 server_cookie=891663a1157687b3 in_seq=0 out_seq=0 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb65aa00640 1 -- 172.21.15.116:0/3883961092 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558a81755270 con 0x558a817f86b0 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb65aa00640 10 client.27320 handle_command_reply: tid=0 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 client.27320 shutdown 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 msgr2=0x558a817f44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f44a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb66c000f80 tx=0x7fb66c005bd0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 shutdown_connections 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a817f86b0 0x558a817faaa0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f44a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb65aa00640 0 client.27320 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817ed460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/3883961092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817ecf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 >> 172.21.15.116:0/3883961092 conn(0x558a817de0c0 msgr2=0x558a817dd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 shutdown_connections 2024-03-04T07:48:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 1 -- 172.21.15.116:0/3883961092 wait complete. 2024-03-04T07:48:50.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.371+0000 7fb67eb9a740 20 client.27320 trim_cache size 0 max 16384 2024-03-04T07:48:50.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 Processor -- start 2024-03-04T07:48:50.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 -- start start 2024-03-04T07:48:50.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817f4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a8cc0 con 0x558a817e3540 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741ab2a0 con 0x558a817e2b50 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a92c0 con 0x558a817e86b0 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb679e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.389+0000 7fb679e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60986/0 (socket says 172.21.15.116:60986) 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 -- 172.21.15.116:0/961311744 learned_addr learned my addr 172.21.15.116:0/961311744 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb678a00640 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 -- 172.21.15.116:0/961311744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 msgr2=0x558a817f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 -- 172.21.15.116:0/961311744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 msgr2=0x558a817e4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 -- 172.21.15.116:0/961311744 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a817ed7b0 con 0x558a817e2b50 2024-03-04T07:48:50.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb679e00640 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e4a00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb66c005d70 tx=0x7fb66c005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c1f548809515bfe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb665c00640 1 -- 172.21.15.116:0/961311744 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c03c070 con 0x558a817e2b50 2024-03-04T07:48:50.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb665c00640 1 -- 172.21.15.116:0/961311744 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c01abe0 con 0x558a817e2b50 2024-03-04T07:48:50.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.390+0000 7fb665c00640 1 -- 172.21.15.116:0/961311744 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c005410 con 0x558a817e2b50 2024-03-04T07:48:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 -- 172.21.15.116:0/961311744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 msgr2=0x558a817e4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e4a00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb66c005d70 tx=0x7fb66c005f10 comp rx=0 tx=0).stop 2024-03-04T07:48:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 -- 172.21.15.116:0/961311744 shutdown_connections 2024-03-04T07:48:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e86b0 0x558a817f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/961311744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e2b50 0x558a817e4a00 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 -- 172.21.15.116:0/961311744 >> 172.21.15.116:0/961311744 conn(0x558a817de0c0 msgr2=0x558a817eba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 -- 172.21.15.116:0/961311744 shutdown_connections 2024-03-04T07:48:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.391+0000 7fb67eb9a740 1 -- 172.21.15.116:0/961311744 wait complete. 2024-03-04T07:48:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 -- start start 2024-03-04T07:48:50.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 Processor -- start 2024-03-04T07:48:50.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817f2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817f2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a9920 con 0x558a817e3540 2024-03-04T07:48:50.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741aac40 con 0x558a817e86b0 2024-03-04T07:48:50.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a9f80 con 0x558a817e2b50 2024-03-04T07:48:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb67eb9a740 10 client.0 _ll_register_callbacks cb 0x7fb66c0114b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb678a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb678a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33006/0 (socket says 172.21.15.116:33006) 2024-03-04T07:48:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb678a00640 1 -- 172.21.15.116:0/1119606422 learned_addr learned my addr 172.21.15.116:0/1119606422 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:50.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.392+0000 7fb679e00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb679e00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb678a00640 1 -- 172.21.15.116:0/1119606422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 msgr2=0x558a817f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb678a00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb678a00640 1 -- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 msgr2=0x558a817f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:50.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb678a00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb678a00640 1 -- 172.21.15.116:0/1119606422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a817f3b90 con 0x558a817e3540 2024-03-04T07:48:50.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb679e00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:50.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb678a00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817f2590 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x558a817fb2b0 tx=0x7fb660020bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa10f76bd78cac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb65aa00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:50.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb65aa00640 1 -- 172.21.15.116:0/1119606422 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6600202b0 con 0x558a817e3540 2024-03-04T07:48:50.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb65aa00640 1 -- 172.21.15.116:0/1119606422 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb660003a40 con 0x558a817e3540 2024-03-04T07:48:50.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66c01abe0 con 0x558a817e3540 2024-03-04T07:48:50.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.393+0000 7fb65aa00640 1 -- 172.21.15.116:0/1119606422 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb66002f050 con 0x558a817e3540 2024-03-04T07:48:50.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 10 client.17196 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:50.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558a81854450 con 0x558a817e3540 2024-03-04T07:48:50.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb65aa00640 1 -- 172.21.15.116:0/1119606422 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c01abe0 con 0x558a817e3540 2024-03-04T07:48:50.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb65aa00640 1 -- 172.21.15.116:0/1119606422 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb6600048c0 con 0x558a817e3540 2024-03-04T07:48:50.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 10 client.17196 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:50.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:50.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 20 client.17196 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:50.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a81854e30 0x558a81857250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 4 client.17196 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:48:50.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558a8174f5c0 con 0x558a81854e30 2024-03-04T07:48:50.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.394+0000 7fb679e00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a81854e30 0x558a81857250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb679e00640 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a81854e30 0x558a81857250 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b861bf45513f920 server_cookie=1119fe4cbea53c74 in_seq=0 out_seq=0 2024-03-04T07:48:50.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb65aa00640 10 client.17196 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:50.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb65aa00640 1 -- 172.21.15.116:0/1119606422 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558a8174f5c0 con 0x558a81854e30 2024-03-04T07:48:50.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb65aa00640 10 client.17196 handle_command_reply: tid=0 2024-03-04T07:48:50.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 client.17196 shutdown 2024-03-04T07:48:50.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 msgr2=0x558a817f2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817f2590 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x558a817fb2b0 tx=0x7fb660020bb0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 shutdown_connections 2024-03-04T07:48:50.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x558a81854e30 0x558a81857250 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558a817e86b0 0x558a817f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a817e3540 0x558a817f2590 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb65aa00640 0 client.17196 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 --2- 172.21.15.116:0/1119606422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558a817e2b50 0x558a817f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 >> 172.21.15.116:0/1119606422 conn(0x558a817de0c0 msgr2=0x558a817e4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 shutdown_connections 2024-03-04T07:48:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 1 -- 172.21.15.116:0/1119606422 wait complete. 2024-03-04T07:48:50.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.395+0000 7fb67eb9a740 20 client.17196 trim_cache size 0 max 16384 2024-03-04T07:48:50.412 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:48:50.412 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:48:50.413 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:50.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb65c0a1d80 msgr2=0x7fb65c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb65c0a1d80 0x7fb65c0a4240 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb6680b9530 tx=0x7fb6680f38b0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 msgr2=0x7fb674199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674199af0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb668000c00 tx=0x7fb66802c6c0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 shutdown_connections 2024-03-04T07:48:50.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6741053a0 0x7fb6741a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb65c0a1d80 0x7fb65c0a4240 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6741049e0 0x7fb67419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 --2- 172.21.15.116:0/2682461638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6741037f0 0x7fb674199af0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 >> 172.21.15.116:0/2682461638 conn(0x7fb6740fed80 msgr2=0x7fb674107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 shutdown_connections 2024-03-04T07:48:50.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.396+0000 7fb67be00640 1 -- 172.21.15.116:0/2682461638 wait complete. 2024-03-04T07:48:50.417 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:48:50.418 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:48:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 -- 172.21.15.116:0/1032836170 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea01053c0 msgr2=0x7fcea010bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 --2- 172.21.15.116:0/1032836170 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea01053c0 0x7fcea010bc50 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fce98009ca0 tx=0x7fce9802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 -- 172.21.15.116:0/1032836170 shutdown_connections 2024-03-04T07:48:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 --2- 172.21.15.116:0/1032836170 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea01053c0 0x7fcea010bc50 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 --2- 172.21.15.116:0/1032836170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea0104a00 0x7fcea0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 --2- 172.21.15.116:0/1032836170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcea0103810 0x7fcea0103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 -- 172.21.15.116:0/1032836170 >> 172.21.15.116:0/1032836170 conn(0x7fcea00fed80 msgr2=0x7fcea01011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 -- 172.21.15.116:0/1032836170 shutdown_connections 2024-03-04T07:48:50.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.628+0000 7fcea6200640 1 -- 172.21.15.116:0/1032836170 wait complete. 2024-03-04T07:48:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.629+0000 7fcea6200640 1 Processor -- start 2024-03-04T07:48:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.629+0000 7fcea6200640 1 -- start start 2024-03-04T07:48:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fcea6200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea0103810 0x7fcea0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fcea6200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea0104a00 0x7fcea0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fcea6200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcea01053c0 0x7fcea01a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fcea6200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea010f470 con 0x7fcea0104a00 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fcea6200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea010f2f0 con 0x7fcea0103810 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fcea6200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea010f5f0 con 0x7fcea01053c0 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea0103810 0x7fcea0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea0103810 0x7fcea0199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60992/0 (socket says 172.21.15.116:60992) 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 -- 172.21.15.116:0/3117031476 learned_addr learned my addr 172.21.15.116:0/3117031476 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce97e00640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea0104a00 0x7fcea0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9fe00640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcea01053c0 0x7fcea01a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 -- 172.21.15.116:0/3117031476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcea01053c0 msgr2=0x7fcea01a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcea01053c0 0x7fcea01a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 -- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea0104a00 msgr2=0x7fcea0199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea0104a00 0x7fcea0199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.630+0000 7fce9f400640 1 -- 172.21.15.116:0/3117031476 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea01a16d0 con 0x7fcea0103810 2024-03-04T07:48:50.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.631+0000 7fce9f400640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea0103810 0x7fcea0199ab0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fce9000d770 tx=0x7fce9000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.631+0000 7fce9cc00640 1 -- 172.21.15.116:0/3117031476 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90017070 con 0x7fcea0103810 2024-03-04T07:48:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.631+0000 7fce9cc00640 1 -- 172.21.15.116:0/3117031476 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce90009c30 con 0x7fcea0103810 2024-03-04T07:48:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.631+0000 7fce9cc00640 1 -- 172.21.15.116:0/3117031476 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90003dd0 con 0x7fcea0103810 2024-03-04T07:48:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.631+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01a19c0 con 0x7fcea0103810 2024-03-04T07:48:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.631+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01a1f10 con 0x7fcea0103810 2024-03-04T07:48:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.632+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea00683c0 con 0x7fcea0103810 2024-03-04T07:48:50.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.634+0000 7fce9cc00640 1 -- 172.21.15.116:0/3117031476 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fce90003f70 con 0x7fcea0103810 2024-03-04T07:48:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.635+0000 7fce9cc00640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce840a1e00 0x7fce840a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.635+0000 7fce9cc00640 1 -- 172.21.15.116:0/3117031476 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fce900cd730 con 0x7fcea0103810 2024-03-04T07:48:50.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.635+0000 7fce97e00640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce840a1e00 0x7fce840a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:50.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.636+0000 7fce97e00640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce840a1e00 0x7fce840a42c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcea019afc0 tx=0x7fce88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.640+0000 7fce9cc00640 1 -- 172.21.15.116:0/3117031476 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fce90009de0 con 0x7fcea0103810 2024-03-04T07:48:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.875+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea010aaf0 con 0x7fcea0103810 2024-03-04T07:48:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.877+0000 7fce9cc00640 1 -- 172.21.15.116:0/3117031476 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fcea010aaf0 con 0x7fcea0103810 2024-03-04T07:48:50.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:48:50.878 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:48:50.878 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:48:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce840a1e00 msgr2=0x7fce840a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce840a1e00 0x7fce840a42c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcea019afc0 tx=0x7fce88005f90 comp rx=0 tx=0).stop 2024-03-04T07:48:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea0103810 msgr2=0x7fcea0199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea0103810 0x7fcea0199ab0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fce9000d770 tx=0x7fce9000dc30 comp rx=0 tx=0).stop 2024-03-04T07:48:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 shutdown_connections 2024-03-04T07:48:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fce840a1e00 0x7fce840a42c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcea01053c0 0x7fcea01a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcea0104a00 0x7fcea0199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 --2- 172.21.15.116:0/3117031476 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcea0103810 0x7fcea0199ab0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 >> 172.21.15.116:0/3117031476 conn(0x7fcea00fed80 msgr2=0x7fcea0107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 shutdown_connections 2024-03-04T07:48:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:50.880+0000 7fcea6200640 1 -- 172.21.15.116:0/3117031476 wait complete. 2024-03-04T07:48:50.900 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:48:50.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:48:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 -- 172.21.15.116:0/252357885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 msgr2=0x7fed0c10ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 --2- 172.21.15.116:0/252357885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c10ff30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fed04009860 tx=0x7fed0402bb70 comp rx=0 tx=0).stop 2024-03-04T07:48:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 -- 172.21.15.116:0/252357885 shutdown_connections 2024-03-04T07:48:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 --2- 172.21.15.116:0/252357885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c10ff30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 --2- 172.21.15.116:0/252357885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c108ce0 0x7fed0c109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 --2- 172.21.15.116:0/252357885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed0c107af0 0x7fed0c107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 -- 172.21.15.116:0/252357885 >> 172.21.15.116:0/252357885 conn(0x7fed0c076aa0 msgr2=0x7fed0c078ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 -- 172.21.15.116:0/252357885 shutdown_connections 2024-03-04T07:48:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.106+0000 7fed0b200640 1 -- 172.21.15.116:0/252357885 wait complete. 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 Processor -- start 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 -- start start 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed0c107af0 0x7fed0c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c108ce0 0x7fed0c19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c1a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c113770 con 0x7fed0c108ce0 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1135f0 con 0x7fed0c1096a0 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.107+0000 7fed0b200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1138f0 con 0x7fed0c107af0 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed01400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c108ce0 0x7fed0c19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c1a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c1a5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32776/0 (socket says 172.21.15.116:32776) 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 -- 172.21.15.116:0/2584355364 learned_addr learned my addr 172.21.15.116:0/2584355364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed09e00640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed0c107af0 0x7fed0c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 -- 172.21.15.116:0/2584355364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed0c107af0 msgr2=0x7fed0c19dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed0c107af0 0x7fed0c19dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 -- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c108ce0 msgr2=0x7fed0c19e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c108ce0 0x7fed0c19e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 -- 172.21.15.116:0/2584355364 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c1a59b0 con 0x7fed0c1096a0 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.108+0000 7fed0a800640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c1a5310 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fed04034b10 tx=0x7fed04032470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.109+0000 7fed03400640 1 -- 172.21.15.116:0/2584355364 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04041030 con 0x7fed0c1096a0 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.109+0000 7fed03400640 1 -- 172.21.15.116:0/2584355364 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed0402c510 con 0x7fed0c1096a0 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.109+0000 7fed03400640 1 -- 172.21.15.116:0/2584355364 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04039040 con 0x7fed0c1096a0 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.109+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c1a5c40 con 0x7fed0c1096a0 2024-03-04T07:48:51.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.109+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c1a6130 con 0x7fed0c1096a0 2024-03-04T07:48:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 Processor -- start 2024-03-04T07:48:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed03400640 1 -- 172.21.15.116:0/2584355364 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fed04044070 con 0x7fed0c1096a0 2024-03-04T07:48:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 -- start start 2024-03-04T07:48:51.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023c5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556302329d60 con 0x5563023c4e70 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563022d2cf0 con 0x5563023ca9d0 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.112+0000 7fed13f91740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556302325e00 con 0x5563023c5860 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33066/0 (socket says 172.21.15.116:33066) 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 -- 172.21.15.116:0/505494978 learned_addr learned my addr 172.21.15.116:0/505494978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed09e00640 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 -- 172.21.15.116:0/505494978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 msgr2=0x5563023ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 -- 172.21.15.116:0/505494978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 msgr2=0x5563023cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed01400640 1 -- 172.21.15.116:0/505494978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55630203e3f0 con 0x5563023c4e70 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.113+0000 7fed03400640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fecf00a1d80 0x7fecf00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.114+0000 7fed03400640 1 -- 172.21.15.116:0/2584355364 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed040ef2c0 con 0x7fed0c1096a0 2024-03-04T07:48:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.114+0000 7fed01400640 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023c5290 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fecf4013850 tx=0x7fecf4009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2198fdc70016d649 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:51.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.114+0000 7fed09e00640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fecf00a1d80 0x7fecf00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.114+0000 7fecfb400640 1 -- 172.21.15.116:0/505494978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4013dd0 con 0x5563023c4e70 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.114+0000 7fecfb400640 1 -- 172.21.15.116:0/505494978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecf4013170 con 0x5563023c4e70 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.114+0000 7fecfb400640 1 -- 172.21.15.116:0/505494978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4010940 con 0x5563023c4e70 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 -- 172.21.15.116:0/505494978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 msgr2=0x5563023c5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023c5290 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fecf4013850 tx=0x7fecf4009660 comp rx=0 tx=0).stop 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed09e00640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fecf00a1d80 0x7fecf00a4240 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5563023c5c60 tx=0x7fecfc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 -- 172.21.15.116:0/505494978 shutdown_connections 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 --2- 172.21.15.116:0/505494978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023c5290 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 -- 172.21.15.116:0/505494978 >> 172.21.15.116:0/505494978 conn(0x5563023c03e0 msgr2=0x5563023c07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:51.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 -- 172.21.15.116:0/505494978 shutdown_connections 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 -- 172.21.15.116:0/505494978 wait complete. 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.115+0000 7fed13f91740 1 -- start start 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 1 Processor -- start 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023c4e70 0x5563023cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023ca9d0 0x5563023d68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563023cd920 con 0x5563023ca9d0 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563022c8220 con 0x5563023c4e70 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563023cdaa0 con 0x5563023c5860 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed13f91740 10 client.0 _ll_register_callbacks cb 0x55630203dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed09e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023c4e70 0x5563023cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed0a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023ca9d0 0x5563023d68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.116+0000 7fed09e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023c4e70 0x5563023cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32796/0 (socket says 172.21.15.116:32796) 2024-03-04T07:48:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed0a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023ca9d0 0x5563023d68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33068/0 (socket says 172.21.15.116:33068) 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed09e00640 1 -- 172.21.15.116:0/1973995612 learned_addr learned my addr 172.21.15.116:0/1973995612 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed01400640 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed09e00640 1 -- 172.21.15.116:0/1973995612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 msgr2=0x5563023cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed09e00640 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed09e00640 1 -- 172.21.15.116:0/1973995612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023ca9d0 msgr2=0x5563023d68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed09e00640 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023ca9d0 0x5563023d68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed09e00640 1 -- 172.21.15.116:0/1973995612 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563021087d0 con 0x5563023c4e70 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed0a800640 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023ca9d0 0x5563023d68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fed09e00640 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023c4e70 0x5563023cf370 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fecfc003fa0 tx=0x7fecfc004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e35e14765dae0777 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fecefe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.117+0000 7fecefe00640 1 -- 172.21.15.116:0/1973995612 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc004b20 con 0x5563023c4e70 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fecefe00640 1 -- 172.21.15.116:0/1973995612 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fecfc004cc0 con 0x5563023c4e70 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecf4013170 con 0x5563023c4e70 2024-03-04T07:48:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fecefe00640 1 -- 172.21.15.116:0/1973995612 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecfc00f8f0 con 0x5563023c4e70 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 10 client.27344 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5563023d9cd0 con 0x5563023c4e70 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fecefe00640 1 -- 172.21.15.116:0/1973995612 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecf4013170 con 0x5563023c4e70 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fecefe00640 1 -- 172.21.15.116:0/1973995612 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fecfc026070 con 0x5563023c4e70 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 10 client.27344 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 10 client.27344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 20 client.27344 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x5563023dd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 4 client.27344 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.118+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563023372a0 con 0x5563023db2f0 2024-03-04T07:48:51.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.119+0000 7fed01400640 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x5563023dd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.119+0000 7fed01400640 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x5563023dd770 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a82be3b3124e38d server_cookie=dc372aa662c8fa01 in_seq=0 out_seq=0 2024-03-04T07:48:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.119+0000 7fecefe00640 10 client.27344 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.120+0000 7fecefe00640 1 -- 172.21.15.116:0/1973995612 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5563023372a0 con 0x5563023db2f0 2024-03-04T07:48:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.120+0000 7fecefe00640 10 client.27344 handle_command_reply: tid=0 2024-03-04T07:48:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.120+0000 7fed13f91740 1 client.27344 shutdown 2024-03-04T07:48:51.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.120+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023c4e70 msgr2=0x5563023cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.120+0000 7fed13f91740 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023c4e70 0x5563023cf370 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fecfc003fa0 tx=0x7fecfc004ab0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.120+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 shutdown_connections 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.120+0000 7fed13f91740 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x5563023dd770 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fecefe00640 0 client.27344 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fed13f91740 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023ca9d0 0x5563023d68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fed13f91740 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fed13f91740 1 --2- 172.21.15.116:0/1973995612 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023c4e70 0x5563023cf370 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 >> 172.21.15.116:0/1973995612 conn(0x5563023c03e0 msgr2=0x5563023bf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 shutdown_connections 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fed13f91740 1 -- 172.21.15.116:0/1973995612 wait complete. 2024-03-04T07:48:51.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.121+0000 7fed13f91740 20 client.27344 trim_cache size 0 max 16384 2024-03-04T07:48:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.134+0000 7fed13f91740 1 Processor -- start 2024-03-04T07:48:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.134+0000 7fed13f91740 1 -- start start 2024-03-04T07:48:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed13f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed13f91740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023d68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed13f91740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed13f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1ace80 con 0x5563023c4e70 2024-03-04T07:48:51.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed13f91740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1af460 con 0x5563023ca9d0 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed13f91740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1ad480 con 0x5563023c5860 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed09e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed0a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed01400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023d68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed09e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023cf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33080/0 (socket says 172.21.15.116:33080) 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed09e00640 1 -- 172.21.15.116:0/18349530 learned_addr learned my addr 172.21.15.116:0/18349530 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed0a800640 1 -- 172.21.15.116:0/18349530 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 msgr2=0x5563023d68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed0a800640 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023d68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed0a800640 1 -- 172.21.15.116:0/18349530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 msgr2=0x5563023cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed0a800640 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed0a800640 1 -- 172.21.15.116:0/18349530 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563023c0d00 con 0x5563023ca9d0 2024-03-04T07:48:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.135+0000 7fed09e00640 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.136+0000 7fed0a800640 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023c07c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fed040bbcf0 tx=0x7fed040bbbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84ce6fe706f1e18b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.136+0000 7fecfaa00640 1 -- 172.21.15.116:0/18349530 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040d9070 con 0x5563023ca9d0 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.136+0000 7fecfaa00640 1 -- 172.21.15.116:0/18349530 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed0402c6c0 con 0x5563023ca9d0 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.136+0000 7fecfaa00640 1 -- 172.21.15.116:0/18349530 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040c4300 con 0x5563023ca9d0 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.136+0000 7fed13f91740 1 -- 172.21.15.116:0/18349530 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 msgr2=0x5563023c07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023c07c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fed040bbcf0 tx=0x7fed040bbbf0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 -- 172.21.15.116:0/18349530 shutdown_connections 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023c07c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c5860 0x5563023d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 --2- 172.21.15.116:0/18349530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c4e70 0x5563023cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 -- 172.21.15.116:0/18349530 >> 172.21.15.116:0/18349530 conn(0x5563023c03e0 msgr2=0x5563023cdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:51.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 -- 172.21.15.116:0/18349530 shutdown_connections 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 -- 172.21.15.116:0/18349530 wait complete. 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.137+0000 7fed13f91740 1 -- start start 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 1 Processor -- start 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c4e70 0x5563023d45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c5860 0x5563023d4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023d5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1adae0 con 0x5563023c5860 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1aee00 con 0x5563023ca9d0 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1ae140 con 0x5563023c4e70 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed13f91740 10 client.0 _ll_register_callbacks cb 0x7fecfc003e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed0a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023d5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed01400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c5860 0x5563023d4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed09e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c4e70 0x5563023d45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed0a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023d5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32812/0 (socket says 172.21.15.116:32812) 2024-03-04T07:48:51.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.138+0000 7fed0a800640 1 -- 172.21.15.116:0/3805710623 learned_addr learned my addr 172.21.15.116:0/3805710623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed0a800640 1 -- 172.21.15.116:0/3805710623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c4e70 msgr2=0x5563023d45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed0a800640 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c4e70 0x5563023d45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed0a800640 1 -- 172.21.15.116:0/3805710623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c5860 msgr2=0x5563023d4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed0a800640 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c5860 0x5563023d4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed0a800640 1 -- 172.21.15.116:0/3805710623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563023d6110 con 0x5563023ca9d0 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed09e00640 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c4e70 0x5563023d45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed01400640 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c5860 0x5563023d4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed0a800640 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023d5b20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fed04005940 tx=0x7fed040ba500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af200ef80737cb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fecefe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fecefe00640 1 -- 172.21.15.116:0/3805710623 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040d9020 con 0x5563023ca9d0 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fecefe00640 1 -- 172.21.15.116:0/3805710623 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed040da220 con 0x5563023ca9d0 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed0402c6c0 con 0x5563023ca9d0 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fecefe00640 1 -- 172.21.15.116:0/3805710623 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed040bc6f0 con 0x5563023ca9d0 2024-03-04T07:48:51.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed13f91740 10 client.27350 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5563024371e0 con 0x5563023ca9d0 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.139+0000 7fecefe00640 1 -- 172.21.15.116:0/3805710623 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed0402c6c0 con 0x5563023ca9d0 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fecefe00640 1 -- 172.21.15.116:0/3805710623 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fed0404e060 con 0x5563023ca9d0 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed13f91740 10 client.27350 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed13f91740 10 client.27350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed13f91740 20 client.27350 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed13f91740 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x556302439980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed13f91740 4 client.27350 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5563023315f0 con 0x5563023db2f0 2024-03-04T07:48:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed09e00640 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x556302439980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fed09e00640 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x556302439980 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500fb722af42f031 server_cookie=726c459c7aecf25a in_seq=0 out_seq=0 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.140+0000 7fecefe00640 10 client.27350 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fecefe00640 1 -- 172.21.15.116:0/3805710623 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5563023315f0 con 0x5563023db2f0 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fecefe00640 10 client.27350 handle_command_reply: tid=0 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 client.27350 shutdown 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 msgr2=0x5563023d5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023d5b20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fed04005940 tx=0x7fed040ba500 comp rx=0 tx=0).stop 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 shutdown_connections 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5563023db2f0 0x556302439980 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5563023ca9d0 0x5563023d5b20 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563023c5860 0x5563023d4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fecefe00640 0 client.27350 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 --2- 172.21.15.116:0/3805710623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563023c4e70 0x5563023d45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 >> 172.21.15.116:0/3805710623 conn(0x5563023c03e0 msgr2=0x5563023c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 shutdown_connections 2024-03-04T07:48:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 1 -- 172.21.15.116:0/3805710623 wait complete. 2024-03-04T07:48:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.141+0000 7fed13f91740 20 client.27350 trim_cache size 0 max 16384 2024-03-04T07:48:51.147 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:48:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fecf00a1d80 msgr2=0x7fecf00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fecf00a1d80 0x7fecf00a4240 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5563023c5c60 tx=0x7fecfc009500 comp rx=0 tx=0).stop 2024-03-04T07:48:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 msgr2=0x7fed0c1a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c1a5310 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fed04034b10 tx=0x7fed04032470 comp rx=0 tx=0).stop 2024-03-04T07:48:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 shutdown_connections 2024-03-04T07:48:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed0c1096a0 0x7fed0c1a5310 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed0c108ce0 0x7fed0c19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fecf00a1d80 0x7fecf00a4240 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 --2- 172.21.15.116:0/2584355364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed0c107af0 0x7fed0c19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 >> 172.21.15.116:0/2584355364 conn(0x7fed0c076aa0 msgr2=0x7fed0c10bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 shutdown_connections 2024-03-04T07:48:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:51.142+0000 7fed0b200640 1 -- 172.21.15.116:0/2584355364 wait complete. 2024-03-04T07:48:51.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:50 smithi116 ceph-mon[32150]: pgmap v643: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:50 smithi152 ceph-mon[37154]: pgmap v643: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:51.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:50 smithi133 ceph-mon[36505]: pgmap v643: 97 pgs: 97 active+clean; 2.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:48:52.162 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:48:52.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:48:52.162 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:48:52.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:51 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3117031476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:52.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:51 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3117031476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:52.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:51 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3117031476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:48:52.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.390+0000 7fa18de00640 1 -- 172.21.15.116:0/44391148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa188105170 msgr2=0x7fa1881055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.390+0000 7fa18de00640 1 --2- 172.21.15.116:0/44391148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa188105170 0x7fa1881055f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fa180009900 tx=0x7fa18002bc30 comp rx=0 tx=0).stop 2024-03-04T07:48:52.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.391+0000 7fa18de00640 1 -- 172.21.15.116:0/44391148 shutdown_connections 2024-03-04T07:48:52.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.391+0000 7fa18de00640 1 --2- 172.21.15.116:0/44391148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa188105170 0x7fa1881055f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.391+0000 7fa18de00640 1 --2- 172.21.15.116:0/44391148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1881027c0 0x7fa188104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.391+0000 7fa18de00640 1 --2- 172.21.15.116:0/44391148 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa1880ffe50 0x7fa188102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.391+0000 7fa18de00640 1 -- 172.21.15.116:0/44391148 >> 172.21.15.116:0/44391148 conn(0x7fa1880fba70 msgr2=0x7fa1880fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.391+0000 7fa18de00640 1 -- 172.21.15.116:0/44391148 shutdown_connections 2024-03-04T07:48:52.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.391+0000 7fa18de00640 1 -- 172.21.15.116:0/44391148 wait complete. 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 Processor -- start 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 -- start start 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1880ffe50 0x7fa18819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1881027c0 0x7fa18819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa188105170 0x7fa1881a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881138a0 con 0x7fa1880ffe50 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa188113720 con 0x7fa1881027c0 2024-03-04T07:48:52.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa18de00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa188113a20 con 0x7fa188105170 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1880ffe50 0x7fa18819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa186000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1881027c0 0x7fa18819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1880ffe50 0x7fa18819def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33100/0 (socket says 172.21.15.116:33100) 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.392+0000 7fa186a00640 1 -- 172.21.15.116:0/1335044541 learned_addr learned my addr 172.21.15.116:0/1335044541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa187400640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa188105170 0x7fa1881a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa186000640 1 -- 172.21.15.116:0/1335044541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa188105170 msgr2=0x7fa1881a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa186000640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa188105170 0x7fa1881a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa186000640 1 -- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1880ffe50 msgr2=0x7fa18819def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa186000640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1880ffe50 0x7fa18819def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa186000640 1 -- 172.21.15.116:0/1335044541 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1881a5b10 con 0x7fa1881027c0 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa187400640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa188105170 0x7fa1881a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa186000640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1881027c0 0x7fa18819e430 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa17000eb00 tx=0x7fa17000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa18ca00640 1 -- 172.21.15.116:0/1335044541 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170010430 con 0x7fa1881027c0 2024-03-04T07:48:52.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa18ca00640 1 -- 172.21.15.116:0/1335044541 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa170010ab0 con 0x7fa1881027c0 2024-03-04T07:48:52.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1881a5e00 con 0x7fa1881027c0 2024-03-04T07:48:52.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa18ca00640 1 -- 172.21.15.116:0/1335044541 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170017bb0 con 0x7fa1881027c0 2024-03-04T07:48:52.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.393+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1881a6310 con 0x7fa1881027c0 2024-03-04T07:48:52.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.396+0000 7fa190b71740 1 Processor -- start 2024-03-04T07:48:52.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa190b71740 1 -- start start 2024-03-04T07:48:52.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa190b71740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba5480d0 0x5636ba5484f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa190b71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba548ac0 0x5636ba54d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa190b71740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa190b71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ba45be20 con 0x5636ba548ac0 2024-03-04T07:48:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa190b71740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ba4ada20 con 0x5636ba54dc30 2024-03-04T07:48:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa190b71740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ba456b00 con 0x5636ba5480d0 2024-03-04T07:48:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa18ca00640 1 -- 172.21.15.116:0/1335044541 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa170010c60 con 0x7fa1881027c0 2024-03-04T07:48:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba548ac0 0x5636ba54d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa187400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba5480d0 0x5636ba5484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba548ac0 0x5636ba54d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33116/0 (socket says 172.21.15.116:33116) 2024-03-04T07:48:52.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa187400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba5480d0 0x5636ba5484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:36694/0 (socket says 172.21.15.116:36694) 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa186a00640 1 -- 172.21.15.116:0/1009141537 learned_addr learned my addr 172.21.15.116:0/1009141537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.397+0000 7fa186000640 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa186a00640 1 -- 172.21.15.116:0/1009141537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba5480d0 msgr2=0x5636ba5484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa186a00640 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba5480d0 0x5636ba5484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa186a00640 1 -- 172.21.15.116:0/1009141537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 msgr2=0x5636ba5500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa186a00640 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa186a00640 1 -- 172.21.15.116:0/1009141537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636ba46be10 con 0x5636ba548ac0 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa186a00640 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba548ac0 0x5636ba54d5c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa17800bdd0 tx=0x7fa1780107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618e9279a82c2fef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa177e00640 1 -- 172.21.15.116:0/1009141537 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178005220 con 0x5636ba548ac0 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa177e00640 1 -- 172.21.15.116:0/1009141537 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa178004c80 con 0x5636ba548ac0 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.398+0000 7fa177e00640 1 -- 172.21.15.116:0/1009141537 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178011880 con 0x5636ba548ac0 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa18ca00640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1680a1e00 0x7fa1680a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa18ca00640 1 -- 172.21.15.116:0/1335044541 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa1700ce700 con 0x7fa1881027c0 2024-03-04T07:48:52.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa187400640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1680a1e00 0x7fa1680a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 -- 172.21.15.116:0/1009141537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba548ac0 msgr2=0x5636ba54d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba548ac0 0x5636ba54d5c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa17800bdd0 tx=0x7fa1780107a0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 -- 172.21.15.116:0/1009141537 shutdown_connections 2024-03-04T07:48:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba548ac0 0x5636ba54d5c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 --2- 172.21.15.116:0/1009141537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba5480d0 0x5636ba5484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 -- 172.21.15.116:0/1009141537 >> 172.21.15.116:0/1009141537 conn(0x5636ba543640 msgr2=0x5636ba543a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 -- 172.21.15.116:0/1009141537 shutdown_connections 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.399+0000 7fa190b71740 1 -- 172.21.15.116:0/1009141537 wait complete. 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.400+0000 7fa187400640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1680a1e00 0x7fa1680a42c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa180036040 tx=0x7fa18002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.400+0000 7fa190b71740 1 -- start start 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.400+0000 7fa190b71740 1 Processor -- start 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa190b71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba5480d0 0x5636ba552460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa190b71740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba5529a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa190b71740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa190b71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ba469f40 con 0x5636ba5480d0 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa190b71740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ba4a9ac0 con 0x5636ba54dc30 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa190b71740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636ba550be0 con 0x5636ba548ac0 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa190b71740 10 client.0 _ll_register_callbacks cb 0x5636ba4a8aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba5480d0 0x5636ba552460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba5480d0 0x5636ba552460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33130/0 (socket says 172.21.15.116:33130) 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 -- 172.21.15.116:0/1394195402 learned_addr learned my addr 172.21.15.116:0/1394195402 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa187400640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186000640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba5529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 -- 172.21.15.116:0/1394195402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 msgr2=0x5636ba5529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba5529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 -- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 msgr2=0x5636ba5599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa186a00640 1 -- 172.21.15.116:0/1394195402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636ba28a7e0 con 0x5636ba5480d0 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.401+0000 7fa187400640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa186a00640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba5480d0 0x5636ba552460 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa178010ca0 tx=0x7fa178005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46472540b8cce82 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa174c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa174c00640 1 -- 172.21.15.116:0/1394195402 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1780053a0 con 0x5636ba5480d0 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa174c00640 1 -- 172.21.15.116:0/1394195402 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa1780048d0 con 0x5636ba5480d0 2024-03-04T07:48:52.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa178004c80 con 0x5636ba5480d0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa174c00640 1 -- 172.21.15.116:0/1394195402 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa178004ad0 con 0x5636ba5480d0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa190b71740 10 client.17232 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.402+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5636ba2eaf70 con 0x5636ba5480d0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa174c00640 1 -- 172.21.15.116:0/1394195402 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa178004c80 con 0x5636ba5480d0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa174c00640 1 -- 172.21.15.116:0/1394195402 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa17801b5d0 con 0x5636ba5480d0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa190b71740 10 client.17232 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa190b71740 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa190b71740 20 client.17232 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa190b71740 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba55dff0 0x5636ba5603e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa190b71740 4 client.17232 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636ba4baf60 con 0x5636ba55dff0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.403+0000 7fa186000640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba55dff0 0x5636ba5603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.404+0000 7fa186000640 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba55dff0 0x5636ba5603e0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33f6f03be250ddf2 server_cookie=f7ac3d463b289a11 in_seq=0 out_seq=0 2024-03-04T07:48:52.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.404+0000 7fa174c00640 10 client.17232 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa174c00640 1 -- 172.21.15.116:0/1394195402 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5636ba4baf60 con 0x5636ba55dff0 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa174c00640 10 client.17232 handle_command_reply: tid=0 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 client.17232 shutdown 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba5480d0 msgr2=0x5636ba552460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba5480d0 0x5636ba552460 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa178010ca0 tx=0x7fa178005230 comp rx=0 tx=0).stop 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 shutdown_connections 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba55dff0 0x5636ba5603e0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba54dc30 0x5636ba5599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba5529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 --2- 172.21.15.116:0/1394195402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba5480d0 0x5636ba552460 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa174c00640 0 client.17232 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 >> 172.21.15.116:0/1394195402 conn(0x5636ba543640 msgr2=0x5636ba544890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 shutdown_connections 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 1 -- 172.21.15.116:0/1394195402 wait complete. 2024-03-04T07:48:52.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.405+0000 7fa190b71740 20 client.17232 trim_cache size 0 max 16384 2024-03-04T07:48:52.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 Processor -- start 2024-03-04T07:48:52.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 -- start start 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba542a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba5599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba546fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881ad0a0 con 0x5636ba543640 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881af680 con 0x5636ba5480d0 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.423+0000 7fa190b71740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881ad6a0 con 0x5636ba548ac0 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba542a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba542a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33140/0 (socket says 172.21.15.116:33140) 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 -- 172.21.15.116:0/1378269221 learned_addr learned my addr 172.21.15.116:0/1378269221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186000640 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba5599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa187400640 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba546fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 -- 172.21.15.116:0/1378269221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 msgr2=0x5636ba546fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba546fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 -- 172.21.15.116:0/1378269221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 msgr2=0x5636ba5599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba5599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186a00640 1 -- 172.21.15.116:0/1378269221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636ba560750 con 0x5636ba543640 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa186000640 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba5599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.424+0000 7fa187400640 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba546fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.425+0000 7fa186a00640 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba542a60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa17801b970 tx=0x7fa1780107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d67e6cc4b240e7d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.425+0000 7fa177400640 1 -- 172.21.15.116:0/1378269221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17800b810 con 0x5636ba543640 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.425+0000 7fa177400640 1 -- 172.21.15.116:0/1378269221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1780056b0 con 0x5636ba543640 2024-03-04T07:48:52.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.425+0000 7fa177400640 1 -- 172.21.15.116:0/1378269221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178004d70 con 0x5636ba543640 2024-03-04T07:48:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.425+0000 7fa190b71740 1 -- 172.21.15.116:0/1378269221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 msgr2=0x5636ba542a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.425+0000 7fa190b71740 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba542a60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa17801b970 tx=0x7fa1780107a0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 -- 172.21.15.116:0/1378269221 shutdown_connections 2024-03-04T07:48:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba546fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba5599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 --2- 172.21.15.116:0/1378269221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba542a60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 -- 172.21.15.116:0/1378269221 >> 172.21.15.116:0/1378269221 conn(0x5636ba5498a0 msgr2=0x5636ba5468a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 -- 172.21.15.116:0/1378269221 shutdown_connections 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 -- 172.21.15.116:0/1378269221 wait complete. 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 -- start start 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.426+0000 7fa190b71740 1 Processor -- start 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa190b71740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba5576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa190b71740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba557bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa190b71740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba558bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa190b71740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881add00 con 0x5636ba543640 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa190b71740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881af020 con 0x5636ba5480d0 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa190b71740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881ae360 con 0x5636ba548ac0 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa190b71740 10 client.0 _ll_register_callbacks cb 0x7fa1700093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba557bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba557bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32856/0 (socket says 172.21.15.116:32856) 2024-03-04T07:48:52.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 -- 172.21.15.116:0/2730676911 learned_addr learned my addr 172.21.15.116:0/2730676911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186a00640 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba5576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa187400640 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba558bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 -- 172.21.15.116:0/2730676911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 msgr2=0x5636ba558bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba558bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 -- 172.21.15.116:0/2730676911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 msgr2=0x5636ba5576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba5576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.427+0000 7fa186000640 1 -- 172.21.15.116:0/2730676911 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636ba182170 con 0x5636ba5480d0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa186000640 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba557bf0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa17009b3c0 tx=0x7fa17000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b46f33d70fc8ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa174c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa174c00640 1 -- 172.21.15.116:0/2730676911 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1700a3bd0 con 0x5636ba5480d0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa174c00640 1 -- 172.21.15.116:0/2730676911 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa170017d50 con 0x5636ba5480d0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1780056b0 con 0x5636ba5480d0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa174c00640 1 -- 172.21.15.116:0/2730676911 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa170097aa0 con 0x5636ba5480d0 2024-03-04T07:48:52.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 10 client.27368 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5636ba5b9cc0 con 0x5636ba5480d0 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa174c00640 1 -- 172.21.15.116:0/2730676911 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1780056b0 con 0x5636ba5480d0 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa174c00640 1 -- 172.21.15.116:0/2730676911 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa1700b9c30 con 0x5636ba5480d0 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 10 client.27368 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 10 client.27368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 20 client.27368 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba54dc30 0x5636ba5bc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 4 client.27368 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.428+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5636ba4b52b0 con 0x5636ba54dc30 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.429+0000 7fa186a00640 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba54dc30 0x5636ba5bc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.429+0000 7fa186a00640 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba54dc30 0x5636ba5bc5b0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e079787c8bc841e1 server_cookie=76a79a16104d8e50 in_seq=0 out_seq=0 2024-03-04T07:48:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.429+0000 7fa174c00640 10 client.27368 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa174c00640 1 -- 172.21.15.116:0/2730676911 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5636ba4b52b0 con 0x5636ba54dc30 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa174c00640 10 client.27368 handle_command_reply: tid=0 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 client.27368 shutdown 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 msgr2=0x5636ba557bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba557bf0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa17009b3c0 tx=0x7fa17000c900 comp rx=0 tx=0).stop 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 shutdown_connections 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5636ba54dc30 0x5636ba5bc5b0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5636ba548ac0 0x5636ba558bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5636ba5480d0 0x5636ba557bf0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 --2- 172.21.15.116:0/2730676911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636ba543640 0x5636ba5576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 >> 172.21.15.116:0/2730676911 conn(0x5636ba5498a0 msgr2=0x5636ba547130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.430+0000 7fa174c00640 0 client.27368 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 shutdown_connections 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa190b71740 1 -- 172.21.15.116:0/2730676911 wait complete. 2024-03-04T07:48:52.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa190b71740 20 client.27368 trim_cache size 0 max 16384 2024-03-04T07:48:52.436 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:52.436 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:48:52.436 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "a47c0f2d-127e-4051-bff5-9059a38fec4c", 2024-03-04T07:48:52.436 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:48:52.436 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1680a1e00 msgr2=0x7fa1680a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1680a1e00 0x7fa1680a42c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa180036040 tx=0x7fa18002df90 comp rx=0 tx=0).stop 2024-03-04T07:48:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1881027c0 msgr2=0x7fa18819e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1881027c0 0x7fa18819e430 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa17000eb00 tx=0x7fa17000c2d0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 shutdown_connections 2024-03-04T07:48:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa1680a1e00 0x7fa1680a42c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa188105170 0x7fa1881a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa1881027c0 0x7fa18819e430 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 --2- 172.21.15.116:0/1335044541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1880ffe50 0x7fa18819def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.431+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 >> 172.21.15.116:0/1335044541 conn(0x7fa1880fba70 msgr2=0x7fa1881007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.432+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 shutdown_connections 2024-03-04T07:48:52.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.432+0000 7fa18de00640 1 -- 172.21.15.116:0/1335044541 wait complete. 2024-03-04T07:48:52.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:48:52.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.661+0000 7ff557e00640 1 -- 172.21.15.116:0/3569426669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff558101f40 msgr2=0x7ff558104330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.661+0000 7ff557e00640 1 --2- 172.21.15.116:0/3569426669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff558101f40 0x7ff558104330 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff548009ca0 tx=0x7ff54802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.661+0000 7ff557e00640 1 -- 172.21.15.116:0/3569426669 shutdown_connections 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.661+0000 7ff557e00640 1 --2- 172.21.15.116:0/3569426669 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff5581071a0 0x7ff5581095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.661+0000 7ff557e00640 1 --2- 172.21.15.116:0/3569426669 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff558104870 0x7ff558106c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.661+0000 7ff557e00640 1 --2- 172.21.15.116:0/3569426669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff558101f40 0x7ff558104330 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.661+0000 7ff557e00640 1 -- 172.21.15.116:0/3569426669 >> 172.21.15.116:0/3569426669 conn(0x7ff5580fb950 msgr2=0x7ff5580fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.662+0000 7ff557e00640 1 -- 172.21.15.116:0/3569426669 shutdown_connections 2024-03-04T07:48:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.662+0000 7ff557e00640 1 -- 172.21.15.116:0/3569426669 wait complete. 2024-03-04T07:48:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.662+0000 7ff557e00640 1 Processor -- start 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557e00640 1 -- start start 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff558101f40 0x7ff558199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff558104870 0x7ff558199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5581071a0 0x7ff5581a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55810d150 con 0x7ff5581071a0 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55810cfd0 con 0x7ff558104870 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55810d2d0 con 0x7ff558101f40 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff557400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5581071a0 0x7ff5581a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff556000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff558104870 0x7ff558199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff556000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff558104870 0x7ff558199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32878/0 (socket says 172.21.15.116:32878) 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff556000640 1 -- 172.21.15.116:0/2285299628 learned_addr learned my addr 172.21.15.116:0/2285299628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.663+0000 7ff556a00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff558101f40 0x7ff558199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff556000640 1 -- 172.21.15.116:0/2285299628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff558101f40 msgr2=0x7ff558199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff556000640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff558101f40 0x7ff558199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff556000640 1 -- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5581071a0 msgr2=0x7ff5581a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff556000640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5581071a0 0x7ff5581a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff556000640 1 -- 172.21.15.116:0/2285299628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5581a16a0 con 0x7ff558104870 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff556a00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff558101f40 0x7ff558199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff557400640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5581071a0 0x7ff5581a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff556000640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff558104870 0x7ff558199fc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff54000e9c0 tx=0x7ff54000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff55ca00640 1 -- 172.21.15.116:0/2285299628 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54000ccc0 con 0x7ff558104870 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5581a1990 con 0x7ff558104870 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff55ca00640 1 -- 172.21.15.116:0/2285299628 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff540005500 con 0x7ff558104870 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff55ca00640 1 -- 172.21.15.116:0/2285299628 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540010430 con 0x7ff558104870 2024-03-04T07:48:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.664+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5581a1f30 con 0x7ff558104870 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.667+0000 7ff560a52740 1 Processor -- start 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.667+0000 7ff55ca00640 1 -- 172.21.15.116:0/2285299628 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff540005000 con 0x7ff558104870 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff560a52740 1 -- start start 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff560a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1aaa130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff560a52740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1aaf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff560a52740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1ab1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff560a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbb19d7090 con 0x55fbb1aa9d10 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff560a52740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbb1a0ed30 con 0x55fbb1aaf870 2024-03-04T07:48:52.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff560a52740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbb19b7a30 con 0x55fbb1aaa700 2024-03-04T07:48:52.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff557400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1aaa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff556a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1aaf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff557400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1aaa130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33184/0 (socket says 172.21.15.116:33184) 2024-03-04T07:48:52.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff557400640 1 -- 172.21.15.116:0/3264686240 learned_addr learned my addr 172.21.15.116:0/3264686240 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.668+0000 7ff556000640 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1ab1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff556a00640 1 -- 172.21.15.116:0/3264686240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 msgr2=0x55fbb1ab1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff556a00640 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1ab1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff556a00640 1 -- 172.21.15.116:0/3264686240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 msgr2=0x55fbb1aaa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff556a00640 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1aaa130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff556a00640 1 -- 172.21.15.116:0/3264686240 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbb17ed7d0 con 0x55fbb1aaa700 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff556a00640 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1aaf200 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff548009bf0 tx=0x7ff548005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17699ddad04f0bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff55ca00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff53c0a1e00 0x7ff53c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff547e00640 1 -- 172.21.15.116:0/3264686240 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548043020 con 0x55fbb1aaa700 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff547e00640 1 -- 172.21.15.116:0/3264686240 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff548032c90 con 0x55fbb1aaa700 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff547e00640 1 -- 172.21.15.116:0/3264686240 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548039040 con 0x55fbb1aaa700 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff55ca00640 1 -- 172.21.15.116:0/2285299628 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff5400ce240 con 0x7ff558104870 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.669+0000 7ff557400640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff53c0a1e00 0x7ff53c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 -- 172.21.15.116:0/3264686240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 msgr2=0x55fbb1aaf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1aaf200 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff548009bf0 tx=0x7ff548005f70 comp rx=0 tx=0).stop 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff557400640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff53c0a1e00 0x7ff53c0a42c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff550009fd0 tx=0x7ff5500096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 -- 172.21.15.116:0/3264686240 shutdown_connections 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1ab1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1aaf200 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 --2- 172.21.15.116:0/3264686240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1aaa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 -- 172.21.15.116:0/3264686240 >> 172.21.15.116:0/3264686240 conn(0x55fbb1aa5280 msgr2=0x55fbb1aa5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 -- 172.21.15.116:0/3264686240 shutdown_connections 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.670+0000 7ff560a52740 1 -- 172.21.15.116:0/3264686240 wait complete. 2024-03-04T07:48:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 -- start start 2024-03-04T07:48:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 Processor -- start 2024-03-04T07:48:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ab40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1ab45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1abb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbb19ad0c0 con 0x55fbb1aa9d10 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbb1a0add0 con 0x55fbb1aaf870 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbb1ab27c0 con 0x55fbb1aaa700 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.671+0000 7ff560a52740 10 client.0 _ll_register_callbacks cb 0x55fbb19ca860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff557400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1abb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ab40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ab40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33188/0 (socket says 172.21.15.116:33188) 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff557400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1abb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32892/0 (socket says 172.21.15.116:32892) 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 -- 172.21.15.116:0/3570013466 learned_addr learned my addr 172.21.15.116:0/3570013466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556000640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1ab45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 -- 172.21.15.116:0/3570013466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 msgr2=0x55fbb1ab45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1ab45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 -- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 msgr2=0x55fbb1abb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1abb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 -- 172.21.15.116:0/3570013466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbb17233f0 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556000640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1ab45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff557400640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1abb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff556a00640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ab40a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff5480324c0 tx=0x7ff54802c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7395b21d836c53d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.672+0000 7ff544c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff544c00640 1 -- 172.21.15.116:0/3570013466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54804c070 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff544c00640 1 -- 172.21.15.116:0/3570013466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff548003f50 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff548032c90 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff544c00640 1 -- 172.21.15.116:0/3570013466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff548058050 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff560a52740 10 client.17250 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55fbb16e30f0 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff544c00640 1 -- 172.21.15.116:0/3570013466 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff548032c90 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.673+0000 7ff544c00640 1 -- 172.21.15.116:0/3570013466 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff54803f7e0 con 0x55fbb1aa9d10 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff560a52740 10 client.17250 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff560a52740 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff560a52740 20 client.17250 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff560a52740 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1abf780 0x55fbb1ac1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff560a52740 4 client.17250 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fbb1a1c270 con 0x55fbb1abf780 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff556000640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1abf780 0x55fbb1ac1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff556000640 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1abf780 0x55fbb1ac1b70 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6b8d9db48ed878 server_cookie=60c58c2d0fd73784 in_seq=0 out_seq=0 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.674+0000 7ff544c00640 10 client.17250 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.675+0000 7ff544c00640 1 -- 172.21.15.116:0/3570013466 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fbb1a1c270 con 0x55fbb1abf780 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.675+0000 7ff544c00640 10 client.17250 handle_command_reply: tid=0 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.675+0000 7ff560a52740 1 client.17250 shutdown 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.675+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 msgr2=0x55fbb1ab40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.675+0000 7ff560a52740 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ab40a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff5480324c0 tx=0x7ff54802c790 comp rx=0 tx=0).stop 2024-03-04T07:48:52.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 shutdown_connections 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1abf780 0x55fbb1ac1b70 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaf870 0x55fbb1abb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaa700 0x55fbb1ab45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff544c00640 0 client.17250 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 --2- 172.21.15.116:0/3570013466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ab40a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 >> 172.21.15.116:0/3570013466 conn(0x55fbb1aa5280 msgr2=0x55fbb1aa44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 shutdown_connections 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 1 -- 172.21.15.116:0/3570013466 wait complete. 2024-03-04T07:48:52.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.676+0000 7ff560a52740 20 client.17250 trim_cache size 0 max 16384 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.693+0000 7ff560a52740 1 Processor -- start 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.693+0000 7ff560a52740 1 -- start start 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff560a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ac6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff560a52740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1abb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff560a52740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1ac67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff560a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581a8b40 con 0x55fbb1aa9d10 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff560a52740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581ab120 con 0x55fbb1aaa700 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff560a52740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581a9140 con 0x55fbb1aaf870 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ac6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1abb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ac6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33190/0 (socket says 172.21.15.116:33190) 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556a00640 1 -- 172.21.15.116:0/3172672456 learned_addr learned my addr 172.21.15.116:0/3172672456 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff557400640 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1ac67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556000640 1 -- 172.21.15.116:0/3172672456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 msgr2=0x55fbb1ac67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556000640 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1ac67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556000640 1 -- 172.21.15.116:0/3172672456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 msgr2=0x55fbb1ac6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556000640 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ac6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556000640 1 -- 172.21.15.116:0/3172672456 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbb1ab28f0 con 0x55fbb1aaa700 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.694+0000 7ff556a00640 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ac6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff556000640 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1abb5e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff54000c6a0 tx=0x7ff5400d1c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d121749492f030 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff547400640 1 -- 172.21.15.116:0/3172672456 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54009aa70 con 0x55fbb1aaa700 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff547400640 1 -- 172.21.15.116:0/3172672456 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff54009b630 con 0x55fbb1aaa700 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff547400640 1 -- 172.21.15.116:0/3172672456 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54009a890 con 0x55fbb1aaa700 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 -- 172.21.15.116:0/3172672456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 msgr2=0x55fbb1abb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1abb5e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff54000c6a0 tx=0x7ff5400d1c80 comp rx=0 tx=0).stop 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 -- 172.21.15.116:0/3172672456 shutdown_connections 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1ac67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1abb5e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 --2- 172.21.15.116:0/3172672456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1ac6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 -- 172.21.15.116:0/3172672456 >> 172.21.15.116:0/3172672456 conn(0x55fbb1aa5280 msgr2=0x55fbb1aa8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 -- 172.21.15.116:0/3172672456 shutdown_connections 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.695+0000 7ff560a52740 1 -- 172.21.15.116:0/3172672456 wait complete. 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 -- start start 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 Processor -- start 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1b1d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1b1fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1b220c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581a97a0 con 0x55fbb1aa9d10 2024-03-04T07:48:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581aaac0 con 0x55fbb1aaa700 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.696+0000 7ff560a52740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581a9e00 con 0x55fbb1aaf870 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff560a52740 10 client.0 _ll_register_callbacks cb 0x7ff550005e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1b1d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1b1fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1b1fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32900/0 (socket says 172.21.15.116:32900) 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 -- 172.21.15.116:0/50527791 learned_addr learned my addr 172.21.15.116:0/50527791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff557400640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1b220c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 -- 172.21.15.116:0/50527791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 msgr2=0x55fbb1b220c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1b220c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 -- 172.21.15.116:0/50527791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 msgr2=0x55fbb1b1d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1b1d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 -- 172.21.15.116:0/50527791 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbb1b22600 con 0x55fbb1aaa700 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff557400640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1b220c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:48:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556a00640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1b1d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff556000640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1b1fb70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff5400a31a0 tx=0x7ff5400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=246aa36c98746b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff544c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff544c00640 1 -- 172.21.15.116:0/50527791 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5400d1e60 con 0x55fbb1aaa700 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff544c00640 1 -- 172.21.15.116:0/50527791 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff540016db0 con 0x55fbb1aaa700 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.697+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff54009b630 con 0x55fbb1aaa700 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff544c00640 1 -- 172.21.15.116:0/50527791 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54009f080 con 0x55fbb1aaa700 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 10 client.27386 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55fbb1b237d0 con 0x55fbb1aaa700 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff544c00640 1 -- 172.21.15.116:0/50527791 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff54009b630 con 0x55fbb1aaa700 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff544c00640 1 -- 172.21.15.116:0/50527791 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff5400975a0 con 0x55fbb1aaa700 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 10 client.27386 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 10 client.27386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 20 client.27386 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1b240e0 0x55fbb1b26500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 4 client.27386 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fbb1a165c0 con 0x55fbb1b240e0 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.698+0000 7ff556a00640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1b240e0 0x55fbb1b26500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff556a00640 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1b240e0 0x55fbb1b26500 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ec5aae88b9a0a2a server_cookie=50b744811d0929ae in_seq=0 out_seq=0 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff544c00640 10 client.27386 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff544c00640 1 -- 172.21.15.116:0/50527791 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55fbb1a165c0 con 0x55fbb1b240e0 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff544c00640 10 client.27386 handle_command_reply: tid=0 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 client.27386 shutdown 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 msgr2=0x55fbb1b1fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1b1fb70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff5400a31a0 tx=0x7ff5400046c0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 shutdown_connections 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55fbb1b240e0 0x55fbb1b26500 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fbb1aaf870 0x55fbb1b220c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff544c00640 0 client.27386 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55fbb1aaa700 0x55fbb1b1fb70 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 --2- 172.21.15.116:0/50527791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbb1aa9d10 0x55fbb1b1d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 >> 172.21.15.116:0/50527791 conn(0x55fbb1aa5280 msgr2=0x55fbb1aa3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 shutdown_connections 2024-03-04T07:48:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 1 -- 172.21.15.116:0/50527791 wait complete. 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.699+0000 7ff560a52740 20 client.27386 trim_cache size 0 max 16384 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout: "a47c0f2d-127e-4051-bff5-9059a38fec4c": { 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "a47c0f2d-127e-4051-bff5-9059a38fec4c", 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:48:52.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:48:52.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:48:52.711 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:48:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff53c0a1e00 msgr2=0x7ff53c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff53c0a1e00 0x7ff53c0a42c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff550009fd0 tx=0x7ff5500096c0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff558104870 msgr2=0x7ff558199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff558104870 0x7ff558199fc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff54000e9c0 tx=0x7ff54000edc0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 shutdown_connections 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5581071a0 0x7ff5581a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff558104870 0x7ff558199fc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff53c0a1e00 0x7ff53c0a42c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 --2- 172.21.15.116:0/2285299628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff558101f40 0x7ff558199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 >> 172.21.15.116:0/2285299628 conn(0x7ff5580fb950 msgr2=0x7ff5581016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 shutdown_connections 2024-03-04T07:48:52.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:48:52.700+0000 7ff557e00640 1 -- 172.21.15.116:0/2285299628 wait complete. 2024-03-04T07:48:52.717 INFO:tasks.cephfs.filesystem:scrub status for tag:a47c0f2d-127e-4051-bff5-9059a38fec4c - {'path': '/', 'tag': 'a47c0f2d-127e-4051-bff5-9059a38fec4c', 'options': 'recursive,force'} 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 -- 172.21.15.116:0/2188974544 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c109600 msgr2=0x7ff53c10fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 --2- 172.21.15.116:0/2188974544 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c109600 0x7ff53c10fe90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff534009ca0 tx=0x7ff53402c0f0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 -- 172.21.15.116:0/2188974544 shutdown_connections 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 --2- 172.21.15.116:0/2188974544 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c109600 0x7ff53c10fe90 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 --2- 172.21.15.116:0/2188974544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c108c40 0x7ff53c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 --2- 172.21.15.116:0/2188974544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff53c107a50 0x7ff53c107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 -- 172.21.15.116:0/2188974544 >> 172.21.15.116:0/2188974544 conn(0x7ff53c076a10 msgr2=0x7ff53c078e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 -- 172.21.15.116:0/2188974544 shutdown_connections 2024-03-04T07:48:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.215+0000 7ff542200640 1 -- 172.21.15.116:0/2188974544 wait complete. 2024-03-04T07:48:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.216+0000 7ff542200640 1 Processor -- start 2024-03-04T07:48:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff542200640 1 -- start start 2024-03-04T07:48:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff542200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c107a50 0x7ff53c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff542200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c108c40 0x7ff53c19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff542200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff53c109600 0x7ff53c1a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff542200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1136d0 con 0x7ff53c108c40 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff542200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c113550 con 0x7ff53c107a50 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff542200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c113850 con 0x7ff53c109600 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff533e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c108c40 0x7ff53c19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff540e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c107a50 0x7ff53c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff540e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c107a50 0x7ff53c19dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32928/0 (socket says 172.21.15.116:32928) 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff540e00640 1 -- 172.21.15.116:0/2281087211 learned_addr learned my addr 172.21.15.116:0/2281087211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.217+0000 7ff541800640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff53c109600 0x7ff53c1a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff533e00640 1 -- 172.21.15.116:0/2281087211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff53c109600 msgr2=0x7ff53c1a5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff533e00640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff53c109600 0x7ff53c1a5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff533e00640 1 -- 172.21.15.116:0/2281087211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c107a50 msgr2=0x7ff53c19dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff533e00640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c107a50 0x7ff53c19dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff533e00640 1 -- 172.21.15.116:0/2281087211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c1a5910 con 0x7ff53c108c40 2024-03-04T07:48:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff540e00640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c107a50 0x7ff53c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff533e00640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c108c40 0x7ff53c19e240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e9f0 tx=0x7ff52400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff53a000640 1 -- 172.21.15.116:0/2281087211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52400ccc0 con 0x7ff53c108c40 2024-03-04T07:48:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff53a000640 1 -- 172.21.15.116:0/2281087211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff524010910 con 0x7ff53c108c40 2024-03-04T07:48:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff53a000640 1 -- 172.21.15.116:0/2281087211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524018960 con 0x7ff53c108c40 2024-03-04T07:48:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.218+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c1a5c00 con 0x7ff53c108c40 2024-03-04T07:48:53.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.219+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c1a6100 con 0x7ff53c108c40 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.222+0000 7ff53a000640 1 -- 172.21.15.116:0/2281087211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff524010ac0 con 0x7ff53c108c40 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.222+0000 7ff5463f0740 1 Processor -- start 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.222+0000 7ff5463f0740 1 -- start start 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbe8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe9400 0x562fcdbedf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff5463f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fcdb4e370 con 0x562fcdbe8a10 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff5463f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fcdaf7550 con 0x562fcdbe9400 2024-03-04T07:48:53.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff5463f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fcdb4a410 con 0x562fcdbee570 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff541800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbe8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe9400 0x562fcdbedf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe9400 0x562fcdbedf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32938/0 (socket says 172.21.15.116:32938) 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 -- 172.21.15.116:0/2840521800 learned_addr learned my addr 172.21.15.116:0/2840521800 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff533e00640 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 -- 172.21.15.116:0/2840521800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 msgr2=0x562fcdbf0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 -- 172.21.15.116:0/2840521800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 msgr2=0x562fcdbe8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbe8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff540e00640 1 -- 172.21.15.116:0/2840521800 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fcdb0b3c0 con 0x562fcdbe9400 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.223+0000 7ff533e00640 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.224+0000 7ff53a000640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5200a1d70 0x7ff5200a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.224+0000 7ff540e00640 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe9400 0x562fcdbedf00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0115f0 tx=0x7ff52c011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34abcf04a499de20 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.224+0000 7ff53a000640 1 -- 172.21.15.116:0/2281087211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff5240ce200 con 0x7ff53c108c40 2024-03-04T07:48:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.224+0000 7ff532000640 1 -- 172.21.15.116:0/2840521800 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c005020 con 0x562fcdbe9400 2024-03-04T07:48:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.224+0000 7ff541800640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5200a1d70 0x7ff5200a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.224+0000 7ff532000640 1 -- 172.21.15.116:0/2840521800 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff52c0047a0 con 0x562fcdbe9400 2024-03-04T07:48:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.224+0000 7ff532000640 1 -- 172.21.15.116:0/2840521800 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c00f9c0 con 0x562fcdbe9400 2024-03-04T07:48:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff541800640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5200a1d70 0x7ff5200a4230 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff53402c630 tx=0x7ff534032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 -- 172.21.15.116:0/2840521800 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe9400 msgr2=0x562fcdbedf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe9400 0x562fcdbedf00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0115f0 tx=0x7ff52c011ab0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 -- 172.21.15.116:0/2840521800 shutdown_connections 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe9400 0x562fcdbedf00 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2840521800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbe8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 -- 172.21.15.116:0/2840521800 >> 172.21.15.116:0/2840521800 conn(0x562fcdbe3f80 msgr2=0x562fcdbe4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 -- 172.21.15.116:0/2840521800 shutdown_connections 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.225+0000 7ff5463f0740 1 -- 172.21.15.116:0/2840521800 wait complete. 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 -- start start 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 Processor -- start 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbf2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe9400 0x562fcdbf32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdbfa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fcdbf1520 con 0x562fcdbe8a10 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fcdb0a9b0 con 0x562fcdbee570 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fcdbf16a0 con 0x562fcdbe9400 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff5463f0740 10 client.0 _ll_register_callbacks cb 0x562fcdb46340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff540e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbf2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.226+0000 7ff540e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbf2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33224/0 (socket says 172.21.15.116:33224) 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff540e00640 1 -- 172.21.15.116:0/2914959529 learned_addr learned my addr 172.21.15.116:0/2914959529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff541800640 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdbfa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff533e00640 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe9400 0x562fcdbf32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff540e00640 1 -- 172.21.15.116:0/2914959529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe9400 msgr2=0x562fcdbf32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff540e00640 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe9400 0x562fcdbf32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff540e00640 1 -- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 msgr2=0x562fcdbfa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff540e00640 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdbfa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff540e00640 1 -- 172.21.15.116:0/2914959529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562fcd863210 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff540e00640 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbf2da0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff52c01b040 tx=0x7ff52c005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342dd2a8880bd707 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff52aa00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.227+0000 7ff52aa00640 1 -- 172.21.15.116:0/2914959529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c028070 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff52aa00640 1 -- 172.21.15.116:0/2914959529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff52c00f700 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff52c0047a0 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff52aa00640 1 -- 172.21.15.116:0/2914959529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52c033050 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 10 client.17268 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562fcd92f620 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff52aa00640 1 -- 172.21.15.116:0/2914959529 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff52c0047a0 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff52aa00640 1 -- 172.21.15.116:0/2914959529 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff52c024960 con 0x562fcdbe8a10 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 10 client.17268 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 10 client.17268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 20 client.17268 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:53.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdbfe900 0x562fcdc00cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 4 client.17268 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.228+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562fcdb5b8b0 con 0x562fcdbfe900 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.229+0000 7ff533e00640 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdbfe900 0x562fcdc00cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.229+0000 7ff533e00640 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdbfe900 0x562fcdc00cf0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=470bbcba8af94ce2 server_cookie=24035d6fa1c9b8f8 in_seq=0 out_seq=0 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.229+0000 7ff52aa00640 10 client.17268 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff52aa00640 1 -- 172.21.15.116:0/2914959529 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562fcdb5b8b0 con 0x562fcdbfe900 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff52aa00640 10 client.17268 handle_command_reply: tid=0 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 client.17268 shutdown 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 msgr2=0x562fcdbf2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbf2da0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff52c01b040 tx=0x7ff52c005230 comp rx=0 tx=0).stop 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 shutdown_connections 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdbfe900 0x562fcdc00cf0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdbfa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe9400 0x562fcdbf32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 --2- 172.21.15.116:0/2914959529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe8a10 0x562fcdbf2da0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff52aa00640 0 client.17268 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.230+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 >> 172.21.15.116:0/2914959529 conn(0x562fcdbe3f80 msgr2=0x562fcdbe3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.231+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 shutdown_connections 2024-03-04T07:48:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.231+0000 7ff5463f0740 1 -- 172.21.15.116:0/2914959529 wait complete. 2024-03-04T07:48:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.231+0000 7ff5463f0740 20 client.17268 trim_cache size 0 max 16384 2024-03-04T07:48:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.245+0000 7ff5463f0740 1 Processor -- start 2024-03-04T07:48:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff5463f0740 1 -- start start 2024-03-04T07:48:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe8a10 0x562fcdbe78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbfa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdc055a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff5463f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1acea0 con 0x562fcdbe9400 2024-03-04T07:48:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff5463f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1af480 con 0x562fcdbee570 2024-03-04T07:48:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff5463f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1ad4a0 con 0x562fcdbe8a10 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff533e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbfa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff541800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdc055a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff541800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdc055a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32958/0 (socket says 172.21.15.116:32958) 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff541800640 1 -- 172.21.15.116:0/1635196002 learned_addr learned my addr 172.21.15.116:0/1635196002 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.246+0000 7ff540e00640 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe8a10 0x562fcdbe78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff541800640 1 -- 172.21.15.116:0/1635196002 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe8a10 msgr2=0x562fcdbe78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff541800640 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe8a10 0x562fcdbe78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff541800640 1 -- 172.21.15.116:0/1635196002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 msgr2=0x562fcdbfa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff541800640 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbfa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff541800640 1 -- 172.21.15.116:0/1635196002 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fcdbe4cf0 con 0x562fcdbee570 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff541800640 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdc055a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff53402c700 tx=0x7ff534041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a0760a12c95a412 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff531600640 1 -- 172.21.15.116:0/1635196002 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534039040 con 0x562fcdbee570 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff531600640 1 -- 172.21.15.116:0/1635196002 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff534003e50 con 0x562fcdbee570 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.247+0000 7ff531600640 1 -- 172.21.15.116:0/1635196002 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53404e460 con 0x562fcdbee570 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 -- 172.21.15.116:0/1635196002 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 msgr2=0x562fcdc055a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdc055a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff53402c700 tx=0x7ff534041000 comp rx=0 tx=0).stop 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 -- 172.21.15.116:0/1635196002 shutdown_connections 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbee570 0x562fcdc055a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbfa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 --2- 172.21.15.116:0/1635196002 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbe8a10 0x562fcdbe78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 -- 172.21.15.116:0/1635196002 >> 172.21.15.116:0/1635196002 conn(0x562fcdbe3f80 msgr2=0x562fcdbe71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 -- 172.21.15.116:0/1635196002 shutdown_connections 2024-03-04T07:48:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 -- 172.21.15.116:0/1635196002 wait complete. 2024-03-04T07:48:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.248+0000 7ff5463f0740 1 -- start start 2024-03-04T07:48:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 1 Processor -- start 2024-03-04T07:48:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe8a10 0x562fcdbf79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbf7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1adb00 con 0x562fcdbe9400 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1aee20 con 0x562fcdbe8a10 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1ae160 con 0x562fcdbee570 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff5463f0740 10 client.0 _ll_register_callbacks cb 0x7ff524005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff533e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbf7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff540e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe8a10 0x562fcdbf79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff540e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe8a10 0x562fcdbf79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:32970/0 (socket says 172.21.15.116:32970) 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff540e00640 1 -- 172.21.15.116:0/783328354 learned_addr learned my addr 172.21.15.116:0/783328354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.249+0000 7ff541800640 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff533e00640 1 -- 172.21.15.116:0/783328354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 msgr2=0x562fcdbf8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff533e00640 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff533e00640 1 -- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe8a10 msgr2=0x562fcdbf79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff533e00640 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe8a10 0x562fcdbf79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff533e00640 1 -- 172.21.15.116:0/783328354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562fcdbf9430 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff540e00640 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe8a10 0x562fcdbf79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff533e00640 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbf7ef0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff52400c6a0 tx=0x7ff5240d18d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efa04e044c5a201 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff52aa00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff52aa00640 1 -- 172.21.15.116:0/783328354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5240a3690 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff52aa00640 1 -- 172.21.15.116:0/783328354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff52409a3f0 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff534003e50 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff52aa00640 1 -- 172.21.15.116:0/783328354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52409a5f0 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff5463f0740 10 client.17274 fetch_fsmap learned FSMap version 19 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.250+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x562fcdc5a170 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff52aa00640 1 -- 172.21.15.116:0/783328354 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff534003e50 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff52aa00640 1 -- 172.21.15.116:0/783328354 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff524018b90 con 0x562fcdbe9400 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff5463f0740 10 client.17274 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:48:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff5463f0740 10 client.17274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:48:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff5463f0740 20 client.17274 populate_metadata read hostname 'smithi116' 2024-03-04T07:48:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff5463f0740 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdc5ab20 0x562fcdc5cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:48:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff5463f0740 4 client.17274 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:48:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562fcdb55c00 con 0x562fcdc5ab20 2024-03-04T07:48:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff540e00640 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdc5ab20 0x562fcdc5cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:48:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff540e00640 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdc5ab20 0x562fcdc5cf40 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27eaa1439a58d441 server_cookie=60437ddf5ced2a72 in_seq=0 out_seq=0 2024-03-04T07:48:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.251+0000 7ff52aa00640 10 client.17274 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:52 smithi116 ceph-mon[32150]: pgmap v644: 97 pgs: 97 active+clean; 2.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:52 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:48:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:52 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:52 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:48:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:52 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:52 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:48:53.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff52aa00640 1 -- 172.21.15.116:0/783328354 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562fcdb55c00 con 0x562fcdc5ab20 2024-03-04T07:48:53.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff52aa00640 10 client.17274 handle_command_reply: tid=0 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 client.17274 shutdown 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 msgr2=0x562fcdbf7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbf7ef0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff52400c6a0 tx=0x7ff5240d18d0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 shutdown_connections 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x562fcdc5ab20 0x562fcdc5cf40 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562fcdbee570 0x562fcdbf8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562fcdbe9400 0x562fcdbf7ef0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff52aa00640 0 client.17274 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 --2- 172.21.15.116:0/783328354 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x562fcdbe8a10 0x562fcdbf79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.266+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 >> 172.21.15.116:0/783328354 conn(0x562fcdbe3f80 msgr2=0x562fcdbe7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.267+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 shutdown_connections 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.267+0000 7ff5463f0740 1 -- 172.21.15.116:0/783328354 wait complete. 2024-03-04T07:48:53.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.267+0000 7ff5463f0740 20 client.17274 trim_cache size 0 max 16384 2024-03-04T07:48:53.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:48:53.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:48:53.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:48:53.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:48:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.267+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5200a1d70 msgr2=0x7ff5200a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.267+0000 7ff542200640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5200a1d70 0x7ff5200a4230 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff53402c630 tx=0x7ff534032590 comp rx=0 tx=0).stop 2024-03-04T07:48:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.267+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c108c40 msgr2=0x7ff53c19e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:48:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.267+0000 7ff542200640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c108c40 0x7ff53c19e240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e9f0 tx=0x7ff52400eeb0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 shutdown_connections 2024-03-04T07:48:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff5200a1d70 0x7ff5200a4230 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff53c109600 0x7ff53c1a5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff53c108c40 0x7ff53c19e240 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 --2- 172.21.15.116:0/2281087211 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff53c107a50 0x7ff53c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:48:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 >> 172.21.15.116:0/2281087211 conn(0x7ff53c076a10 msgr2=0x7ff53c10bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:48:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 shutdown_connections 2024-03-04T07:48:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:48:53.268+0000 7ff542200640 1 -- 172.21.15.116:0/2281087211 wait complete. 2024-03-04T07:48:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:52 smithi152 ceph-mon[37154]: pgmap v644: 97 pgs: 97 active+clean; 2.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:53.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:52 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:48:53.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:52 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:53.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:52 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:48:53.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:52 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:53.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:52 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:48:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:52 smithi133 ceph-mon[36505]: pgmap v644: 97 pgs: 97 active+clean; 2.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:48:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:52 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:48:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:52 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:52 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:48:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:52 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:48:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:52 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:48:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:54 smithi116 ceph-mon[32150]: pgmap v645: 97 pgs: 97 active+clean; 2.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:48:54.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:54 smithi152 ceph-mon[37154]: pgmap v645: 97 pgs: 97 active+clean; 2.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:48:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:54 smithi133 ceph-mon[36505]: pgmap v645: 97 pgs: 97 active+clean; 2.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:48:57.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:56 smithi133 ceph-mon[36505]: pgmap v646: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:48:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:56 smithi116 ceph-mon[32150]: pgmap v646: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:48:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:56 smithi152 ceph-mon[37154]: pgmap v646: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:48:59.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:48:58 smithi133 ceph-mon[36505]: pgmap v647: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:48:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:48:58 smithi116 ceph-mon[32150]: pgmap v647: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:48:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:48:58 smithi152 ceph-mon[37154]: pgmap v647: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:49:01.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:00 smithi133 ceph-mon[36505]: pgmap v648: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:49:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:00 smithi116 ceph-mon[32150]: pgmap v648: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:49:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:00 smithi152 ceph-mon[37154]: pgmap v648: 97 pgs: 97 active+clean; 2.1 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:49:03.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:02 smithi133 ceph-mon[36505]: pgmap v649: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:02 smithi116 ceph-mon[32150]: pgmap v649: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:02 smithi152 ceph-mon[37154]: pgmap v649: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:04 smithi116 ceph-mon[32150]: pgmap v650: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:04 smithi152 ceph-mon[37154]: pgmap v650: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:05.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:04 smithi133 ceph-mon[36505]: pgmap v650: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.410+0000 7f47f8a00640 1 -- 172.21.15.116:0/3861681677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 msgr2=0x7f47f4104cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.410+0000 7f47f8a00640 1 --2- 172.21.15.116:0/3861681677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 0x7f47f4104cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009900 tx=0x7f47e002bc30 comp rx=0 tx=0).stop 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.410+0000 7f47f8a00640 1 -- 172.21.15.116:0/3861681677 shutdown_connections 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.410+0000 7f47f8a00640 1 --2- 172.21.15.116:0/3861681677 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 0x7f47f410ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.410+0000 7f47f8a00640 1 --2- 172.21.15.116:0/3861681677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 0x7f47f4104cc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.410+0000 7f47f8a00640 1 --2- 172.21.15.116:0/3861681677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47f4103650 0x7f47f4103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.411+0000 7f47f8a00640 1 -- 172.21.15.116:0/3861681677 >> 172.21.15.116:0/3861681677 conn(0x7f47f40febe0 msgr2=0x7f47f4101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.411+0000 7f47f8a00640 1 -- 172.21.15.116:0/3861681677 shutdown_connections 2024-03-04T07:49:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.411+0000 7f47f8a00640 1 -- 172.21.15.116:0/3861681677 wait complete. 2024-03-04T07:49:06.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 Processor -- start 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 -- start start 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47f4103650 0x7f47f4199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 0x7f47f4199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 0x7f47f41a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f410f2e0 con 0x7f47f4104840 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f410f160 con 0x7f47f4105200 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f8a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f410f460 con 0x7f47f4103650 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f3200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 0x7f47f41a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f3200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 0x7f47f41a0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46166/0 (socket says 172.21.15.116:46166) 2024-03-04T07:49:06.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 0x7f47f4199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f3200640 1 -- 172.21.15.116:0/971913936 learned_addr learned my addr 172.21.15.116:0/971913936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.412+0000 7f47f2800640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47f4103650 0x7f47f4199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f3200640 1 -- 172.21.15.116:0/971913936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47f4103650 msgr2=0x7f47f4199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f3200640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47f4103650 0x7f47f4199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f3200640 1 -- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 msgr2=0x7f47f4199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f3200640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 0x7f47f4199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f3200640 1 -- 172.21.15.116:0/971913936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f41a15a0 con 0x7f47f4105200 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f3200640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 0x7f47f41a0f30 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00efd0 tx=0x7f47ec00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47eb400640 1 -- 172.21.15.116:0/971913936 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec003b70 con 0x7f47f4105200 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f41a1890 con 0x7f47f4105200 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47eb400640 1 -- 172.21.15.116:0/971913936 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47ec003d10 con 0x7f47f4105200 2024-03-04T07:49:06.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47eb400640 1 -- 172.21.15.116:0/971913936 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec0109f0 con 0x7f47f4105200 2024-03-04T07:49:06.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.413+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f41a1db0 con 0x7f47f4105200 2024-03-04T07:49:06.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.416+0000 7f47fc015740 1 Processor -- start 2024-03-04T07:49:06.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47eb400640 1 -- 172.21.15.116:0/971913936 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f47ec004020 con 0x7f47f4105200 2024-03-04T07:49:06.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47fc015740 1 -- start start 2024-03-04T07:49:06.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47fc015740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c025e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47fc015740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c02af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47fc015740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c02da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47fc015740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56028bf8b370 con 0x56028c02b570 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47fc015740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56028bf34550 con 0x56028c026400 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47fc015740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56028bf87410 con 0x56028c025a10 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47f2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c02af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47f2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c02af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46182/0 (socket says 172.21.15.116:46182) 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.417+0000 7f47f2800640 1 -- 172.21.15.116:0/1982447205 learned_addr learned my addr 172.21.15.116:0/1982447205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c025e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c02da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f2800640 1 -- 172.21.15.116:0/1982447205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 msgr2=0x56028c025e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f2800640 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c025e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f2800640 1 -- 172.21.15.116:0/1982447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 msgr2=0x56028c02da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f2800640 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c02da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f2800640 1 -- 172.21.15.116:0/1982447205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56028bf483c0 con 0x56028c026400 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c02da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47f2800640 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c02af00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009ca0 tx=0x7f47e4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f614fe51e0553bf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47eb400640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47d80a1d70 0x7f47d80a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.418+0000 7f47df400640 1 -- 172.21.15.116:0/1982447205 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4004220 con 0x56028c026400 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47df400640 1 -- 172.21.15.116:0/1982447205 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47e40043c0 con 0x56028c026400 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47df400640 1 -- 172.21.15.116:0/1982447205 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4019910 con 0x56028c026400 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47eb400640 1 -- 172.21.15.116:0/971913936 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f47ec0ce220 con 0x7f47f4105200 2024-03-04T07:49:06.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47f1e00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47d80a1d70 0x7f47d80a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47fc015740 1 -- 172.21.15.116:0/1982447205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 msgr2=0x56028c02af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47fc015740 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c02af00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009ca0 tx=0x7f47e4011810 comp rx=0 tx=0).stop 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47f1e00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47d80a1d70 0x7f47d80a4230 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f47e002c7b0 tx=0x7f47e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47fc015740 1 -- 172.21.15.116:0/1982447205 shutdown_connections 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47fc015740 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c02da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47fc015740 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c02af00 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47fc015740 1 --2- 172.21.15.116:0/1982447205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c025e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.419+0000 7f47fc015740 1 -- 172.21.15.116:0/1982447205 >> 172.21.15.116:0/1982447205 conn(0x56028c020f80 msgr2=0x56028c021380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.420+0000 7f47fc015740 1 -- 172.21.15.116:0/1982447205 shutdown_connections 2024-03-04T07:49:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.420+0000 7f47fc015740 1 -- 172.21.15.116:0/1982447205 wait complete. 2024-03-04T07:49:06.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.420+0000 7f47fc015740 1 -- start start 2024-03-04T07:49:06.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.420+0000 7f47fc015740 1 Processor -- start 2024-03-04T07:49:06.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47fc015740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c02fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47fc015740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c030220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47fc015740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c037260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47fc015740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56028c02e520 con 0x56028c02b570 2024-03-04T07:49:06.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47fc015740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56028bf479b0 con 0x56028c026400 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47fc015740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56028c02e6a0 con 0x56028c025a10 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47fc015740 10 client.0 _ll_register_callbacks cb 0x56028bf83340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c030220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c037260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c030220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46196/0 (socket says 172.21.15.116:46196) 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 -- 172.21.15.116:0/1131128781 learned_addr learned my addr 172.21.15.116:0/1131128781 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f2800640 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c02fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 -- 172.21.15.116:0/1131128781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 msgr2=0x56028c02fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c02fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 -- 172.21.15.116:0/1131128781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 msgr2=0x56028c037260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c037260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f1e00640 1 -- 172.21.15.116:0/1131128781 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56028bca0210 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.421+0000 7f47f2800640 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c02fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c030220 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009900 tx=0x7f47e002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3972bba651312fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47d3e00640 1 -- 172.21.15.116:0/1131128781 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e004f020 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47d3e00640 1 -- 172.21.15.116:0/1131128781 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f47e0003970 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47e40043c0 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47d3e00640 1 -- 172.21.15.116:0/1131128781 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e0047040 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47fc015740 10 client.27416 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.422+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56028bd6c620 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47d3e00640 1 -- 172.21.15.116:0/1131128781 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47e40043c0 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47d3e00640 1 -- 172.21.15.116:0/1131128781 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f47e004a030 con 0x56028c026400 2024-03-04T07:49:06.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47fc015740 10 client.27416 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47fc015740 10 client.27416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47fc015740 20 client.27416 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47fc015740 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c03bb30 0x56028c03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47fc015740 4 client.27416 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56028bf988b0 con 0x56028c03bb30 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.423+0000 7f47f2800640 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c03bb30 0x56028c03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.424+0000 7f47d3e00640 10 client.27416 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.424+0000 7f47f2800640 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c03bb30 0x56028c03dfd0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a4460b71542fd16 server_cookie=6b8770ad63d1114 in_seq=0 out_seq=0 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47d3e00640 1 -- 172.21.15.116:0/1131128781 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56028bf988b0 con 0x56028c03bb30 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47d3e00640 10 client.27416 handle_command_reply: tid=0 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 client.27416 shutdown 2024-03-04T07:49:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 msgr2=0x56028c030220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c030220 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009900 tx=0x7f47e002bc30 comp rx=0 tx=0).stop 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 shutdown_connections 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c03bb30 0x56028c03dfd0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c02b570 0x56028c037260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c026400 0x56028c030220 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 --2- 172.21.15.116:0/1131128781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c025a10 0x56028c02fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47d3e00640 0 client.27416 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 >> 172.21.15.116:0/1131128781 conn(0x56028c020f80 msgr2=0x56028c020200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 shutdown_connections 2024-03-04T07:49:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 1 -- 172.21.15.116:0/1131128781 wait complete. 2024-03-04T07:49:06.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.425+0000 7f47fc015740 20 client.27416 trim_cache size 0 max 16384 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.440+0000 7f47fc015740 1 Processor -- start 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.440+0000 7f47fc015740 1 -- start start 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47fc015740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c01f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47fc015740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c037220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47fc015740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c01feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47fc015740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f41a8a70 con 0x56028c026400 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47fc015740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f41ab050 con 0x56028c025a10 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47fc015740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f41a9070 con 0x56028c02b570 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c01f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.442 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c01f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46204/0 (socket says 172.21.15.116:46204) 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 -- 172.21.15.116:0/1037432586 learned_addr learned my addr 172.21.15.116:0/1037432586 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c037220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f3200640 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c01feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 -- 172.21.15.116:0/1037432586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 msgr2=0x56028c01feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c01feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 -- 172.21.15.116:0/1037432586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 msgr2=0x56028c037220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c037220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.441+0000 7f47f2800640 1 -- 172.21.15.116:0/1037432586 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56028c021cf0 con 0x56028c025a10 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47f2800640 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c01f970 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009710 tx=0x7f47e4009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53e2e52bdadd1f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:06.443 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47dea00640 1 -- 172.21.15.116:0/1037432586 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4032070 con 0x56028c025a10 2024-03-04T07:49:06.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47dea00640 1 -- 172.21.15.116:0/1037432586 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47e402c2d0 con 0x56028c025a10 2024-03-04T07:49:06.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47dea00640 1 -- 172.21.15.116:0/1037432586 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4004760 con 0x56028c025a10 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47fc015740 1 -- 172.21.15.116:0/1037432586 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 msgr2=0x56028c01f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47fc015740 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c01f970 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009710 tx=0x7f47e4009680 comp rx=0 tx=0).stop 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47fc015740 1 -- 172.21.15.116:0/1037432586 shutdown_connections 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47fc015740 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c01feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47fc015740 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c037220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47fc015740 1 --2- 172.21.15.116:0/1037432586 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c01f970 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.442+0000 7f47fc015740 1 -- 172.21.15.116:0/1037432586 >> 172.21.15.116:0/1037432586 conn(0x56028c020f80 msgr2=0x56028c027640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 -- 172.21.15.116:0/1037432586 shutdown_connections 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 -- 172.21.15.116:0/1037432586 wait complete. 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 -- start start 2024-03-04T07:49:06.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 Processor -- start 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c035190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c0356d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c0366e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f41a96d0 con 0x56028c026400 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f41aa9f0 con 0x56028c025a10 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f41a9d30 con 0x56028c02b570 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.443+0000 7f47fc015740 10 client.0 _ll_register_callbacks cb 0x7f47e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c035190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c035190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:46216/0 (socket says 172.21.15.116:46216) 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 -- 172.21.15.116:0/1860847384 learned_addr learned my addr 172.21.15.116:0/1860847384 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f3200640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c0366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c0356d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 -- 172.21.15.116:0/1860847384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 msgr2=0x56028c0366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c0366e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 -- 172.21.15.116:0/1860847384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 msgr2=0x56028c0356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c0356d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 -- 172.21.15.116:0/1860847384 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56028c036cd0 con 0x56028c025a10 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c0356d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47f2800640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c035190 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f47e4014040 tx=0x7f47e4009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79de58130158f3be server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.444+0000 7f47d3e00640 1 -- 172.21.15.116:0/1860847384 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4032050 con 0x56028c025a10 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47e402c2d0 con 0x56028c025a10 2024-03-04T07:49:06.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47d3e00640 1 -- 172.21.15.116:0/1860847384 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f47e4005250 con 0x56028c025a10 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47d3e00640 1 -- 172.21.15.116:0/1860847384 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e4004290 con 0x56028c025a10 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 10 client.27422 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56028c0a04a0 con 0x56028c025a10 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47d3e00640 1 -- 172.21.15.116:0/1860847384 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47e402c2d0 con 0x56028c025a10 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47d3e00640 1 -- 172.21.15.116:0/1860847384 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f47e403e030 con 0x56028c025a10 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 10 client.27422 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 10 client.27422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 20 client.27422 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c0a0c50 0x56028c0a3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 4 client.27422 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:49:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56028bf92c00 con 0x56028c0a0c50 2024-03-04T07:49:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.445+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c0a0c50 0x56028c0a3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.446+0000 7f47f1e00640 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c0a0c50 0x56028c0a3070 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e87a974deee06726 server_cookie=60494a65d6431cd9 in_seq=0 out_seq=0 2024-03-04T07:49:06.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.446+0000 7f47d3e00640 10 client.27422 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:06.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47d3e00640 1 -- 172.21.15.116:0/1860847384 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56028bf92c00 con 0x56028c0a0c50 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47d3e00640 10 client.27422 handle_command_reply: tid=0 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 client.27422 shutdown 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 msgr2=0x56028c035190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c035190 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f47e4014040 tx=0x7f47e4009c20 comp rx=0 tx=0).stop 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 shutdown_connections 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56028c0a0c50 0x56028c0a3070 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56028c02b570 0x56028c0366e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56028c026400 0x56028c0356d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47d3e00640 0 client.27422 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 --2- 172.21.15.116:0/1860847384 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56028c025a10 0x56028c035190 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 >> 172.21.15.116:0/1860847384 conn(0x56028c020f80 msgr2=0x56028c0241e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.461+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 shutdown_connections 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.462+0000 7f47fc015740 1 -- 172.21.15.116:0/1860847384 wait complete. 2024-03-04T07:49:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.462+0000 7f47fc015740 20 client.27422 trim_cache size 0 max 16384 2024-03-04T07:49:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:49:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:49:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:49:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.462+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47d80a1d70 msgr2=0x7f47d80a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.462+0000 7f47f8a00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47d80a1d70 0x7f47d80a4230 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f47e002c7b0 tx=0x7f47e0005fd0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.462+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 msgr2=0x7f47f41a0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.462+0000 7f47f8a00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 0x7f47f41a0f30 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00efd0 tx=0x7f47ec00c6a0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 shutdown_connections 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f47f4105200 0x7f47f41a0f30 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f4104840 0x7f47f4199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f47d80a1d70 0x7f47d80a4230 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 --2- 172.21.15.116:0/971913936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47f4103650 0x7f47f4199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:06.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 >> 172.21.15.116:0/971913936 conn(0x7f47f40febe0 msgr2=0x7f47f4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:06.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 shutdown_connections 2024-03-04T07:49:06.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:06.463+0000 7f47f8a00640 1 -- 172.21.15.116:0/971913936 wait complete. 2024-03-04T07:49:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:06 smithi116 ceph-mon[32150]: pgmap v651: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:06 smithi152 ceph-mon[37154]: pgmap v651: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:07.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:06 smithi133 ceph-mon[36505]: pgmap v651: 97 pgs: 97 active+clean; 2.2 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:08 smithi116 ceph-mon[32150]: pgmap v652: 97 pgs: 97 active+clean; 2.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:49:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:08 smithi152 ceph-mon[37154]: pgmap v652: 97 pgs: 97 active+clean; 2.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:49:09.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:08 smithi133 ceph-mon[36505]: pgmap v652: 97 pgs: 97 active+clean; 2.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:49:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:10 smithi116 ceph-mon[32150]: pgmap v653: 97 pgs: 97 active+clean; 2.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:49:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:10 smithi152 ceph-mon[37154]: pgmap v653: 97 pgs: 97 active+clean; 2.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:49:11.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:10 smithi133 ceph-mon[36505]: pgmap v653: 97 pgs: 97 active+clean; 2.2 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:49:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:12 smithi116 ceph-mon[32150]: pgmap v654: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:12 smithi152 ceph-mon[37154]: pgmap v654: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:12 smithi133 ceph-mon[36505]: pgmap v654: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:375/600: open fd = 3 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:376/600: open fd = 3 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:377/600: open fd = 3 2024-03-04T07:49:15.084 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:378/600: open fd = 3 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:379/600: open fd = 3 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:380/600: open fd = 3 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.085 INFO:tasks.workunit.client.0.smithi116.stdout:381/600: open fd = 3 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:382/600: open fd = 3 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:383/600: open fd = 3 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:384/600: open fd = 3 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.086 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:385/600: open fd = 3 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:386/600: open fd = 3 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:387/600: open fd = 3 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:388/600: open fd = 3 2024-03-04T07:49:15.087 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:389/600: open fd = 3 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:390/600: open fd = 3 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:391/600: open fd = 3 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.088 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:392/600: open fd = 3 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:393/600: open fd = 3 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:394/600: open fd = 3 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.089 INFO:tasks.workunit.client.0.smithi116.stdout:395/600: open fd = 3 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:396/600: open fd = 3 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:397/600: open fd = 3 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:398/600: open fd = 3 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.090 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:399/600: open fd = 3 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:400/600: open fd = 3 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:401/600: open fd = 3 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:402/600: open fd = 3 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.091 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:403/600: open fd = 3 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:404/600: open fd = 3 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.092 INFO:tasks.workunit.client.0.smithi116.stdout:405/600: open fd = 3 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:406/600: open fd = 3 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:407/600: open fd = 3 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:408/600: open fd = 3 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.093 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:409/600: open fd = 3 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:410/600: open fd = 3 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:411/600: open fd = 3 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:412/600: open fd = 3 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.094 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:413/600: open fd = 3 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:414/600: open fd = 3 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:415/600: open fd = 3 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:416/600: open fd = 3 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.095 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:417/600: open fd = 3 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:418/600: open fd = 3 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:419/600: open fd = 3 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.096 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:420/600: open fd = 3 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:421/600: open fd = 3 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:422/600: open fd = 3 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:423/600: open fd = 3 2024-03-04T07:49:15.097 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:424/600: open fd = 3 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:425/600: open fd = 3 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:426/600: open fd = 3 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.098 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:427/600: open fd = 3 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:428/600: open fd = 3 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:429/600: open fd = 3 2024-03-04T07:49:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.100 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.100 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.100 INFO:tasks.workunit.client.0.smithi116.stdout:430/600: open fd = 3 2024-03-04T07:49:15.100 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.100 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.104 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.104 INFO:tasks.workunit.client.0.smithi116.stdout:431/600: open fd = 3 2024-03-04T07:49:15.104 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.104 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.104 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.105 INFO:tasks.workunit.client.0.smithi116.stdout:432/600: open fd = 3 2024-03-04T07:49:15.105 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.105 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.105 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.105 INFO:tasks.workunit.client.0.smithi116.stdout:433/600: open fd = 3 2024-03-04T07:49:15.105 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:434/600: open fd = 3 2024-03-04T07:49:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.106 INFO:tasks.workunit.client.0.smithi116.stdout:435/600: open fd = 3 2024-03-04T07:49:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:436/600: open fd = 3 2024-03-04T07:49:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:49:15.107 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:49:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:14 smithi116 ceph-mon[32150]: pgmap v655: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:15.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:14 smithi152 ceph-mon[37154]: pgmap v655: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:15.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:14 smithi133 ceph-mon[36505]: pgmap v655: 97 pgs: 97 active+clean; 2.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:17.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:16 smithi116 ceph-mon[32150]: pgmap v656: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:49:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:16 smithi152 ceph-mon[37154]: pgmap v656: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:49:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:16 smithi133 ceph-mon[36505]: pgmap v656: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:49:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:18 smithi116 ceph-mon[32150]: pgmap v657: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:18 smithi152 ceph-mon[37154]: pgmap v657: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:19.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:18 smithi133 ceph-mon[36505]: pgmap v657: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.603+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2412364733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f0102820 msgr2=0x7fa9f0104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.603+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2412364733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f0102820 0x7fa9f0104c90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009900 tx=0x7fa9d802bc50 comp rx=0 tx=0).stop 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.604+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2412364733 shutdown_connections 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.604+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2412364733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f01051d0 0x7fa9f0105650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.604+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2412364733 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f0102820 0x7fa9f0104c90 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.604+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2412364733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9f00ffeb0 0x7fa9f01022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.604+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2412364733 >> 172.21.15.116:0/2412364733 conn(0x7fa9f00fbab0 msgr2=0x7fa9f00fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:19.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.604+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2412364733 shutdown_connections 2024-03-04T07:49:19.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.604+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2412364733 wait complete. 2024-03-04T07:49:19.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 Processor -- start 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 -- start start 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9f00ffeb0 0x7fa9f019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f0102820 0x7fa9f019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f01051d0 0x7fa9f01a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01138e0 con 0x7fa9f0102820 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f0113760 con 0x7fa9f01051d0 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9f4e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f0113a60 con 0x7fa9f00ffeb0 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f01051d0 0x7fa9f01a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f0102820 0x7fa9f019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f01051d0 0x7fa9f01a53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34426/0 (socket says 172.21.15.116:34426) 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9ee800640 1 -- 172.21.15.116:0/2284029985 learned_addr learned my addr 172.21.15.116:0/2284029985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.605+0000 7fa9ede00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9f00ffeb0 0x7fa9f019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9ee800640 1 -- 172.21.15.116:0/2284029985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9f00ffeb0 msgr2=0x7fa9f019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9ee800640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9f00ffeb0 0x7fa9f019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9ee800640 1 -- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f0102820 msgr2=0x7fa9f019e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9ee800640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f0102820 0x7fa9f019e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9ee800640 1 -- 172.21.15.116:0/2284029985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f01a5a40 con 0x7fa9f01051d0 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9ede00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9f00ffeb0 0x7fa9f019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9ee800640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f01051d0 0x7fa9f01a53d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e9b0 tx=0x7fa9e800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9e7400640 1 -- 172.21.15.116:0/2284029985 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800cd80 con 0x7fa9f01051d0 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9e7400640 1 -- 172.21.15.116:0/2284029985 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e80035f0 con 0x7fa9f01051d0 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9e7400640 1 -- 172.21.15.116:0/2284029985 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8010670 con 0x7fa9f01051d0 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f01a5d30 con 0x7fa9f01051d0 2024-03-04T07:49:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.606+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f01a61a0 con 0x7fa9f01051d0 2024-03-04T07:49:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.609+0000 7fa9e7400640 1 -- 172.21.15.116:0/2284029985 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa9e80037a0 con 0x7fa9f01051d0 2024-03-04T07:49:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.609+0000 7fa9f79fc740 1 Processor -- start 2024-03-04T07:49:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9f79fc740 1 -- start start 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7e1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7e6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7e9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f5a747370 con 0x559f5a7e1850 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f5a6f0550 con 0x559f5a7e73b0 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f5a743410 con 0x559f5a7e2240 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9ede00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7e6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7e1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9ede00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7e6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41092/0 (socket says 172.21.15.116:41092) 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9ede00640 1 -- 172.21.15.116:0/4080359877 learned_addr learned my addr 172.21.15.116:0/4080359877 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.610+0000 7fa9e5400640 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7e9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9ede00640 1 -- 172.21.15.116:0/4080359877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 msgr2=0x559f5a7e9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9ede00640 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7e9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9ede00640 1 -- 172.21.15.116:0/4080359877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 msgr2=0x559f5a7e1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9ede00640 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7e1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9ede00640 1 -- 172.21.15.116:0/4080359877 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f5a7043c0 con 0x559f5a7e2240 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9ede00640 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7e6d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000f460 tx=0x7fa9e000f920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5a3afe4a456833d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9e7400640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9d40a1ea0 0x7fa9d40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9e7400640 1 -- 172.21.15.116:0/2284029985 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa9e80ce050 con 0x7fa9f01051d0 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9df400640 1 -- 172.21.15.116:0/4080359877 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0004d20 con 0x559f5a7e2240 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9e5400640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9d40a1ea0 0x7fa9d40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9df400640 1 -- 172.21.15.116:0/4080359877 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e00119e0 con 0x559f5a7e2240 2024-03-04T07:49:19.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.611+0000 7fa9df400640 1 -- 172.21.15.116:0/4080359877 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00054e0 con 0x559f5a7e2240 2024-03-04T07:49:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 -- 172.21.15.116:0/4080359877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 msgr2=0x559f5a7e6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7e6d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000f460 tx=0x7fa9e000f920 comp rx=0 tx=0).stop 2024-03-04T07:49:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9e5400640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9d40a1ea0 0x7fa9d40a4360 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa9d802c260 tx=0x7fa9d8043040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:19.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 -- 172.21.15.116:0/4080359877 shutdown_connections 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7e9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7e6d40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/4080359877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7e1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 -- 172.21.15.116:0/4080359877 >> 172.21.15.116:0/4080359877 conn(0x559f5a7dcdc0 msgr2=0x559f5a7dd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 -- 172.21.15.116:0/4080359877 shutdown_connections 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.612+0000 7fa9f79fc740 1 -- 172.21.15.116:0/4080359877 wait complete. 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 -- start start 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 Processor -- start 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e1850 0x559f5a7ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7ec0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e73b0 0x559f5a7f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f5a7ea360 con 0x559f5a7e73b0 2024-03-04T07:49:19.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f5a7039b0 con 0x559f5a7e1850 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f5a7ea4e0 con 0x559f5a7e2240 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9f79fc740 10 client.0 _ll_register_callbacks cb 0x559f5a73f340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.613+0000 7fa9ede00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e1850 0x559f5a7ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e1850 0x559f5a7ebbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34438/0 (socket says 172.21.15.116:34438) 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 -- 172.21.15.116:0/521911318 learned_addr learned my addr 172.21.15.116:0/521911318 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ee800640 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e73b0 0x559f5a7f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9e5400640 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7ec0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 -- 172.21.15.116:0/521911318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 msgr2=0x559f5a7ec0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7ec0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 -- 172.21.15.116:0/521911318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e73b0 msgr2=0x559f5a7f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e73b0 0x559f5a7f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 -- 172.21.15.116:0/521911318 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f5a45c210 con 0x559f5a7e1850 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9ede00640 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e1850 0x559f5a7ebbb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa9e001a000 tx=0x7fa9e0012e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f572cd71b25486dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:19.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9d3e00640 1 -- 172.21.15.116:0/521911318 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0029060 con 0x559f5a7e1850 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9d3e00640 1 -- 172.21.15.116:0/521911318 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa9e0004a00 con 0x559f5a7e1850 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.614+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9e00119e0 con 0x559f5a7e1850 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9d3e00640 1 -- 172.21.15.116:0/521911318 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9e0004c00 con 0x559f5a7e1850 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 10 client.27434 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559f5a528620 con 0x559f5a7e1850 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9d3e00640 1 -- 172.21.15.116:0/521911318 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e00119e0 con 0x559f5a7e1850 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9d3e00640 1 -- 172.21.15.116:0/521911318 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa9e002e070 con 0x559f5a7e1850 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 10 client.27434 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 10 client.27434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 20 client.27434 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a7f7660 0x559f5a7f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 4 client.27434 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:19.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f5a7548b0 con 0x559f5a7f7660 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.615+0000 7fa9e5400640 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a7f7660 0x559f5a7f9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.616+0000 7fa9e5400640 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a7f7660 0x559f5a7f9a50 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285010779f9295b1 server_cookie=5a9c766dc1d5e957 in_seq=0 out_seq=0 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.616+0000 7fa9d3e00640 10 client.27434 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9d3e00640 1 -- 172.21.15.116:0/521911318 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559f5a7548b0 con 0x559f5a7f7660 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9d3e00640 10 client.27434 handle_command_reply: tid=0 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 client.27434 shutdown 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e1850 msgr2=0x559f5a7ebbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e1850 0x559f5a7ebbb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa9e001a000 tx=0x7fa9e0012e40 comp rx=0 tx=0).stop 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 shutdown_connections 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a7f7660 0x559f5a7f9a50 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e73b0 0x559f5a7f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7ec0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/521911318 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e1850 0x559f5a7ebbb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9d3e00640 0 client.27434 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:19.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 >> 172.21.15.116:0/521911318 conn(0x559f5a7dcdc0 msgr2=0x559f5a7dbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:19.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 shutdown_connections 2024-03-04T07:49:19.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 1 -- 172.21.15.116:0/521911318 wait complete. 2024-03-04T07:49:19.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.617+0000 7fa9f79fc740 20 client.27434 trim_cache size 0 max 16384 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.632+0000 7fa9f79fc740 1 Processor -- start 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.632+0000 7fa9f79fc740 1 -- start start 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7dbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7dc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01ad000 con 0x559f5a7e1850 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01af5e0 con 0x559f5a7e73b0 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01ad600 con 0x559f5a7e2240 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7dbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7dbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48222/0 (socket says 172.21.15.116:48222) 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7dc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34440/0 (socket says 172.21.15.116:34440) 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 -- 172.21.15.116:0/130980431 learned_addr learned my addr 172.21.15.116:0/130980431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 -- 172.21.15.116:0/130980431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 msgr2=0x559f5a7dc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7dc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 -- 172.21.15.116:0/130980431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 msgr2=0x559f5a7f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ede00640 1 -- 172.21.15.116:0/130980431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f5a7d8610 con 0x559f5a7e1850 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.633+0000 7fa9ee800640 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.634+0000 7fa9ede00640 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7dbe60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0006010 tx=0x7fa9e0004b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d338a3650a036dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.634+0000 7fa9dea00640 1 -- 172.21.15.116:0/130980431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0034020 con 0x559f5a7e1850 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.634+0000 7fa9dea00640 1 -- 172.21.15.116:0/130980431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0009cb0 con 0x559f5a7e1850 2024-03-04T07:49:19.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.634+0000 7fa9dea00640 1 -- 172.21.15.116:0/130980431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0029070 con 0x559f5a7e1850 2024-03-04T07:49:19.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.634+0000 7fa9f79fc740 1 -- 172.21.15.116:0/130980431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 msgr2=0x559f5a7dbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.634+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7dbe60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0006010 tx=0x7fa9e0004b90 comp rx=0 tx=0).stop 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 -- 172.21.15.116:0/130980431 shutdown_connections 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e2240 0x559f5a7dc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/130980431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e1850 0x559f5a7dbe60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 -- 172.21.15.116:0/130980431 >> 172.21.15.116:0/130980431 conn(0x559f5a7dcdc0 msgr2=0x559f5a7db750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 -- 172.21.15.116:0/130980431 shutdown_connections 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 -- 172.21.15.116:0/130980431 wait complete. 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 -- start start 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.635+0000 7fa9f79fc740 1 Processor -- start 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e1850 0x559f5a7f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e2240 0x559f5a7f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9f79fc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01adc60 con 0x559f5a7e2240 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01aef80 con 0x559f5a7e73b0 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9f79fc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01ae2c0 con 0x559f5a7e1850 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9f79fc740 10 client.0 _ll_register_callbacks cb 0x7fa9e0015050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e2240 0x559f5a7f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ede00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e1850 0x559f5a7f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34444/0 (socket says 172.21.15.116:34444) 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 -- 172.21.15.116:0/2600486941 learned_addr learned my addr 172.21.15.116:0/2600486941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 -- 172.21.15.116:0/2600486941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e1850 msgr2=0x559f5a7f2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e1850 0x559f5a7f2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 -- 172.21.15.116:0/2600486941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e2240 msgr2=0x559f5a7f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e2240 0x559f5a7f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.636+0000 7fa9ee800640 1 -- 172.21.15.116:0/2600486941 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f5a7f1c60 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9ee800640 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f1720 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa9e809aa80 tx=0x7fa9e809ad80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ea0c82ee12774d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9d3e00640 1 -- 172.21.15.116:0/2600486941 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80b7070 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9d3e00640 1 -- 172.21.15.116:0/2600486941 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa9e8003b10 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9e0009cb0 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9d3e00640 1 -- 172.21.15.116:0/2600486941 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9e8080050 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 10 client.27440 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559f5a853b30 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9d3e00640 1 -- 172.21.15.116:0/2600486941 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0009cb0 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9d3e00640 1 -- 172.21.15.116:0/2600486941 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa9e8096ad0 con 0x559f5a7e73b0 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 10 client.27440 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 10 client.27440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 20 client.27440 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a854460 0x559f5a856880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.637+0000 7fa9f79fc740 4 client.27440 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:49:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.638+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559f5a74ec00 con 0x559f5a854460 2024-03-04T07:49:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.638+0000 7fa9ede00640 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a854460 0x559f5a856880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.638+0000 7fa9ede00640 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a854460 0x559f5a856880 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba5a786829d00dc server_cookie=5cbb03cf01036c8d in_seq=0 out_seq=0 2024-03-04T07:49:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.638+0000 7fa9d3e00640 10 client.27440 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9d3e00640 1 -- 172.21.15.116:0/2600486941 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559f5a74ec00 con 0x559f5a854460 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9d3e00640 10 client.27440 handle_command_reply: tid=0 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 client.27440 shutdown 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 msgr2=0x559f5a7f1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f1720 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa9e809aa80 tx=0x7fa9e809ad80 comp rx=0 tx=0).stop 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 shutdown_connections 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559f5a854460 0x559f5a856880 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559f5a7e73b0 0x559f5a7f1720 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f5a7e2240 0x559f5a7f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9d3e00640 0 client.27440 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 --2- 172.21.15.116:0/2600486941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f5a7e1850 0x559f5a7f2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.654+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 >> 172.21.15.116:0/2600486941 conn(0x559f5a7dcdc0 msgr2=0x559f5a7dbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.655+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 shutdown_connections 2024-03-04T07:49:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.655+0000 7fa9f79fc740 1 -- 172.21.15.116:0/2600486941 wait complete. 2024-03-04T07:49:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.655+0000 7fa9f79fc740 20 client.27440 trim_cache size 0 max 16384 2024-03-04T07:49:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:49:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:49:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:49:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:49:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.655+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9d40a1ea0 msgr2=0x7fa9d40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.655+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9d40a1ea0 0x7fa9d40a4360 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa9d802c260 tx=0x7fa9d8043040 comp rx=0 tx=0).stop 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.655+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f01051d0 msgr2=0x7fa9f01a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.655+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f01051d0 0x7fa9f01a53d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e9b0 tx=0x7fa9e800ee70 comp rx=0 tx=0).stop 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 shutdown_connections 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f01051d0 0x7fa9f01a53d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f0102820 0x7fa9f019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9d40a1ea0 0x7fa9d40a4360 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 --2- 172.21.15.116:0/2284029985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9f00ffeb0 0x7fa9f019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 >> 172.21.15.116:0/2284029985 conn(0x7fa9f00fbab0 msgr2=0x7fa9f0100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 shutdown_connections 2024-03-04T07:49:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:19.656+0000 7fa9f4e00640 1 -- 172.21.15.116:0/2284029985 wait complete. 2024-03-04T07:49:21.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:20 smithi116 ceph-mon[32150]: pgmap v658: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:49:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:20 smithi152 ceph-mon[37154]: pgmap v658: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:49:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:20 smithi133 ceph-mon[36505]: pgmap v658: 97 pgs: 97 active+clean; 2.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:49:22.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:49:22.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:49:22.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 -- 172.21.15.116:0/3990856844 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90081027d0 msgr2=0x7f9008102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 --2- 172.21.15.116:0/3990856844 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90081027d0 0x7f9008102c50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009900 tx=0x7f8ff402bc30 comp rx=0 tx=0).stop 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 -- 172.21.15.116:0/3990856844 shutdown_connections 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 --2- 172.21.15.116:0/3990856844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9008103190 0x7f9008109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 --2- 172.21.15.116:0/3990856844 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90081027d0 0x7f9008102c50 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 --2- 172.21.15.116:0/3990856844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90081015e0 0x7f9008101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 -- 172.21.15.116:0/3990856844 >> 172.21.15.116:0/3990856844 conn(0x7f90080fcb50 msgr2=0x7f90080fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 -- 172.21.15.116:0/3990856844 shutdown_connections 2024-03-04T07:49:22.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.943+0000 7f900e800640 1 -- 172.21.15.116:0/3990856844 wait complete. 2024-03-04T07:49:22.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.944+0000 7f900e800640 1 Processor -- start 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.944+0000 7f900e800640 1 -- start start 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f900e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90081015e0 0x7f9008075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f900e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90081027d0 0x7f90080739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f900e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9008103190 0x7f9008073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f900e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900810d2a0 con 0x7f90081015e0 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f900e800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900810d120 con 0x7f9008103190 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f900e800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900810d420 con 0x7f90081027d0 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f9007400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90081015e0 0x7f9008075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f9007e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9008103190 0x7f9008073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f9006a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90081027d0 0x7f90080739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f9007e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9008103190 0x7f9008073f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34454/0 (socket says 172.21.15.116:34454) 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.945+0000 7f9007e00640 1 -- 172.21.15.116:0/762761077 learned_addr learned my addr 172.21.15.116:0/762761077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f9006a00640 1 -- 172.21.15.116:0/762761077 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9008103190 msgr2=0x7f9008073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f9006a00640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9008103190 0x7f9008073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f9006a00640 1 -- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90081015e0 msgr2=0x7f9008075380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f9006a00640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90081015e0 0x7f9008075380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f9006a00640 1 -- 172.21.15.116:0/762761077 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008074790 con 0x7f90081027d0 2024-03-04T07:49:22.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f9006a00640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90081027d0 0x7f90080739c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4036040 tx=0x7f8ff402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:22.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f8fffe00640 1 -- 172.21.15.116:0/762761077 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4043020 con 0x7f90081027d0 2024-03-04T07:49:22.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f8fffe00640 1 -- 172.21.15.116:0/762761077 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff40054e0 con 0x7f90081027d0 2024-03-04T07:49:22.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90081a1a60 con 0x7f90081027d0 2024-03-04T07:49:22.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.946+0000 7f8fffe00640 1 -- 172.21.15.116:0/762761077 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4003c30 con 0x7f90081027d0 2024-03-04T07:49:22.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.947+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90081a1f80 con 0x7f90081027d0 2024-03-04T07:49:22.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 Processor -- start 2024-03-04T07:49:22.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 -- start start 2024-03-04T07:49:22.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f8fffe00640 1 -- 172.21.15.116:0/762761077 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f8ff4046020 con 0x7f90081027d0 2024-03-04T07:49:22.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d767f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d76d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d76fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d695090 con 0x55b21d76d6b0 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d6ccd30 con 0x55b21d768540 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f901140d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d675a30 con 0x55b21d767b50 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f9007400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d76d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f9007400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d76d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34460/0 (socket says 172.21.15.116:34460) 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f9007400640 1 -- 172.21.15.116:0/3475352268 learned_addr learned my addr 172.21.15.116:0/3475352268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.950+0000 7f9007e00640 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d767f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9006a00640 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d76fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9007400640 1 -- 172.21.15.116:0/3475352268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 msgr2=0x55b21d767f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9007400640 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d767f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9007400640 1 -- 172.21.15.116:0/3475352268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 msgr2=0x55b21d76fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9007400640 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d76fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9007400640 1 -- 172.21.15.116:0/3475352268 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b21d4ab7d0 con 0x55b21d768540 2024-03-04T07:49:22.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9006a00640 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d76fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:22.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f9007400640 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d76d040 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009ca0 tx=0x7f8ff8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c740e0adce0585 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:22.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f8ffcc00640 1 -- 172.21.15.116:0/3475352268 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8004220 con 0x55b21d768540 2024-03-04T07:49:22.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f8ffcc00640 1 -- 172.21.15.116:0/3475352268 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff8009e50 con 0x55b21d768540 2024-03-04T07:49:22.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.951+0000 7f8ffcc00640 1 -- 172.21.15.116:0/3475352268 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8019830 con 0x55b21d768540 2024-03-04T07:49:22.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f8fffe00640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fec0a1d80 0x7f8fec0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f8fffe00640 1 -- 172.21.15.116:0/762761077 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8ff40efdb0 con 0x7f90081027d0 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f9007e00640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fec0a1d80 0x7f8fec0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f901140d740 1 -- 172.21.15.116:0/3475352268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 msgr2=0x55b21d76d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f901140d740 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d76d040 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009ca0 tx=0x7f8ff8011810 comp rx=0 tx=0).stop 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f901140d740 1 -- 172.21.15.116:0/3475352268 shutdown_connections 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f901140d740 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d76fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f901140d740 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d76d040 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f901140d740 1 --2- 172.21.15.116:0/3475352268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d767f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.952+0000 7f901140d740 1 -- 172.21.15.116:0/3475352268 >> 172.21.15.116:0/3475352268 conn(0x55b21d7630c0 msgr2=0x55b21d7634c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:22.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.953+0000 7f901140d740 1 -- 172.21.15.116:0/3475352268 shutdown_connections 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.953+0000 7f9007e00640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fec0a1d80 0x7f8fec0a4240 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55b21d768da0 tx=0x7f9000009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.953+0000 7f901140d740 1 -- 172.21.15.116:0/3475352268 wait complete. 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.953+0000 7f901140d740 1 -- start start 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.953+0000 7f901140d740 1 Processor -- start 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f901140d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d767b50 0x55b21d771f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f901140d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d772460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f901140d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d76d6b0 0x55b21d7794a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f901140d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d66b0c0 con 0x55b21d767b50 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f901140d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d6c8dd0 con 0x55b21d768540 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f901140d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d770660 con 0x55b21d76d6b0 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f901140d740 10 client.0 _ll_register_callbacks cb 0x55b21d688860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9007400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d767b50 0x55b21d771f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9006a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d772460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9007400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d767b50 0x55b21d771f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48268/0 (socket says 172.21.15.116:48268) 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9007400640 1 -- 172.21.15.116:0/4062439743 learned_addr learned my addr 172.21.15.116:0/4062439743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9007e00640 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d76d6b0 0x55b21d7794a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9006a00640 1 -- 172.21.15.116:0/4062439743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d76d6b0 msgr2=0x55b21d7794a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9006a00640 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d76d6b0 0x55b21d7794a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9006a00640 1 -- 172.21.15.116:0/4062439743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d767b50 msgr2=0x55b21d771f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9006a00640 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d767b50 0x55b21d771f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9006a00640 1 -- 172.21.15.116:0/4062439743 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b21d3e13f0 con 0x55b21d768540 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.954+0000 7f9007400640 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d767b50 0x55b21d771f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f9006a00640 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d772460 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40ed870 tx=0x7f8ff4003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64b9adada18dac37 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f8ff1600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f8ff1600640 1 -- 172.21.15.116:0/4062439743 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff40bc400 con 0x55b21d768540 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f8ff1600640 1 -- 172.21.15.116:0/4062439743 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8ff40bada0 con 0x55b21d768540 2024-03-04T07:49:22.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ff8009e50 con 0x55b21d768540 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f8ff1600640 1 -- 172.21.15.116:0/4062439743 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ff4005690 con 0x55b21d768540 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f901140d740 10 client.27452 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b21d3a10f0 con 0x55b21d768540 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f8ff1600640 1 -- 172.21.15.116:0/4062439743 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff8009e50 con 0x55b21d768540 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.955+0000 7f8ff1600640 1 -- 172.21.15.116:0/4062439743 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8ff40c4db0 con 0x55b21d768540 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f901140d740 10 client.27452 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f901140d740 10 client.27452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f901140d740 20 client.27452 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f901140d740 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d77d940 0x55b21d77fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f901140d740 4 client.27452 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b21d6da270 con 0x55b21d77d940 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f9007400640 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d77d940 0x55b21d77fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f9007400640 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d77d940 0x55b21d77fdb0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36a05a3dd8888b41 server_cookie=f71ab5ab18b2d408 in_seq=0 out_seq=0 2024-03-04T07:49:22.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.956+0000 7f8ff1600640 10 client.27452 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f8ff1600640 1 -- 172.21.15.116:0/4062439743 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b21d6da270 con 0x55b21d77d940 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f8ff1600640 10 client.27452 handle_command_reply: tid=0 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 client.27452 shutdown 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 msgr2=0x55b21d772460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d772460 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40ed870 tx=0x7f8ff4003760 comp rx=0 tx=0).stop 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 shutdown_connections 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d77d940 0x55b21d77fdb0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d76d6b0 0x55b21d7794a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d768540 0x55b21d772460 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f8ff1600640 0 client.27452 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 --2- 172.21.15.116:0/4062439743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d767b50 0x55b21d771f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.957+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 >> 172.21.15.116:0/4062439743 conn(0x55b21d7630c0 msgr2=0x55b21d762360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.958+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 shutdown_connections 2024-03-04T07:49:22.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.958+0000 7f901140d740 1 -- 172.21.15.116:0/4062439743 wait complete. 2024-03-04T07:49:22.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.958+0000 7f901140d740 20 client.27452 trim_cache size 0 max 16384 2024-03-04T07:49:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.975+0000 7f901140d740 1 Processor -- start 2024-03-04T07:49:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.975+0000 7f901140d740 1 -- start start 2024-03-04T07:49:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f901140d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d767b50 0x55b21d761ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f901140d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d768540 0x55b21d779460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f901140d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d762010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f901140d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081a8c90 con 0x55b21d76d6b0 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f901140d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081ab270 con 0x55b21d767b50 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f901140d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081a9290 con 0x55b21d768540 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d762010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d767b50 0x55b21d761ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9006a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d768540 0x55b21d779460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d767b50 0x55b21d761ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34480/0 (socket says 172.21.15.116:34480) 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d762010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48274/0 (socket says 172.21.15.116:48274) 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 -- 172.21.15.116:0/850859511 learned_addr learned my addr 172.21.15.116:0/850859511 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 -- 172.21.15.116:0/850859511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d768540 msgr2=0x55b21d779460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 --2- 172.21.15.116:0/850859511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d768540 0x55b21d779460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 -- 172.21.15.116:0/850859511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 msgr2=0x55b21d762010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 --2- 172.21.15.116:0/850859511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d762010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.976+0000 7f9007400640 1 -- 172.21.15.116:0/850859511 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b21d771000 con 0x55b21d767b50 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f9007400640 1 --2- 172.21.15.116:0/850859511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d767b50 0x55b21d761ad0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8004590 tx=0x7f8ff8009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fb63d48ec0a7448 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f8ff3e00640 1 -- 172.21.15.116:0/850859511 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff802c2d0 con 0x55b21d767b50 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f8ff3e00640 1 -- 172.21.15.116:0/850859511 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff8030030 con 0x55b21d767b50 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f8ff3e00640 1 -- 172.21.15.116:0/850859511 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8004950 con 0x55b21d767b50 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f901140d740 1 -- 172.21.15.116:0/850859511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d767b50 msgr2=0x55b21d761ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f901140d740 1 --2- 172.21.15.116:0/850859511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d767b50 0x55b21d761ad0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8004590 tx=0x7f8ff8009310 comp rx=0 tx=0).stop 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f901140d740 1 -- 172.21.15.116:0/850859511 shutdown_connections 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f901140d740 1 --2- 172.21.15.116:0/850859511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d76d6b0 0x55b21d762010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f901140d740 1 --2- 172.21.15.116:0/850859511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d768540 0x55b21d779460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f901140d740 1 --2- 172.21.15.116:0/850859511 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d767b50 0x55b21d761ad0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.977+0000 7f901140d740 1 -- 172.21.15.116:0/850859511 >> 172.21.15.116:0/850859511 conn(0x55b21d7630c0 msgr2=0x55b21d769a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.978+0000 7f901140d740 1 -- 172.21.15.116:0/850859511 shutdown_connections 2024-03-04T07:49:22.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.978+0000 7f901140d740 1 -- 172.21.15.116:0/850859511 wait complete. 2024-03-04T07:49:22.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.978+0000 7f901140d740 1 -- start start 2024-03-04T07:49:22.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.978+0000 7f901140d740 1 Processor -- start 2024-03-04T07:49:22.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f901140d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d778dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f901140d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d768540 0x55b21d779300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f901140d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d76d6b0 0x55b21d777420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f901140d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081a98f0 con 0x55b21d768540 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f901140d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081aac10 con 0x55b21d76d6b0 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f901140d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90081a9f50 con 0x55b21d767b50 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f901140d740 10 client.0 _ll_register_callbacks cb 0x7f8ff4003610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9006a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d768540 0x55b21d779300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9007e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d76d6b0 0x55b21d777420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9006a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d768540 0x55b21d779300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48284/0 (socket says 172.21.15.116:48284) 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9006a00640 1 -- 172.21.15.116:0/421461061 learned_addr learned my addr 172.21.15.116:0/421461061 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9007400640 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d778dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9007e00640 1 -- 172.21.15.116:0/421461061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 msgr2=0x55b21d778dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9007e00640 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d778dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9007e00640 1 -- 172.21.15.116:0/421461061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d768540 msgr2=0x55b21d779300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9007e00640 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d768540 0x55b21d779300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9007e00640 1 -- 172.21.15.116:0/421461061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b21d777c50 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.979+0000 7f9006a00640 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d768540 0x55b21d779300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f9007e00640 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d76d6b0 0x55b21d777420 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9000009d90 tx=0x7f9000004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a630efab94ccc1bc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f8ff1600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f8ff1600640 1 -- 172.21.15.116:0/421461061 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000011b20 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f8ff1600640 1 -- 172.21.15.116:0/421461061 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9000011cc0 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ff8030030 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f8ff1600640 1 -- 172.21.15.116:0/421461061 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9000011f00 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f901140d740 10 client.27458 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.980+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b21d7e25f0 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f8ff1600640 1 -- 172.21.15.116:0/421461061 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff8030030 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f8ff1600640 1 -- 172.21.15.116:0/421461061 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f900002a5c0 con 0x55b21d76d6b0 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f901140d740 10 client.27458 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f901140d740 10 client.27458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f901140d740 20 client.27458 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f901140d740 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d7e2fd0 0x55b21d7e53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f901140d740 4 client.27458 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b21d6d45c0 con 0x55b21d7e2fd0 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f9007400640 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d7e2fd0 0x55b21d7e53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f9007400640 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d7e2fd0 0x55b21d7e53f0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=383f141c898722bb server_cookie=ffe01a1ca4283bfc in_seq=0 out_seq=0 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.981+0000 7f8ff1600640 10 client.27458 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f8ff1600640 1 -- 172.21.15.116:0/421461061 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b21d6d45c0 con 0x55b21d7e2fd0 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f8ff1600640 10 client.27458 handle_command_reply: tid=0 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 client.27458 shutdown 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d76d6b0 msgr2=0x55b21d777420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d76d6b0 0x55b21d777420 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9000009d90 tx=0x7f9000004ab0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 shutdown_connections 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b21d7e2fd0 0x55b21d7e53f0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b21d76d6b0 0x55b21d777420 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b21d768540 0x55b21d779300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 --2- 172.21.15.116:0/421461061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b21d767b50 0x55b21d778dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f8ff1600640 0 client.27458 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 >> 172.21.15.116:0/421461061 conn(0x55b21d7630c0 msgr2=0x55b21d766320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 shutdown_connections 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 1 -- 172.21.15.116:0/421461061 wait complete. 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.982+0000 7f901140d740 20 client.27458 trim_cache size 0 max 16384 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:49:22.988 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:49:22.989 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:49:22.989 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:49:22.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fec0a1d80 msgr2=0x7f8fec0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fec0a1d80 0x7f8fec0a4240 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55b21d768da0 tx=0x7f9000009500 comp rx=0 tx=0).stop 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90081027d0 msgr2=0x7f90080739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90081027d0 0x7f90080739c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4036040 tx=0x7f8ff402c650 comp rx=0 tx=0).stop 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 shutdown_connections 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9008103190 0x7f9008073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f8fec0a1d80 0x7f8fec0a4240 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f90081027d0 0x7f90080739c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 --2- 172.21.15.116:0/762761077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90081015e0 0x7f9008075380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 >> 172.21.15.116:0/762761077 conn(0x7f90080fcb50 msgr2=0x7f9008105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 shutdown_connections 2024-03-04T07:49:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:22.983+0000 7f900e800640 1 -- 172.21.15.116:0/762761077 wait complete. 2024-03-04T07:49:23.001 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:49:23.002 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:49:23.077 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:22 smithi116 ceph-mon[32150]: pgmap v659: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.215+0000 7ffa0de00640 1 -- 172.21.15.116:0/1614002324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa081053c0 msgr2=0x7ffa0810bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.215+0000 7ffa0de00640 1 --2- 172.21.15.116:0/1614002324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa081053c0 0x7ffa0810bc50 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ffa00009ca0 tx=0x7ffa0002c100 comp rx=0 tx=0).stop 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.215+0000 7ffa0de00640 1 -- 172.21.15.116:0/1614002324 shutdown_connections 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.215+0000 7ffa0de00640 1 --2- 172.21.15.116:0/1614002324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa081053c0 0x7ffa0810bc50 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.215+0000 7ffa0de00640 1 --2- 172.21.15.116:0/1614002324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa08104a00 0x7ffa08104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.215+0000 7ffa0de00640 1 --2- 172.21.15.116:0/1614002324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 0x7ffa08103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.215+0000 7ffa0de00640 1 -- 172.21.15.116:0/1614002324 >> 172.21.15.116:0/1614002324 conn(0x7ffa080fed80 msgr2=0x7ffa081011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.216+0000 7ffa0de00640 1 -- 172.21.15.116:0/1614002324 shutdown_connections 2024-03-04T07:49:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.216+0000 7ffa0de00640 1 -- 172.21.15.116:0/1614002324 wait complete. 2024-03-04T07:49:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.216+0000 7ffa0de00640 1 Processor -- start 2024-03-04T07:49:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa0de00640 1 -- start start 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa0de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 0x7ffa08199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa0de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa08104a00 0x7ffa08199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa0de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa081053c0 0x7ffa081a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa0de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0810f470 con 0x7ffa08103810 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa0de00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0810f2f0 con 0x7ffa08104a00 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa0de00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0810f5f0 con 0x7ffa081053c0 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa06a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 0x7ffa08199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa07400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa081053c0 0x7ffa081a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa06000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa08104a00 0x7ffa08199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa07400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa081053c0 0x7ffa081a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41172/0 (socket says 172.21.15.116:41172) 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa07400640 1 -- 172.21.15.116:0/3073956473 learned_addr learned my addr 172.21.15.116:0/3073956473 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.217+0000 7ffa06a00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 0x7ffa08199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48290/0 (socket says 172.21.15.116:48290) 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa06000640 1 -- 172.21.15.116:0/3073956473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa081053c0 msgr2=0x7ffa081a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa06000640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa081053c0 0x7ffa081a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa06000640 1 -- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 msgr2=0x7ffa08199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa06000640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 0x7ffa08199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa06000640 1 -- 172.21.15.116:0/3073956473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa081a16d0 con 0x7ffa08104a00 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa07400640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa081053c0 0x7ffa081a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa06a00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 0x7ffa08199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa06000640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa08104a00 0x7ffa08199ff0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400d6d0 tx=0x7ff9f400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa0ca00640 1 -- 172.21.15.116:0/3073956473 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4017070 con 0x7ffa08104a00 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa0ca00640 1 -- 172.21.15.116:0/3073956473 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9f4009820 con 0x7ffa08104a00 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa0ca00640 1 -- 172.21.15.116:0/3073956473 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4004d00 con 0x7ffa08104a00 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa081a19c0 con 0x7ffa08104a00 2024-03-04T07:49:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.218+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa081a1f10 con 0x7ffa08104a00 2024-03-04T07:49:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.219+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0810ab30 con 0x7ffa08104a00 2024-03-04T07:49:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.222+0000 7ffa0ca00640 1 -- 172.21.15.116:0/3073956473 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff9f40099d0 con 0x7ffa08104a00 2024-03-04T07:49:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.223+0000 7ffa0ca00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9ec0a1d80 0x7ff9ec0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.224+0000 7ffa0ca00640 1 -- 172.21.15.116:0/3073956473 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff9f40cd8c0 con 0x7ffa08104a00 2024-03-04T07:49:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.224+0000 7ffa06a00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9ec0a1d80 0x7ff9ec0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.224+0000 7ffa06a00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9ec0a1d80 0x7ff9ec0a4240 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffa08104860 tx=0x7ff9f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.229+0000 7ffa0ca00640 1 -- 172.21.15.116:0/3073956473 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff9f409a660 con 0x7ffa08104a00 2024-03-04T07:49:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:22 smithi152 ceph-mon[37154]: pgmap v659: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-04T07:49:23.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:22 smithi133 ceph-mon[36505]: pgmap v659: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-03-04T07:49:23.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.464+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa0802cff0 con 0x7ffa08104a00 2024-03-04T07:49:23.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.465+0000 7ffa0ca00640 1 -- 172.21.15.116:0/3073956473 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7ff9f409e700 con 0x7ffa08104a00 2024-03-04T07:49:23.466 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:49:23.466 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:49:23.467 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.467+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9ec0a1d80 msgr2=0x7ff9ec0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.467+0000 7ffa0de00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9ec0a1d80 0x7ff9ec0a4240 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffa08104860 tx=0x7ff9f8005f90 comp rx=0 tx=0).stop 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.467+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa08104a00 msgr2=0x7ffa08199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.467+0000 7ffa0de00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa08104a00 0x7ffa08199ff0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400d6d0 tx=0x7ff9f400db90 comp rx=0 tx=0).stop 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 shutdown_connections 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9ec0a1d80 0x7ff9ec0a4240 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa081053c0 0x7ffa081a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffa08104a00 0x7ffa08199ff0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 --2- 172.21.15.116:0/3073956473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa08103810 0x7ffa08199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 >> 172.21.15.116:0/3073956473 conn(0x7ffa080fed80 msgr2=0x7ffa08107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 shutdown_connections 2024-03-04T07:49:23.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.468+0000 7ffa0de00640 1 -- 172.21.15.116:0/3073956473 wait complete. 2024-03-04T07:49:23.487 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:49:23.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:49:23.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3212374525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 msgr2=0x7fdbbc108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3212374525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc108010 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009900 tx=0x7fdbac02bc30 comp rx=0 tx=0).stop 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3212374525 shutdown_connections 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3212374525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbbc1097a0 0x7fdbbc110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3212374525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbbc108de0 0x7fdbbc109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3212374525 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc108010 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3212374525 >> 172.21.15.116:0/3212374525 conn(0x7fdbbc076b90 msgr2=0x7fdbbc078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3212374525 shutdown_connections 2024-03-04T07:49:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.697+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3212374525 wait complete. 2024-03-04T07:49:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.698+0000 7fdbc0a00640 1 Processor -- start 2024-03-04T07:49:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.698+0000 7fdbc0a00640 1 -- start start 2024-03-04T07:49:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbc0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbc0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbbc108de0 0x7fdbbc19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbc0a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbbc1097a0 0x7fdbbc1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbc0a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1138b0 con 0x7fdbbc108de0 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbc0a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc113730 con 0x7fdbbc107bf0 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbc0a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc113a30 con 0x7fdbbc1097a0 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbbc108de0 0x7fdbbc19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34512/0 (socket says 172.21.15.116:34512) 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 -- 172.21.15.116:0/3665595364 learned_addr learned my addr 172.21.15.116:0/3665595364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbba800640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbbc1097a0 0x7fdbbc1a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 -- 172.21.15.116:0/3665595364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbbc1097a0 msgr2=0x7fdbbc1a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbbc1097a0 0x7fdbbc1a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 -- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbbc108de0 msgr2=0x7fdbbc19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbbc108de0 0x7fdbbc19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb9e00640 1 -- 172.21.15.116:0/3665595364 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc1a5b40 con 0x7fdbbc107bf0 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.699+0000 7fdbb1400640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbbc108de0 0x7fdbbc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.700+0000 7fdbb9e00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc19df20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fdbac000c00 tx=0x7fdbac02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.700+0000 7fdbb3400640 1 -- 172.21.15.116:0/3665595364 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac043020 con 0x7fdbbc107bf0 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.700+0000 7fdbb3400640 1 -- 172.21.15.116:0/3665595364 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbac0054e0 con 0x7fdbbc107bf0 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.700+0000 7fdbb3400640 1 -- 172.21.15.116:0/3665595364 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac003c40 con 0x7fdbbc107bf0 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.700+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc1a5dd0 con 0x7fdbbc107bf0 2024-03-04T07:49:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.700+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc1a61e0 con 0x7fdbbc107bf0 2024-03-04T07:49:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.703+0000 7fdbb3400640 1 -- 172.21.15.116:0/3665595364 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fdbac046020 con 0x7fdbbc107bf0 2024-03-04T07:49:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.703+0000 7fdbc383a740 1 Processor -- start 2024-03-04T07:49:23.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbc383a740 1 -- start start 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x5647628440c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762849190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x56476284bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbc383a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647627a8d60 con 0x564762844690 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbc383a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564762751cf0 con 0x564762849800 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbc383a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647627a4e00 con 0x564762843ca0 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbb9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762849190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbba800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x5647628440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x56476284bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbba800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x5647628440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41206/0 (socket says 172.21.15.116:41206) 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbb9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762849190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48304/0 (socket says 172.21.15.116:48304) 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.704+0000 7fdbba800640 1 -- 172.21.15.116:0/1576886488 learned_addr learned my addr 172.21.15.116:0/1576886488 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbba800640 1 -- 172.21.15.116:0/1576886488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 msgr2=0x56476284bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbba800640 1 --2- 172.21.15.116:0/1576886488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x56476284bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbba800640 1 -- 172.21.15.116:0/1576886488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 msgr2=0x564762849190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbba800640 1 --2- 172.21.15.116:0/1576886488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762849190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbba800640 1 -- 172.21.15.116:0/1576886488 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647624bd3f0 con 0x564762843ca0 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbb3400640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdba00a1e00 0x7fdba00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbba800640 1 --2- 172.21.15.116:0/1576886488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x5647628440c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009fc0 tx=0x7fdbb4010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc93e2f42fe68d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbb3400640 1 -- 172.21.15.116:0/3665595364 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdbac0efdd0 con 0x7fdbbc107bf0 2024-03-04T07:49:23.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbab000640 1 -- 172.21.15.116:0/1576886488 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4003b70 con 0x564762843ca0 2024-03-04T07:49:23.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbb9e00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdba00a1e00 0x7fdba00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbab000640 1 -- 172.21.15.116:0/1576886488 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb4003d10 con 0x564762843ca0 2024-03-04T07:49:23.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.705+0000 7fdbab000640 1 -- 172.21.15.116:0/1576886488 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4011a40 con 0x564762843ca0 2024-03-04T07:49:23.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 -- 172.21.15.116:0/1576886488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 msgr2=0x5647628440c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1576886488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x5647628440c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009fc0 tx=0x7fdbb4010860 comp rx=0 tx=0).stop 2024-03-04T07:49:23.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbb9e00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdba00a1e00 0x7fdba00a42c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0b9490 tx=0x7fdbac0f3810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 -- 172.21.15.116:0/1576886488 shutdown_connections 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1576886488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x56476284bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1576886488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762849190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1576886488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x5647628440c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 -- 172.21.15.116:0/1576886488 >> 172.21.15.116:0/1576886488 conn(0x56476283f210 msgr2=0x56476283f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 -- 172.21.15.116:0/1576886488 shutdown_connections 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.706+0000 7fdbc383a740 1 -- 172.21.15.116:0/1576886488 wait complete. 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 -- start start 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 Processor -- start 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x56476284e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x56476284e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762855750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56476284c7b0 con 0x564762844690 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x564762747220 con 0x564762849800 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.707+0000 7fdbc383a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56476284c930 con 0x564762843ca0 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbc383a740 10 client.0 _ll_register_callbacks cb 0x5647624bcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x56476284e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x56476284e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48314/0 (socket says 172.21.15.116:48314) 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 -- 172.21.15.116:0/588799596 learned_addr learned my addr 172.21.15.116:0/588799596 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbba800640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x56476284e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbba800640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762855750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 -- 172.21.15.116:0/588799596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 msgr2=0x56476284e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x56476284e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 -- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 msgr2=0x564762855750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762855750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 -- 172.21.15.116:0/588799596 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647625877d0 con 0x564762844690 2024-03-04T07:49:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbba800640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762855750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdbb1400640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x56476284e710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdba4012d20 tx=0x7fdba400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0aa634ae2132926 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.708+0000 7fdb9f400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdb9f400640 1 -- 172.21.15.116:0/588799596 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4012110 con 0x564762844690 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdb9f400640 1 -- 172.21.15.116:0/588799596 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdba4004050 con 0x564762844690 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbb4003d10 con 0x564762844690 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdb9f400640 1 -- 172.21.15.116:0/588799596 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdba40122b0 con 0x564762844690 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdbc383a740 10 client.17343 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x564762858be0 con 0x564762844690 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdb9f400640 1 -- 172.21.15.116:0/588799596 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb4003d10 con 0x564762844690 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.709+0000 7fdb9f400640 1 -- 172.21.15.116:0/588799596 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdba4010430 con 0x564762844690 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbc383a740 10 client.17343 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbc383a740 10 client.17343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbc383a740 20 client.17343 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:23.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbc383a740 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56476285a330 0x56476285c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbc383a740 4 client.17343 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647627b62a0 con 0x56476285a330 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbba800640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56476285a330 0x56476285c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdbba800640 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56476285a330 0x56476285c750 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3fc179766f3cc0b server_cookie=168614c59f636a58 in_seq=0 out_seq=0 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.710+0000 7fdb9f400640 10 client.17343 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdb9f400640 1 -- 172.21.15.116:0/588799596 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5647627b62a0 con 0x56476285a330 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdb9f400640 10 client.17343 handle_command_reply: tid=0 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 client.17343 shutdown 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 msgr2=0x56476284e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x56476284e710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdba4012d20 tx=0x7fdba400f790 comp rx=0 tx=0).stop 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 shutdown_connections 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56476285a330 0x56476285c750 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762855750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x56476284e710 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdb9f400640 0 client.17343 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 --2- 172.21.15.116:0/588799596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x56476284e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 >> 172.21.15.116:0/588799596 conn(0x56476283f210 msgr2=0x56476283e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 shutdown_connections 2024-03-04T07:49:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.711+0000 7fdbc383a740 1 -- 172.21.15.116:0/588799596 wait complete. 2024-03-04T07:49:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.712+0000 7fdbc383a740 20 client.17343 trim_cache size 0 max 16384 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.725+0000 7fdbc383a740 1 Processor -- start 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.725+0000 7fdbc383a740 1 -- start start 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762842b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762855710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762861010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbc383a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1ad040 con 0x564762844690 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbc383a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1af620 con 0x564762849800 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbc383a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1ad640 con 0x564762843ca0 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762855710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762861010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762855710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48324/0 (socket says 172.21.15.116:48324) 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762861010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34540/0 (socket says 172.21.15.116:34540) 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbb1400640 1 -- 172.21.15.116:0/2479588014 learned_addr learned my addr 172.21.15.116:0/2479588014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762842b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 -- 172.21.15.116:0/2479588014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 msgr2=0x564762842b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762842b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 -- 172.21.15.116:0/2479588014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 msgr2=0x564762855710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762855710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbba800640 1 -- 172.21.15.116:0/2479588014 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564762848e40 con 0x564762849800 2024-03-04T07:49:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.726+0000 7fdbb1400640 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762855710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:23.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.727+0000 7fdbba800640 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762861010 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x56476285cfc0 tx=0x7fdbb4010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ce77fe2115ca0bf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.727+0000 7fdbaa600640 1 -- 172.21.15.116:0/2479588014 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4024040 con 0x564762849800 2024-03-04T07:49:23.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.727+0000 7fdbaa600640 1 -- 172.21.15.116:0/2479588014 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb4011e20 con 0x564762849800 2024-03-04T07:49:23.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.727+0000 7fdbaa600640 1 -- 172.21.15.116:0/2479588014 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40050d0 con 0x564762849800 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 -- 172.21.15.116:0/2479588014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 msgr2=0x564762861010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762861010 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x56476285cfc0 tx=0x7fdbb4010860 comp rx=0 tx=0).stop 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 -- 172.21.15.116:0/2479588014 shutdown_connections 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762849800 0x564762861010 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762844690 0x564762855710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 --2- 172.21.15.116:0/2479588014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762842b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 -- 172.21.15.116:0/2479588014 >> 172.21.15.116:0/2479588014 conn(0x56476283f210 msgr2=0x564762842470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 -- 172.21.15.116:0/2479588014 shutdown_connections 2024-03-04T07:49:23.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 -- 172.21.15.116:0/2479588014 wait complete. 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 -- start start 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.728+0000 7fdbc383a740 1 Processor -- start 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762855270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762844690 0x564762853880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbc383a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762849800 0x564762853dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbc383a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1adca0 con 0x564762849800 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbc383a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1aefc0 con 0x564762844690 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbc383a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1ae300 con 0x564762843ca0 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbc383a740 10 client.0 _ll_register_callbacks cb 0x7fdba4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbba800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762849800 0x564762853dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762844690 0x564762853880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762844690 0x564762853880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34552/0 (socket says 172.21.15.116:34552) 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 -- 172.21.15.116:0/1970861757 learned_addr learned my addr 172.21.15.116:0/1970861757 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbba800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762849800 0x564762853dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48340/0 (socket says 172.21.15.116:48340) 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbba800640 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762855270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 -- 172.21.15.116:0/1970861757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 msgr2=0x564762855270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762855270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 -- 172.21.15.116:0/1970861757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762849800 msgr2=0x564762853dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762849800 0x564762853dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.729+0000 7fdbb1400640 1 -- 172.21.15.116:0/1970861757 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564762854300 con 0x564762844690 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdbb1400640 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762844690 0x564762853880 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdba4010920 tx=0x7fdba400fe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5a682c61734c3c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdb9f400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdb9f400640 1 -- 172.21.15.116:0/1970861757 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4005110 con 0x564762844690 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdb9f400640 1 -- 172.21.15.116:0/1970861757 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdba40052b0 con 0x564762844690 2024-03-04T07:49:23.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbb4011e20 con 0x564762844690 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdb9f400640 1 -- 172.21.15.116:0/1970861757 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdba4037050 con 0x564762844690 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdbc383a740 10 client.27482 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5647628b6870 con 0x564762844690 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.730+0000 7fdb9f400640 1 -- 172.21.15.116:0/1970861757 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb4011e20 con 0x564762844690 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdb9f400640 1 -- 172.21.15.116:0/1970861757 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fdba4027620 con 0x564762844690 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbc383a740 10 client.27482 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbc383a740 10 client.27482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbc383a740 20 client.27482 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5647628b71a0 0x5647628b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbc383a740 4 client.27482 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56476284dfc0 con 0x5647628b71a0 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbba800640 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5647628b71a0 0x5647628b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdbba800640 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5647628b71a0 0x5647628b95c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e66e8ba22d6c0154 server_cookie=582b8583c1e88532 in_seq=0 out_seq=0 2024-03-04T07:49:23.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.731+0000 7fdb9f400640 10 client.27482 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdb9f400640 1 -- 172.21.15.116:0/1970861757 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56476284dfc0 con 0x5647628b71a0 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdb9f400640 10 client.27482 handle_command_reply: tid=0 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 client.27482 shutdown 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762844690 msgr2=0x564762853880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762844690 0x564762853880 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fdba4010920 tx=0x7fdba400fe80 comp rx=0 tx=0).stop 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 shutdown_connections 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5647628b71a0 0x5647628b95c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564762849800 0x564762853dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x564762844690 0x564762853880 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdb9f400640 0 client.27482 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 --2- 172.21.15.116:0/1970861757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564762843ca0 0x564762855270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 >> 172.21.15.116:0/1970861757 conn(0x56476283f210 msgr2=0x564762842d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 shutdown_connections 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 1 -- 172.21.15.116:0/1970861757 wait complete. 2024-03-04T07:49:23.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.732+0000 7fdbc383a740 20 client.27482 trim_cache size 0 max 16384 2024-03-04T07:49:23.740 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdba00a1e00 msgr2=0x7fdba00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdba00a1e00 0x7fdba00a42c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0b9490 tx=0x7fdbac0f3810 comp rx=0 tx=0).stop 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 msgr2=0x7fdbbc19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc19df20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fdbac000c00 tx=0x7fdbac02c650 comp rx=0 tx=0).stop 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 shutdown_connections 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fdba00a1e00 0x7fdba00a42c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbbc1097a0 0x7fdbbc1a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbbc108de0 0x7fdbbc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 --2- 172.21.15.116:0/3665595364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdbbc107bf0 0x7fdbbc19df20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 >> 172.21.15.116:0/3665595364 conn(0x7fdbbc076b90 msgr2=0x7fdbbc10c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:23.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 shutdown_connections 2024-03-04T07:49:23.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:23.733+0000 7fdbc0a00640 1 -- 172.21.15.116:0/3665595364 wait complete. 2024-03-04T07:49:24.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:23 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3073956473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:23 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3073956473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:23 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3073956473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:24.754 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:49:24.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:49:24.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:49:24.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 -- 172.21.15.116:0/2387461388 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5948077b70 msgr2=0x7f5948078410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:24.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 --2- 172.21.15.116:0/2387461388 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5948077b70 0x7f5948078410 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5940009860 tx=0x7f594002bb70 comp rx=0 tx=0).stop 2024-03-04T07:49:24.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 -- 172.21.15.116:0/2387461388 shutdown_connections 2024-03-04T07:49:24.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 --2- 172.21.15.116:0/2387461388 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5948077b70 0x7f5948078410 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 --2- 172.21.15.116:0/2387461388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59480771d0 0x7f5948077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 --2- 172.21.15.116:0/2387461388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5948075f90 0x7f59480763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 -- 172.21.15.116:0/2387461388 >> 172.21.15.116:0/2387461388 conn(0x7f59480fed80 msgr2=0x7f59481011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:24.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 -- 172.21.15.116:0/2387461388 shutdown_connections 2024-03-04T07:49:24.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.983+0000 7f5950000640 1 -- 172.21.15.116:0/2387461388 wait complete. 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.984+0000 7f5950000640 1 Processor -- start 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.984+0000 7f5950000640 1 -- start start 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.984+0000 7f5950000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5948075f90 0x7f594819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.984+0000 7f5950000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f59480771d0 0x7f594819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.984+0000 7f5950000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5948077b70 0x7f59481a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f5950000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481138a0 con 0x7f5948075f90 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f5950000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5948113720 con 0x7f59480771d0 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f5950000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5948113a20 con 0x7f5948077b70 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f59480771d0 0x7f594819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5948075f90 0x7f594819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:24.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f59480771d0 0x7f594819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34570/0 (socket says 172.21.15.116:34570) 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 -- 172.21.15.116:0/2760633069 learned_addr learned my addr 172.21.15.116:0/2760633069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594e000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5948077b70 0x7f59481a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 -- 172.21.15.116:0/2760633069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5948077b70 msgr2=0x7f59481a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5948077b70 0x7f59481a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 -- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5948075f90 msgr2=0x7f594819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5948075f90 0x7f594819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 -- 172.21.15.116:0/2760633069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59481a5ac0 con 0x7f59480771d0 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.985+0000 7f594cc00640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f59480771d0 0x7f594819e3e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f593400e9f0 tx=0x7f593400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.986+0000 7f5946000640 1 -- 172.21.15.116:0/2760633069 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400cd80 con 0x7f59480771d0 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.986+0000 7f5946000640 1 -- 172.21.15.116:0/2760633069 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5934005500 con 0x7f59480771d0 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.986+0000 7f5946000640 1 -- 172.21.15.116:0/2760633069 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934010690 con 0x7f59480771d0 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.986+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59481a5db0 con 0x7f59480771d0 2024-03-04T07:49:24.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.986+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59481a6220 con 0x7f59480771d0 2024-03-04T07:49:24.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.988+0000 7f5946000640 1 -- 172.21.15.116:0/2760633069 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f59340046c0 con 0x7f59480771d0 2024-03-04T07:49:24.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.989+0000 7f5952e1f740 1 Processor -- start 2024-03-04T07:49:24.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.989+0000 7f5952e1f740 1 -- start start 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd7ac0 0x557a19fdc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fdcc30 0x557a19fdf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5952e1f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a19eeae20 con 0x557a19fd70d0 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5952e1f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a19f3ca20 con 0x557a19fd7ac0 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5952e1f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a19ee5b00 con 0x557a19fdcc30 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48378/0 (socket says 172.21.15.116:48378) 2024-03-04T07:49:24.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594e000640 1 -- 172.21.15.116:0/111622910 learned_addr learned my addr 172.21.15.116:0/111622910 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5946000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f592c0a1e00 0x7f592c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594d600640 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd7ac0 0x557a19fdc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594cc00640 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fdcc30 0x557a19fdf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594e000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f592c0a1e00 0x7f592c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f5946000640 1 -- 172.21.15.116:0/2760633069 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f59340ce210 con 0x7f59480771d0 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594d600640 1 -- 172.21.15.116:0/111622910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fdcc30 msgr2=0x557a19fdf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594d600640 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fdcc30 0x557a19fdf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594d600640 1 -- 172.21.15.116:0/111622910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 msgr2=0x557a19fd74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594d600640 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.990+0000 7f594d600640 1 -- 172.21.15.116:0/111622910 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a19efae10 con 0x557a19fd7ac0 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.991+0000 7f594d600640 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd7ac0 0x557a19fdc5c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5938009a00 tx=0x7f59380095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=803071985823ecf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.991+0000 7f594e000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f592c0a1e00 0x7f592c0a42c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x557a19fd8320 tx=0x7f594002bb70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:24.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.991+0000 7f593e600640 1 -- 172.21.15.116:0/111622910 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593801d070 con 0x557a19fd7ac0 2024-03-04T07:49:24.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.991+0000 7f593e600640 1 -- 172.21.15.116:0/111622910 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5938003e40 con 0x557a19fd7ac0 2024-03-04T07:49:24.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.991+0000 7f593e600640 1 -- 172.21.15.116:0/111622910 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593800db20 con 0x557a19fd7ac0 2024-03-04T07:49:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 -- 172.21.15.116:0/111622910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd7ac0 msgr2=0x557a19fdc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd7ac0 0x557a19fdc5c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5938009a00 tx=0x7f59380095c0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 -- 172.21.15.116:0/111622910 shutdown_connections 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fdcc30 0x557a19fdf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd7ac0 0x557a19fdc5c0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 --2- 172.21.15.116:0/111622910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 -- 172.21.15.116:0/111622910 >> 172.21.15.116:0/111622910 conn(0x557a19fd2640 msgr2=0x557a19fd2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 -- 172.21.15.116:0/111622910 shutdown_connections 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 -- 172.21.15.116:0/111622910 wait complete. 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.992+0000 7f5952e1f740 1 -- start start 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 1 Processor -- start 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd70d0 0x557a19fe13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fdcc30 0x557a19fe8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a19ef8f40 con 0x557a19fdcc30 2024-03-04T07:49:24.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a19f38ac0 con 0x557a19fd70d0 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a19fdfbe0 con 0x557a19fd7ac0 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f5952e1f740 10 client.0 _ll_register_callbacks cb 0x557a19f37aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f594d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd70d0 0x557a19fe13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f594d600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd70d0 0x557a19fe13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34592/0 (socket says 172.21.15.116:34592) 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f594d600640 1 -- 172.21.15.116:0/2355511652 learned_addr learned my addr 172.21.15.116:0/2355511652 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f594e000640 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fdcc30 0x557a19fe8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.993+0000 7f594cc00640 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f594d600640 1 -- 172.21.15.116:0/2355511652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 msgr2=0x557a19fe1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f594d600640 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f594d600640 1 -- 172.21.15.116:0/2355511652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fdcc30 msgr2=0x557a19fe8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f594d600640 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fdcc30 0x557a19fe8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f594d600640 1 -- 172.21.15.116:0/2355511652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a19d197e0 con 0x557a19fd70d0 2024-03-04T07:49:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f594d600640 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd70d0 0x557a19fe13e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5938000f80 tx=0x7f593800da90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e4da2b51cb45a1d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f5932a00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f5932a00640 1 -- 172.21.15.116:0/2355511652 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593801d040 con 0x557a19fd70d0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a19f49f60 con 0x557a19fd70d0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f5932a00640 1 -- 172.21.15.116:0/2355511652 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f593800dc80 con 0x557a19fd70d0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f5932a00640 1 -- 172.21.15.116:0/2355511652 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f593800de80 con 0x557a19fd70d0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.994+0000 7f5952e1f740 10 client.27494 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557a19d79f70 con 0x557a19fd70d0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5932a00640 1 -- 172.21.15.116:0/2355511652 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557a19f49f60 con 0x557a19fd70d0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5932a00640 1 -- 172.21.15.116:0/2355511652 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f593802c030 con 0x557a19fd70d0 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5952e1f740 10 client.27494 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5952e1f740 10 client.27494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5952e1f740 20 client.27494 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a19fecf50 0x557a19fef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5952e1f740 4 client.27494 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a19f442b0 con 0x557a19fecf50 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.995+0000 7f594cc00640 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a19fecf50 0x557a19fef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.996+0000 7f594cc00640 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a19fecf50 0x557a19fef340 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adcd4ab94a974597 server_cookie=f0035ffc9b1bbcd1 in_seq=0 out_seq=0 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.996+0000 7f5932a00640 10 client.27494 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.996+0000 7f5932a00640 1 -- 172.21.15.116:0/2355511652 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557a19f442b0 con 0x557a19fecf50 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.996+0000 7f5932a00640 10 client.27494 handle_command_reply: tid=0 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.996+0000 7f5952e1f740 1 client.27494 shutdown 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd70d0 msgr2=0x557a19fe13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd70d0 0x557a19fe13e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5938000f80 tx=0x7f593800da90 comp rx=0 tx=0).stop 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 shutdown_connections 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a19fecf50 0x557a19fef340 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fdcc30 0x557a19fe8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2355511652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fd70d0 0x557a19fe13e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 >> 172.21.15.116:0/2355511652 conn(0x557a19fd2640 msgr2=0x557a19fd3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5932a00640 0 client.27494 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 shutdown_connections 2024-03-04T07:49:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 1 -- 172.21.15.116:0/2355511652 wait complete. 2024-03-04T07:49:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:24.997+0000 7f5952e1f740 20 client.27494 trim_cache size 0 max 16384 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.014+0000 7f5952e1f740 1 Processor -- start 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f5952e1f740 1 -- start start 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fd5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f5952e1f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481acf60 con 0x557a19fd70d0 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f5952e1f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481af540 con 0x557a19fdcc30 2024-03-04T07:49:25.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f5952e1f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481ad560 con 0x557a19fd7ac0 2024-03-04T07:49:25.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f594d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f594e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fd5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f594d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48394/0 (socket says 172.21.15.116:48394) 2024-03-04T07:49:25.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f594d600640 1 -- 172.21.15.116:0/975768754 learned_addr learned my addr 172.21.15.116:0/975768754 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.015+0000 7f594cc00640 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f594e000640 1 -- 172.21.15.116:0/975768754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 msgr2=0x557a19fe8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f594e000640 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f594e000640 1 -- 172.21.15.116:0/975768754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 msgr2=0x557a19fd1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f594e000640 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f594e000640 1 -- 172.21.15.116:0/975768754 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a19c11170 con 0x557a19fdcc30 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f594cc00640 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f594e000640 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fd5fb0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f59400095f0 tx=0x7f594002e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a26fbe0022899fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f593dc00640 1 -- 172.21.15.116:0/975768754 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940039040 con 0x557a19fdcc30 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f593dc00640 1 -- 172.21.15.116:0/975768754 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5940034780 con 0x557a19fdcc30 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.016+0000 7f593dc00640 1 -- 172.21.15.116:0/975768754 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940032dd0 con 0x557a19fdcc30 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 -- 172.21.15.116:0/975768754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 msgr2=0x557a19fd5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fd5fb0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f59400095f0 tx=0x7f594002e070 comp rx=0 tx=0).stop 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 -- 172.21.15.116:0/975768754 shutdown_connections 2024-03-04T07:49:25.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fd5fb0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 --2- 172.21.15.116:0/975768754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fd1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 -- 172.21.15.116:0/975768754 >> 172.21.15.116:0/975768754 conn(0x557a19fd2640 msgr2=0x557a19fd58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 -- 172.21.15.116:0/975768754 shutdown_connections 2024-03-04T07:49:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 -- 172.21.15.116:0/975768754 wait complete. 2024-03-04T07:49:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 -- start start 2024-03-04T07:49:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.017+0000 7f5952e1f740 1 Processor -- start 2024-03-04T07:49:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fe6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f5952e1f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fe79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f5952e1f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481adbc0 con 0x557a19fd70d0 2024-03-04T07:49:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f5952e1f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481aeee0 con 0x557a19fdcc30 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f5952e1f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481ae220 con 0x557a19fd7ac0 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f5952e1f740 10 client.0 _ll_register_callbacks cb 0x7f59340971a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f594d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fe6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f594e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fe79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f594cc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f594d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fe6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48402/0 (socket says 172.21.15.116:48402) 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f594d600640 1 -- 172.21.15.116:0/2067926298 learned_addr learned my addr 172.21.15.116:0/2067926298 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.018+0000 7f594e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fe79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34608/0 (socket says 172.21.15.116:34608) 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594cc00640 1 -- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 msgr2=0x557a19fe79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594cc00640 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fe79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594cc00640 1 -- 172.21.15.116:0/2067926298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 msgr2=0x557a19fe6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594cc00640 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fe6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594cc00640 1 -- 172.21.15.116:0/2067926298 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a19fe7fd0 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594d600640 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fe6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594e000640 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fe79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f594cc00640 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe69d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f593409ac00 tx=0x7f593409b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a86fa14fcfdc867 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f5932a00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f5932a00640 1 -- 172.21.15.116:0/2067926298 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59340a36a0 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f5932a00640 1 -- 172.21.15.116:0/2067926298 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5934097570 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5940034780 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f5932a00640 1 -- 172.21.15.116:0/2067926298 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59340a3210 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f5952e1f740 10 client.27382 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.019+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557a1a048c80 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5932a00640 1 -- 172.21.15.116:0/2067926298 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5940034780 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5932a00640 1 -- 172.21.15.116:0/2067926298 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f59340d00d0 con 0x557a19fd7ac0 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5952e1f740 10 client.27382 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5952e1f740 10 client.27382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5952e1f740 20 client.27382 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a1a049540 0x557a1a04b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5952e1f740 4 client.27382 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557a19ec2a40 con 0x557a1a049540 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.020+0000 7f594d600640 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a1a049540 0x557a1a04b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.021+0000 7f594d600640 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a1a049540 0x557a1a04b980 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba1a90df63d30b31 server_cookie=33993367cf97cc4a in_seq=0 out_seq=0 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.021+0000 7f5932a00640 10 client.27382 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5932a00640 1 -- 172.21.15.116:0/2067926298 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557a19ec2a40 con 0x557a1a049540 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5932a00640 10 client.27382 handle_command_reply: tid=0 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 client.27382 shutdown 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 msgr2=0x557a19fe69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe69d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f593409ac00 tx=0x7f593409b6c0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 shutdown_connections 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557a1a049540 0x557a1a04b980 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557a19fdcc30 0x557a19fe79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5932a00640 0 client.27382 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557a19fd7ac0 0x557a19fe69d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 --2- 172.21.15.116:0/2067926298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a19fd70d0 0x557a19fe6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 >> 172.21.15.116:0/2067926298 conn(0x557a19fd2640 msgr2=0x557a19fd6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 shutdown_connections 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 1 -- 172.21.15.116:0/2067926298 wait complete. 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.022+0000 7f5952e1f740 20 client.27382 trim_cache size 0 max 16384 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:49:25.028 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "955aed48-4190-4c27-83e3-edf4d80d47b2", 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f592c0a1e00 msgr2=0x7f592c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f592c0a1e00 0x7f592c0a42c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x557a19fd8320 tx=0x7f594002bb70 comp rx=0 tx=0).stop 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f59480771d0 msgr2=0x7f594819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f59480771d0 0x7f594819e3e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f593400e9f0 tx=0x7f593400edf0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 shutdown_connections 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f592c0a1e00 0x7f592c0a42c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5948077b70 0x7f59481a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f59480771d0 0x7f594819e3e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 --2- 172.21.15.116:0/2760633069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5948075f90 0x7f594819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 >> 172.21.15.116:0/2760633069 conn(0x7f59480fed80 msgr2=0x7f5948100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 shutdown_connections 2024-03-04T07:49:25.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.023+0000 7f5950000640 1 -- 172.21.15.116:0/2760633069 wait complete. 2024-03-04T07:49:25.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:49:25.041 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:49:25.113 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:24 smithi116 ceph-mon[32150]: pgmap v660: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 -- 172.21.15.116:0/1122764155 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a00771d0 msgr2=0x7fa9a0077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 --2- 172.21.15.116:0/1122764155 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a00771d0 0x7fa9a0077630 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa988009900 tx=0x7fa98802bc40 comp rx=0 tx=0).stop 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 -- 172.21.15.116:0/1122764155 shutdown_connections 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 --2- 172.21.15.116:0/1122764155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9a0077b70 0x7fa9a0078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 --2- 172.21.15.116:0/1122764155 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a00771d0 0x7fa9a0077630 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 --2- 172.21.15.116:0/1122764155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0075f90 0x7fa9a00763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 -- 172.21.15.116:0/1122764155 >> 172.21.15.116:0/1122764155 conn(0x7fa9a00fed80 msgr2=0x7fa9a01011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.247+0000 7fa9a5200640 1 -- 172.21.15.116:0/1122764155 shutdown_connections 2024-03-04T07:49:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.248+0000 7fa9a5200640 1 -- 172.21.15.116:0/1122764155 wait complete. 2024-03-04T07:49:25.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.248+0000 7fa9a5200640 1 Processor -- start 2024-03-04T07:49:25.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa9a5200640 1 -- start start 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa9a5200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a0075f90 0x7fa9a019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa9a5200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9a00771d0 0x7fa9a019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa9a5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0077b70 0x7fa9a01a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa9a5200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01138a0 con 0x7fa9a0077b70 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa9a5200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a0113720 con 0x7fa9a0075f90 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa9a5200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a0113a20 con 0x7fa9a00771d0 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a0075f90 0x7fa9a019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a0075f90 0x7fa9a019ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34626/0 (socket says 172.21.15.116:34626) 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99de00640 1 -- 172.21.15.116:0/617103759 learned_addr learned my addr 172.21.15.116:0/617103759 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa995400640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9a00771d0 0x7fa9a019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99e800640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0077b70 0x7fa9a01a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99de00640 1 -- 172.21.15.116:0/617103759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9a00771d0 msgr2=0x7fa9a019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99de00640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9a00771d0 0x7fa9a019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99de00640 1 -- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0077b70 msgr2=0x7fa9a01a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.249+0000 7fa99de00640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0077b70 0x7fa9a01a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.250+0000 7fa99de00640 1 -- 172.21.15.116:0/617103759 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a01a5af0 con 0x7fa9a0075f90 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.250+0000 7fa99de00640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a0075f90 0x7fa9a019ded0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa99000e9f0 tx=0x7fa99000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.250+0000 7fa997400640 1 -- 172.21.15.116:0/617103759 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99000cd80 con 0x7fa9a0075f90 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.250+0000 7fa997400640 1 -- 172.21.15.116:0/617103759 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa990010910 con 0x7fa9a0075f90 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.250+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a01a5de0 con 0x7fa9a0075f90 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.250+0000 7fa997400640 1 -- 172.21.15.116:0/617103759 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990018990 con 0x7fa9a0075f90 2024-03-04T07:49:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.250+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a01a6250 con 0x7fa9a0075f90 2024-03-04T07:49:25.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.254+0000 7fa9a7fc4740 1 Processor -- start 2024-03-04T07:49:25.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.254+0000 7fa9a7fc4740 1 -- start start 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8d7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8d9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4cf7ff090 con 0x55c4cf8d2540 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4cf836d30 con 0x55c4cf8d76b0 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4cf7dfa30 con 0x55c4cf8d1b50 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa997400640 1 -- 172.21.15.116:0/617103759 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa990010ac0 con 0x7fa9a0075f90 2024-03-04T07:49:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa99e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa995400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8d9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa99e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41306/0 (socket says 172.21.15.116:41306) 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa99e800640 1 -- 172.21.15.116:0/1159450634 learned_addr learned my addr 172.21.15.116:0/1159450634 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.255+0000 7fa99de00640 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8d7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa99e800640 1 -- 172.21.15.116:0/1159450634 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 msgr2=0x55c4cf8d9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa99e800640 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8d9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa99e800640 1 -- 172.21.15.116:0/1159450634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 msgr2=0x55c4cf8d7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa99e800640 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8d7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa99e800640 1 -- 172.21.15.116:0/1159450634 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4cf6157d0 con 0x55c4cf8d1b50 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa995400640 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8d9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa99e800640 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d1f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa9980039f0 tx=0x7fa998003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e84e4dc53875e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa98f400640 1 -- 172.21.15.116:0/1159450634 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99801a470 con 0x55c4cf8d1b50 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa98f400640 1 -- 172.21.15.116:0/1159450634 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa998015070 con 0x55c4cf8d1b50 2024-03-04T07:49:25.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.256+0000 7fa98f400640 1 -- 172.21.15.116:0/1159450634 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998010880 con 0x55c4cf8d1b50 2024-03-04T07:49:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa997400640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9840a1e00 0x7fa9840a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa997400640 1 -- 172.21.15.116:0/617103759 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa9900ce090 con 0x7fa9a0075f90 2024-03-04T07:49:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/1159450634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 msgr2=0x55c4cf8d1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa99de00640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9840a1e00 0x7fa9840a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d1f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa9980039f0 tx=0x7fa998003eb0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/1159450634 shutdown_connections 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8d9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8d7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/1159450634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d1f70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/1159450634 >> 172.21.15.116:0/1159450634 conn(0x55c4cf8cd0c0 msgr2=0x55c4cf8cd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/1159450634 shutdown_connections 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.257+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/1159450634 wait complete. 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa99de00640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9840a1e00 0x7fa9840a42c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa990097290 tx=0x7fa990000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 -- start start 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 Processor -- start 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8dbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8dc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4cf7d50c0 con 0x55c4cf8d76b0 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4cf832dd0 con 0x55c4cf8d1b50 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.258+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4cf8da660 con 0x55c4cf8d2540 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa9a7fc4740 10 client.0 _ll_register_callbacks cb 0x55c4cf7f2860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa99e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8dbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8dc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8dc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:41322/0 (socket says 172.21.15.116:41322) 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 -- 172.21.15.116:0/2859435709 learned_addr learned my addr 172.21.15.116:0/2859435709 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa99e800640 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 -- 172.21.15.116:0/2859435709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 msgr2=0x55c4cf8dbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8dbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 -- 172.21.15.116:0/2859435709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 msgr2=0x55c4cf8e3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 -- 172.21.15.116:0/2859435709 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4cf54b3f0 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa99e800640 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa995400640 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8dc440 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa988009900 tx=0x7fa988004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf5423315c575241 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.259+0000 7fa983e00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.260+0000 7fa983e00640 1 -- 172.21.15.116:0/2859435709 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988005d50 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.260+0000 7fa983e00640 1 -- 172.21.15.116:0/2859435709 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa988005230 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.260+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa998015070 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.260+0000 7fa983e00640 1 -- 172.21.15.116:0/2859435709 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa988005430 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.260+0000 7fa9a7fc4740 10 client.27394 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.260+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c4cf50b0f0 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.260+0000 7fa983e00640 1 -- 172.21.15.116:0/2859435709 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa998015070 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa983e00640 1 -- 172.21.15.116:0/2859435709 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa988003a60 con 0x55c4cf8d2540 2024-03-04T07:49:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa9a7fc4740 10 client.27394 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa9a7fc4740 10 client.27394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa9a7fc4740 20 client.27394 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf8e7a20 0x55c4cf8e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa9a7fc4740 4 client.27394 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4cf844270 con 0x55c4cf8e7a20 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa99e800640 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf8e7a20 0x55c4cf8e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa99e800640 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf8e7a20 0x55c4cf8e9e40 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=804fbe29e55e83a7 server_cookie=7273114177193f13 in_seq=0 out_seq=0 2024-03-04T07:49:25.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.261+0000 7fa983e00640 10 client.27394 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa983e00640 1 -- 172.21.15.116:0/2859435709 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c4cf844270 con 0x55c4cf8e7a20 2024-03-04T07:49:25.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa983e00640 10 client.27394 handle_command_reply: tid=0 2024-03-04T07:49:25.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 client.27394 shutdown 2024-03-04T07:49:25.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 msgr2=0x55c4cf8dc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8dc440 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa988009900 tx=0x7fa988004270 comp rx=0 tx=0).stop 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 shutdown_connections 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf8e7a20 0x55c4cf8e9e40 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8dc440 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2859435709 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8dbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa983e00640 0 client.27394 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 >> 172.21.15.116:0/2859435709 conn(0x55c4cf8cd0c0 msgr2=0x55c4cf8cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.262+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 shutdown_connections 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.263+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2859435709 wait complete. 2024-03-04T07:49:25.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.263+0000 7fa9a7fc4740 20 client.27394 trim_cache size 0 max 16384 2024-03-04T07:49:25.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:24 smithi152 ceph-mon[37154]: pgmap v660: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 Processor -- start 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 -- start start 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01ad0a0 con 0x55c4cf8d76b0 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01af680 con 0x55c4cf8d1b50 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01ad6a0 con 0x55c4cf8d2540 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.280+0000 7fa99e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa99e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:34656/0 (socket says 172.21.15.116:34656) 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa99e800640 1 -- 172.21.15.116:0/4092440307 learned_addr learned my addr 172.21.15.116:0/4092440307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa99e800640 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa995400640 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa995400640 1 -- 172.21.15.116:0/4092440307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 msgr2=0x55c4cf8d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa995400640 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa995400640 1 -- 172.21.15.116:0/4092440307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 msgr2=0x55c4cf8ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa995400640 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8ee6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa995400640 1 -- 172.21.15.116:0/4092440307 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4cf8dcef0 con 0x55c4cf8d2540 2024-03-04T07:49:25.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa99e800640 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa995400640 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e3440 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa988003dd0 tx=0x7fa988001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6dcb7704daf6760 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.281+0000 7fa98ea00640 1 -- 172.21.15.116:0/4092440307 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988005d50 con 0x55c4cf8d2540 2024-03-04T07:49:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa98ea00640 1 -- 172.21.15.116:0/4092440307 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa988041070 con 0x55c4cf8d2540 2024-03-04T07:49:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa98ea00640 1 -- 172.21.15.116:0/4092440307 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9880466b0 con 0x55c4cf8d2540 2024-03-04T07:49:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/4092440307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 msgr2=0x55c4cf8e3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e3440 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa988003dd0 tx=0x7fa988001e70 comp rx=0 tx=0).stop 2024-03-04T07:49:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/4092440307 shutdown_connections 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e3440 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/4092440307 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/4092440307 >> 172.21.15.116:0/4092440307 conn(0x55c4cf8cd0c0 msgr2=0x55c4cf8d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/4092440307 shutdown_connections 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.282+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/4092440307 wait complete. 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 -- start start 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 Processor -- start 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8e13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01add00 con 0x55c4cf8d2540 2024-03-04T07:49:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01af020 con 0x55c4cf8d76b0 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01ae360 con 0x55c4cf8d1b50 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa9a7fc4740 10 client.0 _ll_register_callbacks cb 0x7fa98802cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa995400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa995400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48440/0 (socket says 172.21.15.116:48440) 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa995400640 1 -- 172.21.15.116:0/2665870830 learned_addr learned my addr 172.21.15.116:0/2665870830 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.283+0000 7fa99e800640 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8e13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa99e800640 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa995400640 1 -- 172.21.15.116:0/2665870830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 msgr2=0x55c4cf8e13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa995400640 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8e13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa995400640 1 -- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 msgr2=0x55c4cf8e2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa995400640 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa995400640 1 -- 172.21.15.116:0/2665870830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4cf8e2f20 con 0x55c4cf8d2540 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa995400640 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e1920 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa988003e50 tx=0x7fa98802bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7b5096cbe55e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa983e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa983e00640 1 -- 172.21.15.116:0/2665870830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988005d50 con 0x55c4cf8d2540 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa983e00640 1 -- 172.21.15.116:0/2665870830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa9880344f0 con 0x55c4cf8d2540 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.284+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa988041070 con 0x55c4cf8d2540 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa983e00640 1 -- 172.21.15.116:0/2665870830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa988046450 con 0x55c4cf8d2540 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 10 client.17385 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55c4cf943b60 con 0x55c4cf8d2540 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa983e00640 1 -- 172.21.15.116:0/2665870830 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa988041070 con 0x55c4cf8d2540 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa983e00640 1 -- 172.21.15.116:0/2665870830 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa98805e020 con 0x55c4cf8d2540 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 10 client.17385 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 10 client.17385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 20 client.17385 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf944480 0x55c4cf9468a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 4 client.17385 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:49:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c4cf8dbcf0 con 0x55c4cf944480 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.285+0000 7fa99e800640 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf944480 0x55c4cf9468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa99e800640 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf944480 0x55c4cf9468a0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6867d915d21c740d server_cookie=4139daa4f59caa8d in_seq=0 out_seq=0 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa983e00640 10 client.17385 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa983e00640 1 -- 172.21.15.116:0/2665870830 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c4cf8dbcf0 con 0x55c4cf944480 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa983e00640 10 client.17385 handle_command_reply: tid=0 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 client.17385 shutdown 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 msgr2=0x55c4cf8e1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e1920 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa988003e50 tx=0x7fa98802bc40 comp rx=0 tx=0).stop 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 shutdown_connections 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55c4cf944480 0x55c4cf9468a0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55c4cf8d76b0 0x55c4cf8e2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4cf8d2540 0x55c4cf8e1920 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa983e00640 0 client.17385 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 --2- 172.21.15.116:0/2665870830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4cf8d1b50 0x55c4cf8e13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 >> 172.21.15.116:0/2665870830 conn(0x55c4cf8cd0c0 msgr2=0x55c4cf8d0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.286+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 shutdown_connections 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a7fc4740 1 -- 172.21.15.116:0/2665870830 wait complete. 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a7fc4740 20 client.17385 trim_cache size 0 max 16384 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stdout: "955aed48-4190-4c27-83e3-edf4d80d47b2": { 2024-03-04T07:49:25.293 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:49:25.294 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "955aed48-4190-4c27-83e3-edf4d80d47b2", 2024-03-04T07:49:25.294 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:49:25.294 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:49:25.294 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:49:25.294 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9840a1e00 msgr2=0x7fa9840a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9840a1e00 0x7fa9840a42c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa990097290 tx=0x7fa990000bc0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a0075f90 msgr2=0x7fa9a019ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a0075f90 0x7fa9a019ded0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa99000e9f0 tx=0x7fa99000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 shutdown_connections 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a0077b70 0x7fa9a01a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9840a1e00 0x7fa9840a42c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9a00771d0 0x7fa9a019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 --2- 172.21.15.116:0/617103759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9a0075f90 0x7fa9a019ded0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.287+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 >> 172.21.15.116:0/617103759 conn(0x7fa9a00fed80 msgr2=0x7fa9a0100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.288+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 shutdown_connections 2024-03-04T07:49:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:25.288+0000 7fa9a5200640 1 -- 172.21.15.116:0/617103759 wait complete. 2024-03-04T07:49:25.305 INFO:tasks.cephfs.filesystem:scrub status for tag:955aed48-4190-4c27-83e3-edf4d80d47b2 - {'path': '/', 'tag': '955aed48-4190-4c27-83e3-edf4d80d47b2', 'options': 'recursive,force'} 2024-03-04T07:49:25.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:24 smithi133 ceph-mon[36505]: pgmap v660: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:26.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:25 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:49:26.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:25 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:26.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:25 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:49:26.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:25 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:25 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:49:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:25 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:25 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:49:26.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:25 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:25 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:49:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:25 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:25 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:49:26.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:25 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:27.238 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:26 smithi133 ceph-mon[36505]: pgmap v661: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:49:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:26 smithi116 ceph-mon[32150]: pgmap v661: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:49:27.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:26 smithi152 ceph-mon[37154]: pgmap v661: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-03-04T07:49:28.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:27 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:49:28.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:27 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:49:28.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:27 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:49:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:28 smithi116 ceph-mon[32150]: pgmap v662: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:28 smithi152 ceph-mon[37154]: pgmap v662: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:29.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:28 smithi133 ceph-mon[36505]: pgmap v662: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:30 smithi116 ceph-mon[32150]: pgmap v663: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:30 smithi152 ceph-mon[37154]: pgmap v663: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:31.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:30 smithi133 ceph-mon[36505]: pgmap v663: 97 pgs: 97 active+clean; 2.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:49:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 -- 172.21.15.116:0/1219622175 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc104a20 msgr2=0x7fb2fc104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 --2- 172.21.15.116:0/1219622175 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc104a20 0x7fb2fc104ea0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009900 tx=0x7fb2e802bc50 comp rx=0 tx=0).stop 2024-03-04T07:49:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 -- 172.21.15.116:0/1219622175 shutdown_connections 2024-03-04T07:49:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 --2- 172.21.15.116:0/1219622175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc1053e0 0x7fb2fc10bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 --2- 172.21.15.116:0/1219622175 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc104a20 0x7fb2fc104ea0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 --2- 172.21.15.116:0/1219622175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2fc103830 0x7fb2fc103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 -- 172.21.15.116:0/1219622175 >> 172.21.15.116:0/1219622175 conn(0x7fb2fc0fedc0 msgr2=0x7fb2fc1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 -- 172.21.15.116:0/1219622175 shutdown_connections 2024-03-04T07:49:32.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.794+0000 7fb304600640 1 -- 172.21.15.116:0/1219622175 wait complete. 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.795+0000 7fb304600640 1 Processor -- start 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.795+0000 7fb304600640 1 -- start start 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.795+0000 7fb304600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc103830 0x7fb2fc199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb304600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2fc104a20 0x7fb2fc19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb304600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc1053e0 0x7fb2fc1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb304600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc10f4b0 con 0x7fb2fc1053e0 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb304600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc10f330 con 0x7fb2fc103830 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb304600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc10f630 con 0x7fb2fc104a20 2024-03-04T07:49:32.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb302600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc1053e0 0x7fb2fc1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc103830 0x7fb2fc199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc103830 0x7fb2fc199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47982/0 (socket says 172.21.15.116:47982) 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 -- 172.21.15.116:0/4015499445 learned_addr learned my addr 172.21.15.116:0/4015499445 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301200640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2fc104a20 0x7fb2fc19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 -- 172.21.15.116:0/4015499445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2fc104a20 msgr2=0x7fb2fc19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2fc104a20 0x7fb2fc19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 -- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc1053e0 msgr2=0x7fb2fc1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc1053e0 0x7fb2fc1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb301c00640 1 -- 172.21.15.116:0/4015499445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc1a1720 con 0x7fb2fc103830 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.796+0000 7fb302600640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc1053e0 0x7fb2fc1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.797+0000 7fb301c00640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc103830 0x7fb2fc199b00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000e9c0 tx=0x7fb2f000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.797+0000 7fb2fa000640 1 -- 172.21.15.116:0/4015499445 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f000cd80 con 0x7fb2fc103830 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.797+0000 7fb2fa000640 1 -- 172.21.15.116:0/4015499445 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2f00035f0 con 0x7fb2fc103830 2024-03-04T07:49:32.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.797+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc1a1a10 con 0x7fb2fc103830 2024-03-04T07:49:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.797+0000 7fb2fa000640 1 -- 172.21.15.116:0/4015499445 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0010690 con 0x7fb2fc103830 2024-03-04T07:49:32.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.797+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc1a1f30 con 0x7fb2fc103830 2024-03-04T07:49:32.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 Processor -- start 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb2fa000640 1 -- 172.21.15.116:0/4015499445 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb2f00037a0 con 0x7fb2fc103830 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 -- start start 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d5dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5e240 0x55b229d62d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 0x55b229d65840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b229cc3370 con 0x55b229d5d850 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b229c6c550 con 0x55b229d5e240 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.800+0000 7fb307205740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b229cbf410 con 0x55b229d633b0 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d5dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d5dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50936/0 (socket says 172.21.15.116:50936) 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 -- 172.21.15.116:0/461231488 learned_addr learned my addr 172.21.15.116:0/461231488 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:32.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb301c00640 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5e240 0x55b229d62d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb301200640 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 0x55b229d65840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 -- 172.21.15.116:0/461231488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 msgr2=0x55b229d65840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 0x55b229d65840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 -- 172.21.15.116:0/461231488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5e240 msgr2=0x55b229d62d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5e240 0x55b229d62d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 -- 172.21.15.116:0/461231488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b229c803c0 con 0x55b229d5d850 2024-03-04T07:49:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb301c00640 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5e240 0x55b229d62d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.801+0000 7fb302600640 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d5dc70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb2f40117e0 tx=0x7fb2f4011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf2679cd31b1098 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.802+0000 7fb2eea00640 1 -- 172.21.15.116:0/461231488 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4011310 con 0x55b229d5d850 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.802+0000 7fb2eea00640 1 -- 172.21.15.116:0/461231488 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2f4003e50 con 0x55b229d5d850 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.802+0000 7fb2eea00640 1 -- 172.21.15.116:0/461231488 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f40045b0 con 0x55b229d5d850 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.802+0000 7fb2fa000640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb2e40a1ea0 0x7fb2e40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 -- 172.21.15.116:0/461231488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 msgr2=0x55b229d5dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d5dc70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb2f40117e0 tx=0x7fb2f4011ca0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb301c00640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb2e40a1ea0 0x7fb2e40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb2fa000640 1 -- 172.21.15.116:0/4015499445 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb2f00ce080 con 0x7fb2fc103830 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 -- 172.21.15.116:0/461231488 shutdown_connections 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 0x55b229d65840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5e240 0x55b229d62d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 --2- 172.21.15.116:0/461231488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d5dc70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 -- 172.21.15.116:0/461231488 >> 172.21.15.116:0/461231488 conn(0x55b229d58dc0 msgr2=0x55b229d591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 -- 172.21.15.116:0/461231488 shutdown_connections 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 -- 172.21.15.116:0/461231488 wait complete. 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb301c00640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb2e40a1ea0 0x7fb2e40a4360 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000c6a0 tx=0x7fb2f00d1b00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.803+0000 7fb307205740 1 -- start start 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 1 Processor -- start 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d67b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d680c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b229d66360 con 0x55b229d5d850 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b229c7f9b0 con 0x55b229d633b0 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b229d664e0 con 0x55b229d5e240 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb307205740 10 client.0 _ll_register_callbacks cb 0x55b229cbb340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.804+0000 7fb302600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d67b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d680c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d680c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:54580/0 (socket says 172.21.15.116:54580) 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 -- 172.21.15.116:0/1257403130 learned_addr learned my addr 172.21.15.116:0/1257403130 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb302600640 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 -- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 msgr2=0x55b229d6f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 -- 172.21.15.116:0/1257403130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 msgr2=0x55b229d67b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d67b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 -- 172.21.15.116:0/1257403130 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2299d8210 con 0x55b229d5e240 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb302600640 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d67b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb301200640 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d680c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009900 tx=0x7fb2e802c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12417f1bcafa6679 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.805+0000 7fb2e3400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.806+0000 7fb2e3400640 1 -- 172.21.15.116:0/1257403130 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8043070 con 0x55b229d5e240 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.806+0000 7fb2e3400640 1 -- 172.21.15.116:0/1257403130 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb2e8005030 con 0x55b229d5e240 2024-03-04T07:49:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.806+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2f4003e50 con 0x55b229d5e240 2024-03-04T07:49:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.806+0000 7fb2e3400640 1 -- 172.21.15.116:0/1257403130 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2e80044d0 con 0x55b229d5e240 2024-03-04T07:49:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.806+0000 7fb307205740 10 client.27412 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.806+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b229aa4620 con 0x55b229d5e240 2024-03-04T07:49:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.806+0000 7fb2e3400640 1 -- 172.21.15.116:0/1257403130 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2f4003e50 con 0x55b229d5e240 2024-03-04T07:49:32.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb2e3400640 1 -- 172.21.15.116:0/1257403130 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb2e80035f0 con 0x55b229d5e240 2024-03-04T07:49:32.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb307205740 10 client.27412 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:32.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb307205740 10 client.27412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:32.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb307205740 20 client.27412 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:32.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb307205740 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229d73990 0x55b229d75dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb307205740 4 client.27412 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:32.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b229cd08b0 con 0x55b229d73990 2024-03-04T07:49:32.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.807+0000 7fb302600640 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229d73990 0x55b229d75dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.808+0000 7fb302600640 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229d73990 0x55b229d75dc0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f5a64b551aa5ec7 server_cookie=714c9fdaf5f502c4 in_seq=0 out_seq=0 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.808+0000 7fb2e3400640 10 client.27412 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.808+0000 7fb2e3400640 1 -- 172.21.15.116:0/1257403130 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b229cd08b0 con 0x55b229d73990 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.808+0000 7fb2e3400640 10 client.27412 handle_command_reply: tid=0 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.808+0000 7fb307205740 1 client.27412 shutdown 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 msgr2=0x55b229d680c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d680c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009900 tx=0x7fb2e802c680 comp rx=0 tx=0).stop 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 shutdown_connections 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229d73990 0x55b229d75dc0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d680c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb2e3400640 0 client.27412 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:32.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 --2- 172.21.15.116:0/1257403130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d67b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 >> 172.21.15.116:0/1257403130 conn(0x55b229d58dc0 msgr2=0x55b229d57fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:32.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 shutdown_connections 2024-03-04T07:49:32.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 1 -- 172.21.15.116:0/1257403130 wait complete. 2024-03-04T07:49:32.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.809+0000 7fb307205740 20 client.27412 trim_cache size 0 max 16384 2024-03-04T07:49:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.824+0000 7fb307205740 1 Processor -- start 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.824+0000 7fb307205740 1 -- start start 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb307205740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5d850 0x55b229d5c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb307205740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5e240 0x55b229d6f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb307205740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 0x55b229d5c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb307205740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1a8cb0 con 0x55b229d5e240 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb307205740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1ab290 con 0x55b229d5d850 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb307205740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1a92b0 con 0x55b229d633b0 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5e240 0x55b229d6f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5e240 0x55b229d6f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50944/0 (socket says 172.21.15.116:50944) 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 -- 172.21.15.116:0/4044690953 learned_addr learned my addr 172.21.15.116:0/4044690953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb302600640 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5d850 0x55b229d5c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 -- 172.21.15.116:0/4044690953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 msgr2=0x55b229d5c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 0x55b229d5c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 -- 172.21.15.116:0/4044690953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5d850 msgr2=0x55b229d5c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5d850 0x55b229d5c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb301200640 1 -- 172.21.15.116:0/4044690953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b229d5ead0 con 0x55b229d5e240 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.825+0000 7fb302600640 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5d850 0x55b229d5c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.826+0000 7fb301200640 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5e240 0x55b229d6f0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8003f80 tx=0x7fb2e802c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2613e1572e0f5281 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.826+0000 7fb2ee000640 1 -- 172.21.15.116:0/4044690953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8003c00 con 0x55b229d5e240 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.826+0000 7fb2ee000640 1 -- 172.21.15.116:0/4044690953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2e80035f0 con 0x55b229d5e240 2024-03-04T07:49:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.826+0000 7fb2ee000640 1 -- 172.21.15.116:0/4044690953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80052a0 con 0x55b229d5e240 2024-03-04T07:49:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.826+0000 7fb307205740 1 -- 172.21.15.116:0/4044690953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5e240 msgr2=0x55b229d6f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.826+0000 7fb307205740 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5e240 0x55b229d6f0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8003f80 tx=0x7fb2e802c6f0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 -- 172.21.15.116:0/4044690953 shutdown_connections 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d633b0 0x55b229d5c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5e240 0x55b229d6f0c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 --2- 172.21.15.116:0/4044690953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d5d850 0x55b229d5c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 -- 172.21.15.116:0/4044690953 >> 172.21.15.116:0/4044690953 conn(0x55b229d58dc0 msgr2=0x55b229d67b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 -- 172.21.15.116:0/4044690953 shutdown_connections 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 -- 172.21.15.116:0/4044690953 wait complete. 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 -- start start 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.827+0000 7fb307205740 1 Processor -- start 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb307205740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d6c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb307205740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d6ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb307205740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb307205740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1a9910 con 0x55b229d5d850 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb307205740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1aac30 con 0x55b229d633b0 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb307205740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1a9f70 con 0x55b229d5e240 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb307205740 10 client.0 _ll_register_callbacks cb 0x7fb2e8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d6c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d6c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50954/0 (socket says 172.21.15.116:50954) 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 -- 172.21.15.116:0/2329450743 learned_addr learned my addr 172.21.15.116:0/2329450743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb301200640 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d6ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 -- 172.21.15.116:0/2329450743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 msgr2=0x55b229d6ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d6ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 -- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 msgr2=0x55b229d6dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.828+0000 7fb302600640 1 -- 172.21.15.116:0/2329450743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b229d6e2b0 con 0x55b229d5d850 2024-03-04T07:49:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb302600640 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d6c770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4006010 tx=0x7fb2f40039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5951286659238358 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb2e3400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb2e3400640 1 -- 172.21.15.116:0/2329450743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4023040 con 0x55b229d5d850 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb2e3400640 1 -- 172.21.15.116:0/2329450743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb2f402f040 con 0x55b229d5d850 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2e80035f0 con 0x55b229d5d850 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb2e3400640 1 -- 172.21.15.116:0/2329450743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2f400fd60 con 0x55b229d5d850 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb307205740 10 client.17403 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b229dcf290 con 0x55b229d5d850 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.829+0000 7fb2e3400640 1 -- 172.21.15.116:0/2329450743 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2e80035f0 con 0x55b229d5d850 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb2e3400640 1 -- 172.21.15.116:0/2329450743 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb2f4018020 con 0x55b229d5d850 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb307205740 10 client.17403 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb307205740 10 client.17403 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb307205740 20 client.17403 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb307205740 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229dcfa20 0x55b229dd1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb307205740 4 client.17403 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:49:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b229d630e0 con 0x55b229dcfa20 2024-03-04T07:49:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.830+0000 7fb301200640 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229dcfa20 0x55b229dd1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.831+0000 7fb2e3400640 10 client.17403 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.831+0000 7fb301200640 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229dcfa20 0x55b229dd1e40 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8372b3bfb14fd7e server_cookie=724ee130680aa704 in_seq=0 out_seq=0 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.847+0000 7fb2e3400640 1 -- 172.21.15.116:0/2329450743 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b229d630e0 con 0x55b229dcfa20 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.847+0000 7fb2e3400640 10 client.17403 handle_command_reply: tid=0 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.847+0000 7fb307205740 1 client.17403 shutdown 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.847+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 msgr2=0x55b229d6c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.847+0000 7fb307205740 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d6c770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4006010 tx=0x7fb2f40039f0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 shutdown_connections 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b229dcfa20 0x55b229dd1e40 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b229d633b0 0x55b229d6dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b229d5e240 0x55b229d6ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 --2- 172.21.15.116:0/2329450743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b229d5d850 0x55b229d6c770 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb2e3400640 0 client.17403 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 >> 172.21.15.116:0/2329450743 conn(0x55b229d58dc0 msgr2=0x55b229d67b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 shutdown_connections 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 1 -- 172.21.15.116:0/2329450743 wait complete. 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.848+0000 7fb307205740 20 client.17403 trim_cache size 0 max 16384 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:49:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:49:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb2e40a1ea0 msgr2=0x7fb2e40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb2e40a1ea0 0x7fb2e40a4360 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000c6a0 tx=0x7fb2f00d1b00 comp rx=0 tx=0).stop 2024-03-04T07:49:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc103830 msgr2=0x7fb2fc199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc103830 0x7fb2fc199b00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000e9c0 tx=0x7fb2f000ee80 comp rx=0 tx=0).stop 2024-03-04T07:49:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 shutdown_connections 2024-03-04T07:49:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2fc1053e0 0x7fb2fc1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb2e40a1ea0 0x7fb2e40a4360 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb2fc104a20 0x7fb2fc19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 --2- 172.21.15.116:0/4015499445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb2fc103830 0x7fb2fc199b00 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 >> 172.21.15.116:0/4015499445 conn(0x7fb2fc0fedc0 msgr2=0x7fb2fc107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 shutdown_connections 2024-03-04T07:49:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:32.849+0000 7fb304600640 1 -- 172.21.15.116:0/4015499445 wait complete. 2024-03-04T07:49:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:32 smithi116 ceph-mon[32150]: pgmap v664: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:32 smithi152 ceph-mon[37154]: pgmap v664: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:33.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:32 smithi133 ceph-mon[36505]: pgmap v664: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:49:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:34 smithi116 ceph-mon[32150]: pgmap v665: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:49:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:34 smithi152 ceph-mon[37154]: pgmap v665: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:49:35.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:34 smithi133 ceph-mon[36505]: pgmap v665: 97 pgs: 97 active+clean; 2.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:49:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:36 smithi116 ceph-mon[32150]: pgmap v666: 97 pgs: 97 active+clean; 2.7 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:36 smithi152 ceph-mon[37154]: pgmap v666: 97 pgs: 97 active+clean; 2.7 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:37.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:36 smithi133 ceph-mon[36505]: pgmap v666: 97 pgs: 97 active+clean; 2.7 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:38 smithi116 ceph-mon[32150]: pgmap v667: 97 pgs: 97 active+clean; 2.7 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:38 smithi152 ceph-mon[37154]: pgmap v667: 97 pgs: 97 active+clean; 2.7 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:39.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:38 smithi133 ceph-mon[36505]: pgmap v667: 97 pgs: 97 active+clean; 2.7 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:40 smithi116 ceph-mon[32150]: pgmap v668: 97 pgs: 97 active+clean; 2.7 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:40 smithi152 ceph-mon[37154]: pgmap v668: 97 pgs: 97 active+clean; 2.7 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:41.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:40 smithi133 ceph-mon[36505]: pgmap v668: 97 pgs: 97 active+clean; 2.7 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:49:42.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:42 smithi116 ceph-mon[32150]: pgmap v669: 97 pgs: 97 active+clean; 2.7 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:42.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:42 smithi152 ceph-mon[37154]: pgmap v669: 97 pgs: 97 active+clean; 2.7 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:42.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:42 smithi133 ceph-mon[36505]: pgmap v669: 97 pgs: 97 active+clean; 2.7 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:44 smithi116 ceph-mon[32150]: pgmap v670: 97 pgs: 97 active+clean; 2.7 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:49:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:44 smithi152 ceph-mon[37154]: pgmap v670: 97 pgs: 97 active+clean; 2.7 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:49:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:44 smithi133 ceph-mon[36505]: pgmap v670: 97 pgs: 97 active+clean; 2.7 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:49:45.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 -- 172.21.15.116:0/2138052324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 msgr2=0x7f6944106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 --2- 172.21.15.116:0/2138052324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 0x7f6944106dc0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f692c009860 tx=0x7f692c02bb70 comp rx=0 tx=0).stop 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 -- 172.21.15.116:0/2138052324 shutdown_connections 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 --2- 172.21.15.116:0/2138052324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6944107300 0x7f6944109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 --2- 172.21.15.116:0/2138052324 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 0x7f6944106dc0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 --2- 172.21.15.116:0/2138052324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69441020a0 0x7f6944104490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 -- 172.21.15.116:0/2138052324 >> 172.21.15.116:0/2138052324 conn(0x7f69440fba70 msgr2=0x7f69440fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 -- 172.21.15.116:0/2138052324 shutdown_connections 2024-03-04T07:49:45.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.987+0000 7f6949200640 1 -- 172.21.15.116:0/2138052324 wait complete. 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.988+0000 7f6949200640 1 Processor -- start 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.988+0000 7f6949200640 1 -- start start 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.988+0000 7f6949200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69441020a0 0x7f6944101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6949200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 0x7f6944100050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6949200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6944107300 0x7f6944100590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6949200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694410d290 con 0x7f6944107300 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6949200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694410d110 con 0x7f69441049d0 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6949200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694410d410 con 0x7f69441020a0 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69441020a0 0x7f6944101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69441020a0 0x7f6944101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:37630/0 (socket says 172.21.15.116:37630) 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 -- 172.21.15.116:0/1887731784 learned_addr learned my addr 172.21.15.116:0/1887731784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f693be00640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 0x7f6944100050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6942800640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6944107300 0x7f6944100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 -- 172.21.15.116:0/1887731784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 msgr2=0x7f6944100050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 0x7f6944100050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 -- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6944107300 msgr2=0x7f6944100590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6944107300 0x7f6944100590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.989+0000 7f6941e00640 1 -- 172.21.15.116:0/1887731784 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944100e20 con 0x7f69441020a0 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.990+0000 7f6941e00640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69441020a0 0x7f6944101a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f693400e9c0 tx=0x7f693400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.990+0000 7f693aa00640 1 -- 172.21.15.116:0/1887731784 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693400cde0 con 0x7f69441020a0 2024-03-04T07:49:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.990+0000 7f693aa00640 1 -- 172.21.15.116:0/1887731784 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69340035f0 con 0x7f69441020a0 2024-03-04T07:49:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.990+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69441a19c0 con 0x7f69441020a0 2024-03-04T07:49:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.990+0000 7f693aa00640 1 -- 172.21.15.116:0/1887731784 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934010650 con 0x7f69441020a0 2024-03-04T07:49:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.990+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69441a1ee0 con 0x7f69441020a0 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.993+0000 7f693aa00640 1 -- 172.21.15.116:0/1887731784 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f69340037a0 con 0x7f69441020a0 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.993+0000 7f694bfe1740 1 Processor -- start 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.993+0000 7f694bfe1740 1 -- start start 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d05850 0x561a82d05c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d0ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d0b3b0 0x561a82d0d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f694bfe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a82c6b370 con 0x561a82d06240 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f694bfe1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a82c14550 con 0x561a82d05850 2024-03-04T07:49:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f694bfe1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a82c67410 con 0x561a82d0b3b0 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6942800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d05850 0x561a82d05c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6941e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d0ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f693be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d0b3b0 0x561a82d0d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6942800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d05850 0x561a82d05c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44252/0 (socket says 172.21.15.116:44252) 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6942800640 1 -- 172.21.15.116:0/1742251181 learned_addr learned my addr 172.21.15.116:0/1742251181 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6942800640 1 -- 172.21.15.116:0/1742251181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d0b3b0 msgr2=0x561a82d0d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6942800640 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d0b3b0 0x561a82d0d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6942800640 1 -- 172.21.15.116:0/1742251181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 msgr2=0x561a82d0ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.994+0000 7f6942800640 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d0ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f6942800640 1 -- 172.21.15.116:0/1742251181 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a82c283c0 con 0x561a82d05850 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f6941e00640 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d0ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f6942800640 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d05850 0x561a82d05c70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f693c009fc0 tx=0x7f693c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee4f5812fd04f513 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f693aa00640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69280a1e20 0x7f69280a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f693aa00640 1 -- 172.21.15.116:0/1887731784 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f69340ce230 con 0x7f69441020a0 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f6933400640 1 -- 172.21.15.116:0/1742251181 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c003b70 con 0x561a82d05850 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f6933400640 1 -- 172.21.15.116:0/1742251181 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f693c003d10 con 0x561a82d05850 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f6933400640 1 -- 172.21.15.116:0/1742251181 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c011aa0 con 0x561a82d05850 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.995+0000 7f6941e00640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69280a1e20 0x7f69280a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 -- 172.21.15.116:0/1742251181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d05850 msgr2=0x561a82d05c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f6941e00640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69280a1e20 0x7f69280a42e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f69340978f0 tx=0x7f69340046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d05850 0x561a82d05c70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f693c009fc0 tx=0x7f693c010860 comp rx=0 tx=0).stop 2024-03-04T07:49:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 -- 172.21.15.116:0/1742251181 shutdown_connections 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d0b3b0 0x561a82d0d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d0ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1742251181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d05850 0x561a82d05c70 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 -- 172.21.15.116:0/1742251181 >> 172.21.15.116:0/1742251181 conn(0x561a82d00dc0 msgr2=0x561a82d011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 -- 172.21.15.116:0/1742251181 shutdown_connections 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.996+0000 7f694bfe1740 1 -- 172.21.15.116:0/1742251181 wait complete. 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 -- start start 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 Processor -- start 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d0fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d100c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d17100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a82d0e360 con 0x561a82d06240 2024-03-04T07:49:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a82c279b0 con 0x561a82d0b3b0 2024-03-04T07:49:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a82d0e4e0 con 0x561a82d05850 2024-03-04T07:49:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.997+0000 7f694bfe1740 10 client.0 _ll_register_callbacks cb 0x561a82c63340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49338/0 (socket says 172.21.15.116:49338) 2024-03-04T07:49:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 -- 172.21.15.116:0/3120232559 learned_addr learned my addr 172.21.15.116:0/3120232559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f6942800640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d17100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f6942800640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d0fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 -- 172.21.15.116:0/3120232559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 msgr2=0x561a82d0fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d0fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 -- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 msgr2=0x561a82d17100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d17100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f693be00640 1 -- 172.21.15.116:0/3120232559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a82980210 con 0x561a82d06240 2024-03-04T07:49:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.998+0000 7f6942800640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d17100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f693be00640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d100c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f692c02c180 tx=0x7f692c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74105c3fec263f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f6927e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f6927e00640 1 -- 172.21.15.116:0/3120232559 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c043070 con 0x561a82d06240 2024-03-04T07:49:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f6927e00640 1 -- 172.21.15.116:0/3120232559 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f692c004040 con 0x561a82d06240 2024-03-04T07:49:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f693c003d10 con 0x561a82d06240 2024-03-04T07:49:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f6927e00640 1 -- 172.21.15.116:0/3120232559 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f692c04f050 con 0x561a82d06240 2024-03-04T07:49:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f694bfe1740 10 client.17415 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:45.999+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561a82a4c620 con 0x561a82d06240 2024-03-04T07:49:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f6927e00640 1 -- 172.21.15.116:0/3120232559 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f693c003d10 con 0x561a82d06240 2024-03-04T07:49:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f6927e00640 1 -- 172.21.15.116:0/3120232559 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f692c032640 con 0x561a82d06240 2024-03-04T07:49:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f694bfe1740 10 client.17415 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f694bfe1740 10 client.17415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f694bfe1740 20 client.17415 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d1ba00 0x561a82d1ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f694bfe1740 4 client.17415 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a82c788b0 con 0x561a82d1ba00 2024-03-04T07:49:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.000+0000 7f6942800640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d1ba00 0x561a82d1ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.001+0000 7f6942800640 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d1ba00 0x561a82d1ddf0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec141fb30a0d8709 server_cookie=d2b31e96107101f9 in_seq=0 out_seq=0 2024-03-04T07:49:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.001+0000 7f6927e00640 10 client.17415 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f6927e00640 1 -- 172.21.15.116:0/3120232559 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561a82c788b0 con 0x561a82d1ba00 2024-03-04T07:49:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f6927e00640 10 client.17415 handle_command_reply: tid=0 2024-03-04T07:49:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 client.17415 shutdown 2024-03-04T07:49:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 msgr2=0x561a82d100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d100c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f692c02c180 tx=0x7f692c005d90 comp rx=0 tx=0).stop 2024-03-04T07:49:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 shutdown_connections 2024-03-04T07:49:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d1ba00 0x561a82d1ddf0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d17100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d100c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3120232559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d0fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f6927e00640 0 client.17415 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 >> 172.21.15.116:0/3120232559 conn(0x561a82d00dc0 msgr2=0x561a82cfffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 shutdown_connections 2024-03-04T07:49:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 1 -- 172.21.15.116:0/3120232559 wait complete. 2024-03-04T07:49:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.002+0000 7f694bfe1740 20 client.17415 trim_cache size 0 max 16384 2024-03-04T07:49:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 Processor -- start 2024-03-04T07:49:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 -- start start 2024-03-04T07:49:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d04020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d170c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d04560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441a8bf0 con 0x561a82d06240 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441ab1d0 con 0x561a82d0b3b0 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f694bfe1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441a91f0 con 0x561a82d05850 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f693be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d170c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f6942800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d04020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.018+0000 7f693be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d170c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49350/0 (socket says 172.21.15.116:49350) 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f693be00640 1 -- 172.21.15.116:0/1144600398 learned_addr learned my addr 172.21.15.116:0/1144600398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f6942800640 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d04560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f693be00640 1 -- 172.21.15.116:0/1144600398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 msgr2=0x561a82d04020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f693be00640 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d04020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f693be00640 1 -- 172.21.15.116:0/1144600398 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 msgr2=0x561a82d04560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f693be00640 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d04560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f693be00640 1 -- 172.21.15.116:0/1144600398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a82d1e160 con 0x561a82d06240 2024-03-04T07:49:46.028 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.019+0000 7f693be00640 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d170c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f692c0095f0 tx=0x7f692c037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a5a9713599790b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f6932a00640 1 -- 172.21.15.116:0/1144600398 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c043040 con 0x561a82d06240 2024-03-04T07:49:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f6932a00640 1 -- 172.21.15.116:0/1144600398 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692c032c00 con 0x561a82d06240 2024-03-04T07:49:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f6932a00640 1 -- 172.21.15.116:0/1144600398 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c039040 con 0x561a82d06240 2024-03-04T07:49:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f694bfe1740 1 -- 172.21.15.116:0/1144600398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 msgr2=0x561a82d170c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d170c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f692c0095f0 tx=0x7f692c037000 comp rx=0 tx=0).stop 2024-03-04T07:49:46.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f694bfe1740 1 -- 172.21.15.116:0/1144600398 shutdown_connections 2024-03-04T07:49:46.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d04560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.020+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d170c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.021+0000 7f694bfe1740 1 --2- 172.21.15.116:0/1144600398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d04020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.021+0000 7f694bfe1740 1 -- 172.21.15.116:0/1144600398 >> 172.21.15.116:0/1144600398 conn(0x561a82d00dc0 msgr2=0x561a82d07560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:46.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.021+0000 7f694bfe1740 1 -- 172.21.15.116:0/1144600398 shutdown_connections 2024-03-04T07:49:46.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.021+0000 7f694bfe1740 1 -- 172.21.15.116:0/1144600398 wait complete. 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.021+0000 7f694bfe1740 1 -- start start 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.021+0000 7f694bfe1740 1 Processor -- start 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d146b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d14bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f694bfe1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d15bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f694bfe1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441a9850 con 0x561a82d06240 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f694bfe1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441aab70 con 0x561a82d0b3b0 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f694bfe1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441a9eb0 con 0x561a82d05850 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f694bfe1740 10 client.0 _ll_register_callbacks cb 0x7f693c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:46.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d14bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d14bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49352/0 (socket says 172.21.15.116:49352) 2024-03-04T07:49:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 -- 172.21.15.116:0/3555496352 learned_addr learned my addr 172.21.15.116:0/3555496352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f6942800640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d146b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f6942800640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d15bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 -- 172.21.15.116:0/3555496352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 msgr2=0x561a82d146b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d146b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 -- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 msgr2=0x561a82d15bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d15bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f693be00640 1 -- 172.21.15.116:0/3555496352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a82d161a0 con 0x561a82d06240 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.022+0000 7f6942800640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d146b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f693be00640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d14bf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f692c000c00 tx=0x7f692c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4397274062c8646d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f6927e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f6927e00640 1 -- 172.21.15.116:0/3555496352 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c039070 con 0x561a82d06240 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f692c032c00 con 0x561a82d06240 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f6927e00640 1 -- 172.21.15.116:0/3555496352 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f692c005220 con 0x561a82d06240 2024-03-04T07:49:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f6927e00640 1 -- 172.21.15.116:0/3555496352 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f692c043040 con 0x561a82d06240 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f694bfe1740 10 client.17421 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x561a82d174b0 con 0x561a82d06240 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.023+0000 7f6927e00640 1 -- 172.21.15.116:0/3555496352 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692c032c00 con 0x561a82d06240 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f6927e00640 1 -- 172.21.15.116:0/3555496352 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f692c0046c0 con 0x561a82d06240 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f694bfe1740 10 client.17421 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f694bfe1740 10 client.17421 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f694bfe1740 20 client.17421 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d779c0 0x561a82d79de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f694bfe1740 4 client.17421 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561a82c72c00 con 0x561a82d779c0 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f6942800640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d779c0 0x561a82d79de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f6927e00640 10 client.17421 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:46.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.024+0000 7f6942800640 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d779c0 0x561a82d79de0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab2c35c71a4f1d4b server_cookie=5266c4f1687ddd6e in_seq=0 out_seq=0 2024-03-04T07:49:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.039+0000 7f6927e00640 1 -- 172.21.15.116:0/3555496352 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561a82c72c00 con 0x561a82d779c0 2024-03-04T07:49:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.039+0000 7f6927e00640 10 client.17421 handle_command_reply: tid=0 2024-03-04T07:49:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.039+0000 7f694bfe1740 1 client.17421 shutdown 2024-03-04T07:49:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.039+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 msgr2=0x561a82d14bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.039+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d14bf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f692c000c00 tx=0x7f692c005f10 comp rx=0 tx=0).stop 2024-03-04T07:49:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 shutdown_connections 2024-03-04T07:49:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x561a82d779c0 0x561a82d79de0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x561a82d0b3b0 0x561a82d15bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a82d06240 0x561a82d14bf0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 --2- 172.21.15.116:0/3555496352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a82d05850 0x561a82d146b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f6927e00640 0 client.17421 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 >> 172.21.15.116:0/3555496352 conn(0x561a82d00dc0 msgr2=0x561a82d04020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 shutdown_connections 2024-03-04T07:49:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 1 -- 172.21.15.116:0/3555496352 wait complete. 2024-03-04T07:49:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.040+0000 7f694bfe1740 20 client.17421 trim_cache size 0 max 16384 2024-03-04T07:49:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:49:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:49:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:49:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:49:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69280a1e20 msgr2=0x7f69280a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69280a1e20 0x7f69280a42e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f69340978f0 tx=0x7f69340046c0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69441020a0 msgr2=0x7f6944101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69441020a0 0x7f6944101a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f693400e9c0 tx=0x7f693400ee80 comp rx=0 tx=0).stop 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 shutdown_connections 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6944107300 0x7f6944100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69441049d0 0x7f6944100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69280a1e20 0x7f69280a42e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 --2- 172.21.15.116:0/1887731784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69441020a0 0x7f6944101a10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 >> 172.21.15.116:0/1887731784 conn(0x7f69440fba70 msgr2=0x7f69441029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 shutdown_connections 2024-03-04T07:49:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:46.041+0000 7f6949200640 1 -- 172.21.15.116:0/1887731784 wait complete. 2024-03-04T07:49:46.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:45 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:49:46.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:45 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:49:46.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:45 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:49:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:46 smithi116 ceph-mon[32150]: pgmap v671: 97 pgs: 97 active+clean; 2.8 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:46 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:49:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:46 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:49:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:46 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:49:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:46 smithi152 ceph-mon[37154]: pgmap v671: 97 pgs: 97 active+clean; 2.8 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:46 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:49:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:46 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:49:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:46 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:49:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:46 smithi133 ceph-mon[36505]: pgmap v671: 97 pgs: 97 active+clean; 2.8 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:46 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:49:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:46 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:49:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:46 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:49:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:48 smithi116 ceph-mon[32150]: pgmap v672: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:49:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:48 smithi152 ceph-mon[37154]: pgmap v672: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:49:49.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:48 smithi133 ceph-mon[36505]: pgmap v672: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:49:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:50 smithi116 ceph-mon[32150]: pgmap v673: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:49:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:50 smithi152 ceph-mon[37154]: pgmap v673: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:49:51.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:50 smithi133 ceph-mon[36505]: pgmap v673: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:49:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:52 smithi116 ceph-mon[32150]: pgmap v674: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:52 smithi152 ceph-mon[37154]: pgmap v674: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:52 smithi133 ceph-mon[36505]: pgmap v674: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:49:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:54 smithi116 ceph-mon[32150]: pgmap v675: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:54 smithi152 ceph-mon[37154]: pgmap v675: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:55.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:49:55.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:49:55.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:54 smithi133 ceph-mon[36505]: pgmap v675: 97 pgs: 97 active+clean; 2.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.535+0000 7f706fe00640 1 -- 172.21.15.116:0/1493906383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701026a0 msgr2=0x7f7070102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.535+0000 7f706fe00640 1 --2- 172.21.15.116:0/1493906383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701026a0 0x7f7070102b20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f705c009860 tx=0x7f705c02bb70 comp rx=0 tx=0).stop 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.536+0000 7f706fe00640 1 -- 172.21.15.116:0/1493906383 shutdown_connections 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.536+0000 7f706fe00640 1 --2- 172.21.15.116:0/1493906383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070103060 0x7f70701098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.536+0000 7f706fe00640 1 --2- 172.21.15.116:0/1493906383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701026a0 0x7f7070102b20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.536+0000 7f706fe00640 1 --2- 172.21.15.116:0/1493906383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f70701014b0 0x7f70701018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.536+0000 7f706fe00640 1 -- 172.21.15.116:0/1493906383 >> 172.21.15.116:0/1493906383 conn(0x7f70700fca40 msgr2=0x7f70700fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.536+0000 7f706fe00640 1 -- 172.21.15.116:0/1493906383 shutdown_connections 2024-03-04T07:49:55.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.536+0000 7f706fe00640 1 -- 172.21.15.116:0/1493906383 wait complete. 2024-03-04T07:49:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.537+0000 7f706fe00640 1 Processor -- start 2024-03-04T07:49:55.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.537+0000 7f706fe00640 1 -- start start 2024-03-04T07:49:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701014b0 0x7f70701999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70701026a0 0x7f7070199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070103060 0x7f70701a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010d140 con 0x7f70701026a0 2024-03-04T07:49:55.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706fe00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010cfc0 con 0x7f7070103060 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706fe00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010d2c0 con 0x7f70701014b0 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070103060 0x7f70701a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706ea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701014b0 0x7f70701999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70701026a0 0x7f7070199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706ea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701014b0 0x7f70701999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:37692/0 (socket says 172.21.15.116:37692) 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706ea00640 1 -- 172.21.15.116:0/2743318458 learned_addr learned my addr 172.21.15.116:0/2743318458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706f400640 1 -- 172.21.15.116:0/2743318458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701014b0 msgr2=0x7f70701999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706f400640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701014b0 0x7f70701999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706f400640 1 -- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70701026a0 msgr2=0x7f7070199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706f400640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70701026a0 0x7f7070199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706f400640 1 -- 172.21.15.116:0/2743318458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70701a15e0 con 0x7f7070103060 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.538+0000 7f706ea00640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701014b0 0x7f70701999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.539+0000 7f706e000640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70701026a0 0x7f7070199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:55.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.539+0000 7f706f400640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070103060 0x7f70701a0f70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f706800efc0 tx=0x7f706800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.539+0000 7f7067400640 1 -- 172.21.15.116:0/2743318458 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068003970 con 0x7f7070103060 2024-03-04T07:49:55.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.539+0000 7f7067400640 1 -- 172.21.15.116:0/2743318458 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7068004020 con 0x7f7070103060 2024-03-04T07:49:55.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.539+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70701a18d0 con 0x7f7070103060 2024-03-04T07:49:55.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.539+0000 7f7067400640 1 -- 172.21.15.116:0/2743318458 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800b9f0 con 0x7f7070103060 2024-03-04T07:49:55.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.539+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70701a1da0 con 0x7f7070103060 2024-03-04T07:49:55.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.543+0000 7f707865a740 1 Processor -- start 2024-03-04T07:49:55.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.543+0000 7f7067400640 1 -- 172.21.15.116:0/2743318458 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f706801d020 con 0x7f7070103060 2024-03-04T07:49:55.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f707865a740 1 -- start start 2024-03-04T07:49:55.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f707865a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137411d10 0x55b137412130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f707865a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137412700 0x55b137417200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f707865a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137417870 0x55b137419d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f707865a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b13733f090 con 0x55b137417870 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f707865a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b137376d30 con 0x55b137411d10 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f707865a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b13731fa30 con 0x55b137412700 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f706e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137411d10 0x55b137412130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f706ea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137412700 0x55b137417200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f706e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137411d10 0x55b137412130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44304/0 (socket says 172.21.15.116:44304) 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.544+0000 7f706e000640 1 -- 172.21.15.116:0/3619384884 learned_addr learned my addr 172.21.15.116:0/3619384884 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f706e000640 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137417870 0x55b137419d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f706ea00640 1 -- 172.21.15.116:0/3619384884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137411d10 msgr2=0x55b137412130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f706ea00640 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137411d10 0x55b137412130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f706ea00640 1 -- 172.21.15.116:0/3619384884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137417870 msgr2=0x55b137419d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f706ea00640 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137417870 0x55b137419d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f706ea00640 1 -- 172.21.15.116:0/3619384884 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1371557d0 con 0x55b137412700 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f706ea00640 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137412700 0x55b137417200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f70600114c0 tx=0x7f7060011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69afb8737831273 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f7067400640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f70540a1d80 0x7f70540a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f705be00640 1 -- 172.21.15.116:0/3619384884 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060011250 con 0x55b137412700 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f705be00640 1 -- 172.21.15.116:0/3619384884 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7060005500 con 0x55b137412700 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f705be00640 1 -- 172.21.15.116:0/3619384884 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000f820 con 0x55b137412700 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.545+0000 7f7067400640 1 -- 172.21.15.116:0/2743318458 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f70680cdc80 con 0x7f7070103060 2024-03-04T07:49:55.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f706e000640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f70540a1d80 0x7f70540a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f707865a740 1 -- 172.21.15.116:0/3619384884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137412700 msgr2=0x55b137417200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f707865a740 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137412700 0x55b137417200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f70600114c0 tx=0x7f7060011980 comp rx=0 tx=0).stop 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f706e000640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f70540a1d80 0x7f70540a4240 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f705c02c180 tx=0x7f705c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f707865a740 1 -- 172.21.15.116:0/3619384884 shutdown_connections 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f707865a740 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137417870 0x55b137419d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f707865a740 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137412700 0x55b137417200 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f707865a740 1 --2- 172.21.15.116:0/3619384884 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137411d10 0x55b137412130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.546+0000 7f707865a740 1 -- 172.21.15.116:0/3619384884 >> 172.21.15.116:0/3619384884 conn(0x55b13740d280 msgr2=0x55b13740d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.547+0000 7f707865a740 1 -- 172.21.15.116:0/3619384884 shutdown_connections 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.547+0000 7f707865a740 1 -- 172.21.15.116:0/3619384884 wait complete. 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.547+0000 7f707865a740 1 -- start start 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.547+0000 7f707865a740 1 Processor -- start 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f707865a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b13741c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f707865a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b13741c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f707865a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b1374235f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f707865a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1373150c0 con 0x55b137411d10 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f707865a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b137372dd0 con 0x55b137412700 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f707865a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b13741a820 con 0x55b137417870 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f707865a740 10 client.0 _ll_register_callbacks cb 0x55b137332860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b13741c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b13741c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b13741c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49386/0 (socket says 172.21.15.116:49386) 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b13741c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44314/0 (socket says 172.21.15.116:44314) 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706ea00640 1 -- 172.21.15.116:0/2862526570 learned_addr learned my addr 172.21.15.116:0/2862526570 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706f400640 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b1374235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706e000640 1 -- 172.21.15.116:0/2862526570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 msgr2=0x55b1374235f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706e000640 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b1374235f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706e000640 1 -- 172.21.15.116:0/2862526570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 msgr2=0x55b13741c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706e000640 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b13741c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.548+0000 7f706e000640 1 -- 172.21.15.116:0/2862526570 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b13708b3f0 con 0x55b137412700 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f706e000640 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b13741c620 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c00 tx=0x7f705c02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=830a6e3db76fa83c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f7058c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f7058c00640 1 -- 172.21.15.116:0/2862526570 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c055060 con 0x55b137412700 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f7058c00640 1 -- 172.21.15.116:0/2862526570 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f705c032840 con 0x55b137412700 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7060005500 con 0x55b137412700 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f7058c00640 1 -- 172.21.15.116:0/2862526570 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f705c039040 con 0x55b137412700 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f707865a740 10 client.27566 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.549+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b13704b0f0 con 0x55b137412700 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f7058c00640 1 -- 172.21.15.116:0/2862526570 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7060005500 con 0x55b137412700 2024-03-04T07:49:55.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f7058c00640 1 -- 172.21.15.116:0/2862526570 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f705c05a070 con 0x55b137412700 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f707865a740 10 client.27566 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f707865a740 10 client.27566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f707865a740 20 client.27566 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f707865a740 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b137427b80 0x55b137429f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f707865a740 4 client.27566 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b137384270 con 0x55b137427b80 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.550+0000 7f706ea00640 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b137427b80 0x55b137429f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.551+0000 7f706ea00640 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b137427b80 0x55b137429f90 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1be5266ed2e8475a server_cookie=3dc45e505707376d in_seq=0 out_seq=0 2024-03-04T07:49:55.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.551+0000 7f7058c00640 10 client.27566 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:55.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.551+0000 7f7058c00640 1 -- 172.21.15.116:0/2862526570 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b137384270 con 0x55b137427b80 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.551+0000 7f7058c00640 10 client.27566 handle_command_reply: tid=0 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 client.27566 shutdown 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 msgr2=0x55b13741c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b13741c620 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c00 tx=0x7f705c02bb70 comp rx=0 tx=0).stop 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 shutdown_connections 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b137427b80 0x55b137429f90 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b1374235f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b13741c620 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f7058c00640 0 client.27566 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 --2- 172.21.15.116:0/2862526570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b13741c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 >> 172.21.15.116:0/2862526570 conn(0x55b13740d280 msgr2=0x55b13740c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 shutdown_connections 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 1 -- 172.21.15.116:0/2862526570 wait complete. 2024-03-04T07:49:55.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.552+0000 7f707865a740 20 client.27566 trim_cache size 0 max 16384 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 Processor -- start 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 -- start start 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137411d10 0x55b137410bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137412700 0x55b1374235b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137417870 0x55b13742e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a8aa0 con 0x55b137412700 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701ab080 con 0x55b137417870 2024-03-04T07:49:55.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f707865a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a90a0 con 0x55b137411d10 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f706f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137417870 0x55b13742e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f706f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137417870 0x55b13742e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44326/0 (socket says 172.21.15.116:44326) 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f706f400640 1 -- 172.21.15.116:0/1400799205 learned_addr learned my addr 172.21.15.116:0/1400799205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f706ea00640 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137411d10 0x55b137410bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f706e000640 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137412700 0x55b1374235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f706f400640 1 -- 172.21.15.116:0/1400799205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137411d10 msgr2=0x55b137410bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.570+0000 7f706f400640 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137411d10 0x55b137410bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f706f400640 1 -- 172.21.15.116:0/1400799205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137412700 msgr2=0x55b1374235b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f706f400640 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137412700 0x55b1374235b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f706f400640 1 -- 172.21.15.116:0/1400799205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1374175c0 con 0x55b137417870 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f706e000640 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137412700 0x55b1374235b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:55.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f706f400640 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137417870 0x55b13742e850 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f70680cb740 tx=0x7f7068096570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=185d4c0f48e445a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f705b400640 1 -- 172.21.15.116:0/1400799205 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680a3110 con 0x55b137417870 2024-03-04T07:49:55.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f705b400640 1 -- 172.21.15.116:0/1400799205 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70680b3070 con 0x55b137417870 2024-03-04T07:49:55.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.571+0000 7f705b400640 1 -- 172.21.15.116:0/1400799205 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680b3220 con 0x55b137417870 2024-03-04T07:49:55.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 -- 172.21.15.116:0/1400799205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137417870 msgr2=0x55b13742e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137417870 0x55b13742e850 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f70680cb740 tx=0x7f7068096570 comp rx=0 tx=0).stop 2024-03-04T07:49:55.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 -- 172.21.15.116:0/1400799205 shutdown_connections 2024-03-04T07:49:55.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137417870 0x55b13742e850 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137412700 0x55b1374235b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 --2- 172.21.15.116:0/1400799205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137411d10 0x55b137410bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 -- 172.21.15.116:0/1400799205 >> 172.21.15.116:0/1400799205 conn(0x55b13740d280 msgr2=0x55b1374104e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 -- 172.21.15.116:0/1400799205 shutdown_connections 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 -- 172.21.15.116:0/1400799205 wait complete. 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 -- start start 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.572+0000 7f707865a740 1 Processor -- start 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f707865a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b137421490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f707865a740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b1374219d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f707865a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b137422990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f707865a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a9700 con 0x55b137411d10 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f707865a740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701aaa20 con 0x55b137412700 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f707865a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701a9d60 con 0x55b137417870 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f707865a740 10 client.0 _ll_register_callbacks cb 0x7f705c004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b137421490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b137421490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49408/0 (socket says 172.21.15.116:49408) 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 -- 172.21.15.116:0/2390924021 learned_addr learned my addr 172.21.15.116:0/2390924021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706e000640 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b1374219d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706f400640 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b137422990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 -- 172.21.15.116:0/2390924021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 msgr2=0x55b137422990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b137422990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 -- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 msgr2=0x55b1374219d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b1374219d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.573+0000 7f706ea00640 1 -- 172.21.15.116:0/2390924021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b137422f80 con 0x55b137411d10 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f706ea00640 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b137421490 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7060009380 tx=0x7f7060005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d0fcd001ca3853 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f7058c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f7058c00640 1 -- 172.21.15.116:0/2390924021 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706001d040 con 0x55b137411d10 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f7058c00640 1 -- 172.21.15.116:0/2390924021 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7060004040 con 0x55b137411d10 2024-03-04T07:49:55.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70680b3070 con 0x55b137411d10 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f7058c00640 1 -- 172.21.15.116:0/2390924021 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f706003a050 con 0x55b137411d10 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f707865a740 10 client.17439 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.574+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b13748c380 con 0x55b137411d10 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f7058c00640 1 -- 172.21.15.116:0/2390924021 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70680b3070 con 0x55b137411d10 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f7058c00640 1 -- 172.21.15.116:0/2390924021 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f70600047b0 con 0x55b137411d10 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f707865a740 10 client.17439 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f707865a740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f707865a740 20 client.17439 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f707865a740 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b13748cbe0 0x55b13748f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f707865a740 4 client.17439 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b13737e5c0 con 0x55b13748cbe0 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.575+0000 7f706e000640 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b13748cbe0 0x55b13748f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f706e000640 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b13748cbe0 0x55b13748f000 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9040c95959c55fe2 server_cookie=b5c11df4dc29cf72 in_seq=0 out_seq=0 2024-03-04T07:49:55.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f7058c00640 10 client.17439 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f7058c00640 1 -- 172.21.15.116:0/2390924021 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b13737e5c0 con 0x55b13748cbe0 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f7058c00640 10 client.17439 handle_command_reply: tid=0 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 client.17439 shutdown 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 msgr2=0x55b137421490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b137421490 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7060009380 tx=0x7f7060005fb0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 shutdown_connections 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b13748cbe0 0x55b13748f000 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b137417870 0x55b137422990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b137412700 0x55b1374219d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 --2- 172.21.15.116:0/2390924021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b137411d10 0x55b137421490 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 >> 172.21.15.116:0/2390924021 conn(0x55b13740d280 msgr2=0x55b137410d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f7058c00640 0 client.17439 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 shutdown_connections 2024-03-04T07:49:55.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 1 -- 172.21.15.116:0/2390924021 wait complete. 2024-03-04T07:49:55.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.576+0000 7f707865a740 20 client.17439 trim_cache size 0 max 16384 2024-03-04T07:49:55.583 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:49:55.583 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:49:55.583 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:49:55.583 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:49:55.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f70540a1d80 msgr2=0x7f70540a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f70540a1d80 0x7f70540a4240 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f705c02c180 tx=0x7f705c02e040 comp rx=0 tx=0).stop 2024-03-04T07:49:55.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070103060 msgr2=0x7f70701a0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070103060 0x7f70701a0f70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f706800efc0 tx=0x7f706800c800 comp rx=0 tx=0).stop 2024-03-04T07:49:55.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 shutdown_connections 2024-03-04T07:49:55.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070103060 0x7f70701a0f70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70701026a0 0x7f7070199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f70540a1d80 0x7f70540a4240 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 --2- 172.21.15.116:0/2743318458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70701014b0 0x7f70701999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 >> 172.21.15.116:0/2743318458 conn(0x7f70700fca40 msgr2=0x7f7070073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 shutdown_connections 2024-03-04T07:49:55.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.577+0000 7f706fe00640 1 -- 172.21.15.116:0/2743318458 wait complete. 2024-03-04T07:49:55.596 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:49:55.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:49:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 -- 172.21.15.116:0/2509410555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 msgr2=0x7f96e8106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 --2- 172.21.15.116:0/2509410555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 0x7f96e8106e00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009900 tx=0x7f96d402bc40 comp rx=0 tx=0).stop 2024-03-04T07:49:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 -- 172.21.15.116:0/2509410555 shutdown_connections 2024-03-04T07:49:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 --2- 172.21.15.116:0/2509410555 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e8107340 0x7f96e8109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 --2- 172.21.15.116:0/2509410555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 0x7f96e8106e00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 --2- 172.21.15.116:0/2509410555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96e81020e0 0x7f96e81044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 -- 172.21.15.116:0/2509410555 >> 172.21.15.116:0/2509410555 conn(0x7f96e80fba70 msgr2=0x7f96e80fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:55.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 -- 172.21.15.116:0/2509410555 shutdown_connections 2024-03-04T07:49:55.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.807+0000 7f96ede00640 1 -- 172.21.15.116:0/2509410555 wait complete. 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 Processor -- start 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 -- start start 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e81020e0 0x7f96e8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 0x7f96e819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96e8107340 0x7f96e81a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e810d260 con 0x7f96e8104a10 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e810d0e0 con 0x7f96e81020e0 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96ede00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e810d3e0 con 0x7f96e8107340 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96e6000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 0x7f96e819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.808+0000 7f96e6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e81020e0 0x7f96e8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e81020e0 0x7f96e8199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:44350/0 (socket says 172.21.15.116:44350) 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 -- 172.21.15.116:0/1425019010 learned_addr learned my addr 172.21.15.116:0/1425019010 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e7400640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96e8107340 0x7f96e81a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 -- 172.21.15.116:0/1425019010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96e8107340 msgr2=0x7f96e81a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96e8107340 0x7f96e81a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 -- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 msgr2=0x7f96e819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 0x7f96e819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 -- 172.21.15.116:0/1425019010 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e81a17a0 con 0x7f96e81020e0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96e6a00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e81020e0 0x7f96e8199b80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f96d800d950 tx=0x7f96d800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96eca00640 1 -- 172.21.15.116:0/1425019010 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8017070 con 0x7f96e81020e0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96eca00640 1 -- 172.21.15.116:0/1425019010 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96d8009c30 con 0x7f96e81020e0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96eca00640 1 -- 172.21.15.116:0/1425019010 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d800f6b0 con 0x7f96e81020e0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.809+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e81a1a90 con 0x7f96e81020e0 2024-03-04T07:49:55.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.810+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e81a1f60 con 0x7f96e81020e0 2024-03-04T07:49:55.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.811+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e8108e30 con 0x7f96e81020e0 2024-03-04T07:49:55.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.812+0000 7f96eca00640 1 -- 172.21.15.116:0/1425019010 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f96d80109d0 con 0x7f96e81020e0 2024-03-04T07:49:55.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.814+0000 7f96eca00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96cc0a1e00 0x7f96cc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:55.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.814+0000 7f96eca00640 1 -- 172.21.15.116:0/1425019010 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f96d80cd9c0 con 0x7f96e81020e0 2024-03-04T07:49:55.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.814+0000 7f96e6000640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96cc0a1e00 0x7f96cc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:55.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.815+0000 7f96e6000640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96cc0a1e00 0x7f96cc0a42c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f96d4000c00 tx=0x7f96d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:55.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:55.818+0000 7f96eca00640 1 -- 172.21.15.116:0/1425019010 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f96d8009de0 con 0x7f96e81020e0 2024-03-04T07:49:56.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.057+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96e802cff0 con 0x7f96e81020e0 2024-03-04T07:49:56.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.059+0000 7f96eca00640 1 -- 172.21.15.116:0/1425019010 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f96d80b8e20 con 0x7f96e81020e0 2024-03-04T07:49:56.059 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:49:56.059 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:49:56.060 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96cc0a1e00 msgr2=0x7f96cc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96cc0a1e00 0x7f96cc0a42c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f96d4000c00 tx=0x7f96d4005fd0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e81020e0 msgr2=0x7f96e8199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e81020e0 0x7f96e8199b80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f96d800d950 tx=0x7f96d800de10 comp rx=0 tx=0).stop 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 shutdown_connections 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96cc0a1e00 0x7f96cc0a42c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96e8107340 0x7f96e81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e8104a10 0x7f96e819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 --2- 172.21.15.116:0/1425019010 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96e81020e0 0x7f96e8199b80 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 >> 172.21.15.116:0/1425019010 conn(0x7f96e80fba70 msgr2=0x7f96e8101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 shutdown_connections 2024-03-04T07:49:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.061+0000 7f96ede00640 1 -- 172.21.15.116:0/1425019010 wait complete. 2024-03-04T07:49:56.082 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:49:56.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 -- 172.21.15.116:0/4181657169 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed481097a0 msgr2=0x7fed48110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 --2- 172.21.15.116:0/4181657169 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed481097a0 0x7fed48110030 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fed40009be0 tx=0x7fed4002bef0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 -- 172.21.15.116:0/4181657169 shutdown_connections 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 --2- 172.21.15.116:0/4181657169 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed481097a0 0x7fed48110030 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 --2- 172.21.15.116:0/4181657169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed48108de0 0x7fed48109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 --2- 172.21.15.116:0/4181657169 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed48107bf0 0x7fed48108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 -- 172.21.15.116:0/4181657169 >> 172.21.15.116:0/4181657169 conn(0x7fed48076b90 msgr2=0x7fed48078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 -- 172.21.15.116:0/4181657169 shutdown_connections 2024-03-04T07:49:56.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 -- 172.21.15.116:0/4181657169 wait complete. 2024-03-04T07:49:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.296+0000 7fed4e400640 1 Processor -- start 2024-03-04T07:49:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed4e400640 1 -- start start 2024-03-04T07:49:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed4e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed48107bf0 0x7fed4819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed4e400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed48108de0 0x7fed4819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed4e400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed481097a0 0x7fed481a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed4e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481138a0 con 0x7fed48107bf0 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed4e400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48113720 con 0x7fed481097a0 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed4e400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed48113a20 con 0x7fed48108de0 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed48107bf0 0x7fed4819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed48107bf0 0x7fed4819de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46282/0 (socket says 172.21.15.116:46282) 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47400640 1 -- 172.21.15.116:0/3439954160 learned_addr learned my addr 172.21.15.116:0/3439954160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47e00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed481097a0 0x7fed481a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed46a00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed48108de0 0x7fed4819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47e00640 1 -- 172.21.15.116:0/3439954160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed48108de0 msgr2=0x7fed4819e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47e00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed48108de0 0x7fed4819e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47e00640 1 -- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed48107bf0 msgr2=0x7fed4819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47e00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed48107bf0 0x7fed4819de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.297+0000 7fed47e00640 1 -- 172.21.15.116:0/3439954160 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed481a5aa0 con 0x7fed481097a0 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.298+0000 7fed47e00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed481097a0 0x7fed481a5430 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fed4002c430 tx=0x7fed400337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.298+0000 7fed3fe00640 1 -- 172.21.15.116:0/3439954160 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40043020 con 0x7fed481097a0 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.298+0000 7fed3fe00640 1 -- 172.21.15.116:0/3439954160 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed4002ca60 con 0x7fed481097a0 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.298+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed481a5d30 con 0x7fed481097a0 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.298+0000 7fed3fe00640 1 -- 172.21.15.116:0/3439954160 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40034690 con 0x7fed481097a0 2024-03-04T07:49:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.298+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed481a6270 con 0x7fed481097a0 2024-03-04T07:49:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.301+0000 7fed3fe00640 1 -- 172.21.15.116:0/3439954160 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fed40046020 con 0x7fed481097a0 2024-03-04T07:49:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.301+0000 7fed51222740 1 Processor -- start 2024-03-04T07:49:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.301+0000 7fed51222740 1 -- start start 2024-03-04T07:49:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed51222740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed51222740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8bfc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed51222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8bfec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed51222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8a8b5bd60 con 0x55a8a8bfc800 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed51222740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8a8b04cf0 con 0x55a8a8bf7690 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed51222740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8a8b57e00 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed47400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8bfc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed46a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed47400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8bfc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40944/0 (socket says 172.21.15.116:40944) 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed47400640 1 -- 172.21.15.116:0/2850721143 learned_addr learned my addr 172.21.15.116:0/2850721143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed46a00640 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8bfec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.302+0000 7fed47400640 1 -- 172.21.15.116:0/2850721143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 msgr2=0x55a8a8bf70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed47400640 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed47400640 1 -- 172.21.15.116:0/2850721143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 msgr2=0x55a8a8bfec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed47400640 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8bfec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed47400640 1 -- 172.21.15.116:0/2850721143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8a88703f0 con 0x55a8a8bf7690 2024-03-04T07:49:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed46a00640 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8bfec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed47400640 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8bfc190 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fed38009d60 tx=0x7fed38011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3086954de54ec79f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed3fe00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed2c0a1e00 0x7fed2c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed3cc00640 1 -- 172.21.15.116:0/2850721143 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38004230 con 0x55a8a8bf7690 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed3fe00640 1 -- 172.21.15.116:0/3439954160 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed400f0270 con 0x7fed481097a0 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed46a00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed2c0a1e00 0x7fed2c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed3cc00640 1 -- 172.21.15.116:0/2850721143 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed380043d0 con 0x55a8a8bf7690 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.303+0000 7fed3cc00640 1 -- 172.21.15.116:0/2850721143 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38019840 con 0x55a8a8bf7690 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 -- 172.21.15.116:0/2850721143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 msgr2=0x55a8a8bfc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed46a00640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed2c0a1e00 0x7fed2c0a42c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55a8a8bf7ef0 tx=0x7fed34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8bfc190 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fed38009d60 tx=0x7fed38011810 comp rx=0 tx=0).stop 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 -- 172.21.15.116:0/2850721143 shutdown_connections 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8bfec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8bfc190 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 --2- 172.21.15.116:0/2850721143 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 -- 172.21.15.116:0/2850721143 >> 172.21.15.116:0/2850721143 conn(0x55a8a8bf2210 msgr2=0x55a8a8bf2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 -- 172.21.15.116:0/2850721143 shutdown_connections 2024-03-04T07:49:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 -- 172.21.15.116:0/2850721143 wait complete. 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.304+0000 7fed51222740 1 -- start start 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 1 Processor -- start 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c01190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8c016d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8c08710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8a8bff7b0 con 0x55a8a8bfc800 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8a8afa220 con 0x55a8a8bf7690 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8a8bff930 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed51222740 10 client.0 _ll_register_callbacks cb 0x55a8a886fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed46a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8c016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed47e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8c08710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed46a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8c016d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40946/0 (socket says 172.21.15.116:40946) 2024-03-04T07:49:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed46a00640 1 -- 172.21.15.116:0/405133899 learned_addr learned my addr 172.21.15.116:0/405133899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.305+0000 7fed47400640 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c01190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed46a00640 1 -- 172.21.15.116:0/405133899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 msgr2=0x55a8a8c01190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed46a00640 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c01190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed46a00640 1 -- 172.21.15.116:0/405133899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 msgr2=0x55a8a8c08710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed46a00640 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8c08710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed46a00640 1 -- 172.21.15.116:0/405133899 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8a893a7d0 con 0x55a8a8bf7690 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed46a00640 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8c016d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fed34009fd0 tx=0x7fed340097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d18d0ef1c1859d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed31600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed31600640 1 -- 172.21.15.116:0/405133899 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3401a070 con 0x55a8a8bf7690 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed31600640 1 -- 172.21.15.116:0/405133899 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed3401fd80 con 0x55a8a8bf7690 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed380043d0 con 0x55a8a8bf7690 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed31600640 1 -- 172.21.15.116:0/405133899 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed3402b480 con 0x55a8a8bf7690 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed51222740 10 client.27590 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.306+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a8a8c0baf0 con 0x55a8a8bf7690 2024-03-04T07:49:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed31600640 1 -- 172.21.15.116:0/405133899 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed380043d0 con 0x55a8a8bf7690 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed31600640 1 -- 172.21.15.116:0/405133899 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fed34020750 con 0x55a8a8bf7690 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed51222740 10 client.27590 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed51222740 10 client.27590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed51222740 20 client.27590 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed51222740 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c0d210 0x55a8a8c0f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed51222740 4 client.27590 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8a8b692a0 con 0x55a8a8c0d210 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.307+0000 7fed47400640 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c0d210 0x55a8a8c0f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.308+0000 7fed47400640 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c0d210 0x55a8a8c0f690 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce8935dc6a3ccb95 server_cookie=672a5a11fe56f145 in_seq=0 out_seq=0 2024-03-04T07:49:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.308+0000 7fed31600640 10 client.27590 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:56.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.308+0000 7fed31600640 1 -- 172.21.15.116:0/405133899 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a8a8b692a0 con 0x55a8a8c0d210 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.308+0000 7fed31600640 10 client.27590 handle_command_reply: tid=0 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 client.27590 shutdown 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 msgr2=0x55a8a8c016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8c016d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fed34009fd0 tx=0x7fed340097b0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 shutdown_connections 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c0d210 0x55a8a8c0f690 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bfc800 0x55a8a8c08710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bf7690 0x55a8a8c016d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed31600640 0 client.27590 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 --2- 172.21.15.116:0/405133899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c01190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 >> 172.21.15.116:0/405133899 conn(0x55a8a8bf2210 msgr2=0x55a8a8bf14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 shutdown_connections 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 1 -- 172.21.15.116:0/405133899 wait complete. 2024-03-04T07:49:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.309+0000 7fed51222740 20 client.27590 trim_cache size 0 max 16384 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.322+0000 7fed51222740 1 Processor -- start 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.322+0000 7fed51222740 1 -- start start 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed51222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed51222740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf7690 0x55a8a8c086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed51222740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c13f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed51222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481ad090 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed51222740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481af670 con 0x55a8a8bfc800 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed51222740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481ad690 con 0x55a8a8bf7690 2024-03-04T07:49:56.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46308/0 (socket says 172.21.15.116:46308) 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 -- 172.21.15.116:0/1770768662 learned_addr learned my addr 172.21.15.116:0/1770768662 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47e00640 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c13f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed46a00640 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf7690 0x55a8a8c086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 -- 172.21.15.116:0/1770768662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf7690 msgr2=0x55a8a8c086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf7690 0x55a8a8c086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 -- 172.21.15.116:0/1770768662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 msgr2=0x55a8a8c13f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c13f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.323+0000 7fed47400640 1 -- 172.21.15.116:0/1770768662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8a8bedc80 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.324+0000 7fed47e00640 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c13f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.324+0000 7fed47400640 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf5b80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fed38009250 tx=0x7fed38009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d963264493619ae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.324+0000 7fed33e00640 1 -- 172.21.15.116:0/1770768662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38004180 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.324+0000 7fed33e00640 1 -- 172.21.15.116:0/1770768662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed38004320 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.324+0000 7fed33e00640 1 -- 172.21.15.116:0/1770768662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38003ca0 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 -- 172.21.15.116:0/1770768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf6ca0 msgr2=0x55a8a8bf5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf5b80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fed38009250 tx=0x7fed38009530 comp rx=0 tx=0).stop 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 -- 172.21.15.116:0/1770768662 shutdown_connections 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c13f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf7690 0x55a8a8c086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 --2- 172.21.15.116:0/1770768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8bf5b80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 -- 172.21.15.116:0/1770768662 >> 172.21.15.116:0/1770768662 conn(0x55a8a8bf2210 msgr2=0x55a8a8bf5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 -- 172.21.15.116:0/1770768662 shutdown_connections 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 -- 172.21.15.116:0/1770768662 wait complete. 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 -- start start 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.325+0000 7fed51222740 1 Processor -- start 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed51222740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c08000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed51222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf7690 0x55a8a8c08540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed51222740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c06610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed51222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481adcf0 con 0x55a8a8bf7690 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed51222740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481af010 con 0x55a8a8bfc800 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed51222740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481ae350 con 0x55a8a8bf6ca0 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed51222740 10 client.0 _ll_register_callbacks cb 0x7fed34005a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf7690 0x55a8a8c08540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed47e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c06610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed47400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c08000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf7690 0x55a8a8c08540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46314/0 (socket says 172.21.15.116:46314) 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 -- 172.21.15.116:0/3972299203 learned_addr learned my addr 172.21.15.116:0/3972299203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 -- 172.21.15.116:0/3972299203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 msgr2=0x55a8a8c08000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c08000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 -- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 msgr2=0x55a8a8c06610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c06610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed46a00640 1 -- 172.21.15.116:0/3972299203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8a8c06d80 con 0x55a8a8bf7690 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed47e00640 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c06610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.326+0000 7fed47400640 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c08000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed46a00640 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf7690 0x55a8a8c08540 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fed34020b40 tx=0x7fed340206d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21551192aa95b156 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed31600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed31600640 1 -- 172.21.15.116:0/3972299203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3403b040 con 0x55a8a8bf7690 2024-03-04T07:49:56.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed31600640 1 -- 172.21.15.116:0/3972299203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed34005040 con 0x55a8a8bf7690 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed38004320 con 0x55a8a8bf7690 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed31600640 1 -- 172.21.15.116:0/3972299203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed3400f6d0 con 0x55a8a8bf7690 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed51222740 10 client.17463 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a8a8c695f0 con 0x55a8a8bf7690 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.327+0000 7fed31600640 1 -- 172.21.15.116:0/3972299203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed38004320 con 0x55a8a8bf7690 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed31600640 1 -- 172.21.15.116:0/3972299203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fed34004c70 con 0x55a8a8bf7690 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 10 client.17463 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 10 client.17463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 20 client.17463 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c69f70 0x55a8a8c6c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 4 client.17463 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a8a8b635f0 con 0x55a8a8c69f70 2024-03-04T07:49:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed47400640 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c69f70 0x55a8a8c6c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed47400640 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c69f70 0x55a8a8c6c390 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a1bd72c4b2a61fc server_cookie=8a076ec8b3d8f895 in_seq=0 out_seq=0 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed31600640 10 client.17463 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed31600640 1 -- 172.21.15.116:0/3972299203 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a8a8b635f0 con 0x55a8a8c69f70 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed31600640 10 client.17463 handle_command_reply: tid=0 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 client.17463 shutdown 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf7690 msgr2=0x55a8a8c08540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf7690 0x55a8a8c08540 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fed34020b40 tx=0x7fed340206d0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 shutdown_connections 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a8a8c69f70 0x55a8a8c6c390 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a8a8bfc800 0x55a8a8c06610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8a8bf7690 0x55a8a8c08540 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 --2- 172.21.15.116:0/3972299203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a8a8bf6ca0 0x55a8a8c08000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed31600640 0 client.17463 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 >> 172.21.15.116:0/3972299203 conn(0x55a8a8bf2210 msgr2=0x55a8a8bf5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 shutdown_connections 2024-03-04T07:49:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 1 -- 172.21.15.116:0/3972299203 wait complete. 2024-03-04T07:49:56.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.329+0000 7fed51222740 20 client.17463 trim_cache size 0 max 16384 2024-03-04T07:49:56.336 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed2c0a1e00 msgr2=0x7fed2c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed2c0a1e00 0x7fed2c0a42c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55a8a8bf7ef0 tx=0x7fed34009260 comp rx=0 tx=0).stop 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed481097a0 msgr2=0x7fed481a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed481097a0 0x7fed481a5430 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fed4002c430 tx=0x7fed400337b0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 shutdown_connections 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed481097a0 0x7fed481a5430 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed2c0a1e00 0x7fed2c0a42c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed48108de0 0x7fed4819e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 --2- 172.21.15.116:0/3439954160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed48107bf0 0x7fed4819de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 >> 172.21.15.116:0/3439954160 conn(0x7fed48076b90 msgr2=0x7fed4810c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 shutdown_connections 2024-03-04T07:49:56.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:56.330+0000 7fed4e400640 1 -- 172.21.15.116:0/3439954160 wait complete. 2024-03-04T07:49:57.139 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:56 smithi133 ceph-mon[36505]: pgmap v676: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:49:57.139 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:56 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1425019010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:56 smithi116 ceph-mon[32150]: pgmap v676: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:49:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:56 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1425019010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:56 smithi152 ceph-mon[37154]: pgmap v676: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:49:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:56 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1425019010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:57.350 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:49:57.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:49:57.350 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:49:57.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 -- 172.21.15.116:0/2100483375 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01015e0 msgr2=0x7f78f0101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 --2- 172.21.15.116:0/2100483375 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01015e0 0x7f78f0101a00 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009ca0 tx=0x7f78e002c100 comp rx=0 tx=0).stop 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 -- 172.21.15.116:0/2100483375 shutdown_connections 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 --2- 172.21.15.116:0/2100483375 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f0103190 0x7f78f0109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 --2- 172.21.15.116:0/2100483375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f01027d0 0x7f78f0102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 --2- 172.21.15.116:0/2100483375 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01015e0 0x7f78f0101a00 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 -- 172.21.15.116:0/2100483375 >> 172.21.15.116:0/2100483375 conn(0x7f78f00fcb70 msgr2=0x7f78f00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 -- 172.21.15.116:0/2100483375 shutdown_connections 2024-03-04T07:49:57.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.579+0000 7f78f4a00640 1 -- 172.21.15.116:0/2100483375 wait complete. 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 Processor -- start 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 -- start start 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01015e0 0x7f78f0199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01027d0 0x7f78f019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0103190 0x7f78f01a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f010d250 con 0x7f78f0103190 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f010d0d0 con 0x7f78f01027d0 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78f4a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f010d3d0 con 0x7f78f01015e0 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.581+0000 7f78ed400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01027d0 0x7f78f019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01027d0 0x7f78f019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:40996/0 (socket says 172.21.15.116:40996) 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 -- 172.21.15.116:0/3910514383 learned_addr learned my addr 172.21.15.116:0/3910514383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ee800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0103190 0x7f78f01a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ede00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01015e0 0x7f78f0199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 -- 172.21.15.116:0/3910514383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01015e0 msgr2=0x7f78f0199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01015e0 0x7f78f0199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 -- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0103190 msgr2=0x7f78f01a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0103190 0x7f78f01a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 -- 172.21.15.116:0/3910514383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f01a1730 con 0x7f78f01027d0 2024-03-04T07:49:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78ed400640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01027d0 0x7f78f019a050 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f78d800e9f0 tx=0x7f78d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78e6a00640 1 -- 172.21.15.116:0/3910514383 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d800cd80 con 0x7f78f01027d0 2024-03-04T07:49:57.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78e6a00640 1 -- 172.21.15.116:0/3910514383 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78d8010910 con 0x7f78f01027d0 2024-03-04T07:49:57.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f01a1a20 con 0x7f78f01027d0 2024-03-04T07:49:57.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.582+0000 7f78e6a00640 1 -- 172.21.15.116:0/3910514383 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8018990 con 0x7f78f01027d0 2024-03-04T07:49:57.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.583+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f01a1fc0 con 0x7f78f01027d0 2024-03-04T07:49:57.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78e6a00640 1 -- 172.21.15.116:0/3910514383 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f78d8010ac0 con 0x7f78f01027d0 2024-03-04T07:49:57.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 Processor -- start 2024-03-04T07:49:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 -- start start 2024-03-04T07:49:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643bd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5642cbe20 con 0x55e5643bdc30 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e56431da20 con 0x55e5643b80d0 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.586+0000 7f78f77cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5642c6b00 con 0x55e5643b8ac0 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41004/0 (socket says 172.21.15.116:41004) 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 -- 172.21.15.116:0/3683896984 learned_addr learned my addr 172.21.15.116:0/3683896984 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ed400640 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ede00640 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643bd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 -- 172.21.15.116:0/3683896984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 msgr2=0x55e5643bd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643bd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 -- 172.21.15.116:0/3683896984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 msgr2=0x55e5643c00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 -- 172.21.15.116:0/3683896984 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5642dbe10 con 0x55e5643b80d0 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ed400640 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78e6a00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1e00 0x7f78d40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ee800640 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b84f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009ca0 tx=0x7f78e8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf0e4fa78d016e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78e6a00640 1 -- 172.21.15.116:0/3910514383 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f78d80ce050 con 0x7f78f01027d0 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78df000640 1 -- 172.21.15.116:0/3683896984 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e801e070 con 0x55e5643b80d0 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.587+0000 7f78ede00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1e00 0x7f78d40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78df000640 1 -- 172.21.15.116:0/3683896984 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e8009e50 con 0x55e5643b80d0 2024-03-04T07:49:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78df000640 1 -- 172.21.15.116:0/3683896984 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e800ea60 con 0x55e5643b80d0 2024-03-04T07:49:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 -- 172.21.15.116:0/3683896984 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 msgr2=0x55e5643b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b84f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009ca0 tx=0x7f78e8011810 comp rx=0 tx=0).stop 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78ede00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1e00 0x7f78d40a42c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f78e002c640 tx=0x7f78e002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 -- 172.21.15.116:0/3683896984 shutdown_connections 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3683896984 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b84f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 -- 172.21.15.116:0/3683896984 >> 172.21.15.116:0/3683896984 conn(0x55e5643b3640 msgr2=0x55e5643b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 -- 172.21.15.116:0/3683896984 shutdown_connections 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.588+0000 7f78f77cb740 1 -- 172.21.15.116:0/3683896984 wait complete. 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.589+0000 7f78f77cb740 1 -- start start 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.589+0000 7f78f77cb740 1 Processor -- start 2024-03-04T07:49:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.589+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643c2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78f77cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5642d9f40 con 0x55e5643bdc30 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78f77cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e564319ac0 con 0x55e5643b80d0 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78f77cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5643c0be0 con 0x55e5643b8ac0 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78f77cb740 10 client.0 _ll_register_callbacks cb 0x55e564318aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643c2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643c2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41006/0 (socket says 172.21.15.116:41006) 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 -- 172.21.15.116:0/3186411740 learned_addr learned my addr 172.21.15.116:0/3186411740 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ee800640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ed400640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 -- 172.21.15.116:0/3186411740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 msgr2=0x55e5643c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643c29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 -- 172.21.15.116:0/3186411740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 msgr2=0x55e5643c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ede00640 1 -- 172.21.15.116:0/3186411740 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5640fa7e0 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.590+0000 7f78ed400640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643c29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78ede00640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643c2470 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f78e0005e30 tx=0x7f78e002c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=605a1d657718d945 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78d3400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78d3400640 1 -- 172.21.15.116:0/3186411740 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0039040 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78d3400640 1 -- 172.21.15.116:0/3186411740 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f78e004c660 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78e8009e50 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78d3400640 1 -- 172.21.15.116:0/3186411740 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78e004c860 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78f77cb740 10 client.27608 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e56415af70 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78d3400640 1 -- 172.21.15.116:0/3186411740 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e8009e50 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.591+0000 7f78d3400640 1 -- 172.21.15.116:0/3186411740 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f78e0040b90 con 0x55e5643b80d0 2024-03-04T07:49:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78f77cb740 10 client.27608 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78f77cb740 10 client.27608 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78f77cb740 20 client.27608 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5643ce360 0x55e5643d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78f77cb740 4 client.27608 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e56432af60 con 0x55e5643ce360 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78ed400640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5643ce360 0x55e5643d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78ed400640 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5643ce360 0x55e5643d07c0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f8f93999809e123 server_cookie=5add9f072c723279 in_seq=0 out_seq=0 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.592+0000 7f78d3400640 10 client.27608 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78d3400640 1 -- 172.21.15.116:0/3186411740 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e56432af60 con 0x55e5643ce360 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78d3400640 10 client.27608 handle_command_reply: tid=0 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 client.27608 shutdown 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 msgr2=0x55e5643c2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643c2470 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f78e0005e30 tx=0x7f78e002c100 comp rx=0 tx=0).stop 2024-03-04T07:49:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 shutdown_connections 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e5643ce360 0x55e5643d07c0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643bdc30 0x55e5643c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b8ac0 0x55e5643c29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3186411740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643c2470 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78d3400640 0 client.27608 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.593+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 >> 172.21.15.116:0/3186411740 conn(0x55e5643b3640 msgr2=0x55e5643b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.594+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 shutdown_connections 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.594+0000 7f78f77cb740 1 -- 172.21.15.116:0/3186411740 wait complete. 2024-03-04T07:49:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.594+0000 7f78f77cb740 20 client.27608 trim_cache size 0 max 16384 2024-03-04T07:49:57.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.611+0000 7f78f77cb740 1 Processor -- start 2024-03-04T07:49:57.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.611+0000 7f78f77cb740 1 -- start start 2024-03-04T07:49:57.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643bdc30 0x55e5643c0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78f77cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01a8be0 con 0x55e5643b8ac0 2024-03-04T07:49:57.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78f77cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01ab1c0 con 0x55e5643b80d0 2024-03-04T07:49:57.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78f77cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01a91e0 con 0x55e5643bdc30 2024-03-04T07:49:57.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41010/0 (socket says 172.21.15.116:41010) 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 -- 172.21.15.116:0/819206125 learned_addr learned my addr 172.21.15.116:0/819206125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ed400640 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ee800640 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643bdc30 0x55e5643c0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 -- 172.21.15.116:0/819206125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643bdc30 msgr2=0x55e5643c0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643bdc30 0x55e5643c0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 -- 172.21.15.116:0/819206125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 msgr2=0x55e5643c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.612+0000 7f78ede00640 1 -- 172.21.15.116:0/819206125 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e563ff2170 con 0x55e5643b80d0 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.613+0000 7f78ede00640 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b28e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f78e0000f80 tx=0x7f78e0032f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=587044ae839a8741 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.613+0000 7f78de600640 1 -- 172.21.15.116:0/819206125 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0039070 con 0x55e5643b80d0 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.613+0000 7f78de600640 1 -- 172.21.15.116:0/819206125 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e0051070 con 0x55e5643b80d0 2024-03-04T07:49:57.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.613+0000 7f78de600640 1 -- 172.21.15.116:0/819206125 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0040c30 con 0x55e5643b80d0 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 -- 172.21.15.116:0/819206125 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 msgr2=0x55e5643b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b28e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f78e0000f80 tx=0x7f78e0032f10 comp rx=0 tx=0).stop 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 -- 172.21.15.116:0/819206125 shutdown_connections 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643bdc30 0x55e5643c0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 --2- 172.21.15.116:0/819206125 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643b80d0 0x55e5643b28e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 -- 172.21.15.116:0/819206125 >> 172.21.15.116:0/819206125 conn(0x55e5643b3640 msgr2=0x55e5643b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 -- 172.21.15.116:0/819206125 shutdown_connections 2024-03-04T07:49:57.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 -- 172.21.15.116:0/819206125 wait complete. 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 -- start start 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.614+0000 7f78f77cb740 1 Processor -- start 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b80d0 0x55e5643c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78f77cb740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643bdc30 0x55e5643c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78f77cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01a9840 con 0x55e5643b8ac0 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78f77cb740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01aab60 con 0x55e5643bdc30 2024-03-04T07:49:57.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78f77cb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01a9ea0 con 0x55e5643b80d0 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78f77cb740 10 client.0 _ll_register_callbacks cb 0x7f78d8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78ede00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b80d0 0x55e5643c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643bdc30 0x55e5643c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78ed400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78ee800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643bdc30 0x55e5643c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41024/0 (socket says 172.21.15.116:41024) 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78ede00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b80d0 0x55e5643c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53730/0 (socket says 172.21.15.116:53730) 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.615+0000 7f78ee800640 1 -- 172.21.15.116:0/3659605775 learned_addr learned my addr 172.21.15.116:0/3659605775 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78ee800640 1 -- 172.21.15.116:0/3659605775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b80d0 msgr2=0x55e5643c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78ee800640 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b80d0 0x55e5643c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78ee800640 1 -- 172.21.15.116:0/3659605775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 msgr2=0x55e5643c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78ee800640 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78ee800640 1 -- 172.21.15.116:0/3659605775 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5643c9310 con 0x55e5643bdc30 2024-03-04T07:49:57.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78ed400640 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78ee800640 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643bdc30 0x55e5643c8d20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009250 tx=0x7f78e80050b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e32173efc602ccd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78d3400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78d3400640 1 -- 172.21.15.116:0/3659605775 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e801e040 con 0x55e5643bdc30 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78e0051070 con 0x55e5643bdc30 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78d3400640 1 -- 172.21.15.116:0/3659605775 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f78e80035f0 con 0x55e5643bdc30 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.616+0000 7f78d3400640 1 -- 172.21.15.116:0/3659605775 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78e80037f0 con 0x55e5643bdc30 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 10 client.27614 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55e56442a330 con 0x55e5643bdc30 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78d3400640 1 -- 172.21.15.116:0/3659605775 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e0051070 con 0x55e5643bdc30 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78d3400640 1 -- 172.21.15.116:0/3659605775 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f78e802fc30 con 0x55e5643bdc30 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 10 client.27614 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 10 client.27614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 20 client.27614 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:57.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e56442ab30 0x55e56442cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 4 client.27614 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:49:57.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e5643252b0 con 0x55e56442ab30 2024-03-04T07:49:57.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.617+0000 7f78ede00640 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e56442ab30 0x55e56442cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.618+0000 7f78ede00640 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e56442ab30 0x55e56442cf50 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afb44252080de31f server_cookie=71294bded7c4c7c8 in_seq=0 out_seq=0 2024-03-04T07:49:57.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.618+0000 7f78d3400640 10 client.27614 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.618+0000 7f78d3400640 1 -- 172.21.15.116:0/3659605775 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e5643252b0 con 0x55e56442ab30 2024-03-04T07:49:57.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78d3400640 10 client.27614 handle_command_reply: tid=0 2024-03-04T07:49:57.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 client.27614 shutdown 2024-03-04T07:49:57.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643bdc30 msgr2=0x55e5643c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643bdc30 0x55e5643c8d20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009250 tx=0x7f78e80050b0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 shutdown_connections 2024-03-04T07:49:57.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55e56442ab30 0x55e56442cf50 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55e5643bdc30 0x55e5643c8d20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5643b8ac0 0x55e5643c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 --2- 172.21.15.116:0/3659605775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e5643b80d0 0x55e5643c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 >> 172.21.15.116:0/3659605775 conn(0x55e5643b3640 msgr2=0x55e5643b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78d3400640 0 client.27614 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 shutdown_connections 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 1 -- 172.21.15.116:0/3659605775 wait complete. 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.619+0000 7f78f77cb740 20 client.27614 trim_cache size 0 max 16384 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:49:57.623 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "b5453036-d4a6-4137-a67d-9800e0499aa5", 2024-03-04T07:49:57.624 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:49:57.624 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:49:57.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1e00 msgr2=0x7f78d40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1e00 0x7f78d40a42c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f78e002c640 tx=0x7f78e002e040 comp rx=0 tx=0).stop 2024-03-04T07:49:57.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01027d0 msgr2=0x7f78f019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01027d0 0x7f78f019a050 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f78d800e9f0 tx=0x7f78d800eeb0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 shutdown_connections 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f0103190 0x7f78f01a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f78f01027d0 0x7f78f019a050 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f78d40a1e00 0x7f78d40a42c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 --2- 172.21.15.116:0/3910514383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78f01015e0 0x7f78f0199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 >> 172.21.15.116:0/3910514383 conn(0x7f78f00fcb70 msgr2=0x7f78f0107f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 shutdown_connections 2024-03-04T07:49:57.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.620+0000 7f78f4a00640 1 -- 172.21.15.116:0/3910514383 wait complete. 2024-03-04T07:49:57.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:49:57.639 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:49:57.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 -- 172.21.15.116:0/330836519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 msgr2=0x7f96d410bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 --2- 172.21.15.116:0/330836519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 0x7f96d410bc30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009be0 tx=0x7f96cc02bef0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 -- 172.21.15.116:0/330836519 shutdown_connections 2024-03-04T07:49:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 --2- 172.21.15.116:0/330836519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 0x7f96d410bc30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 --2- 172.21.15.116:0/330836519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41049e0 0x7f96d4104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 --2- 172.21.15.116:0/330836519 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96d41037f0 0x7f96d4103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 -- 172.21.15.116:0/330836519 >> 172.21.15.116:0/330836519 conn(0x7f96d40fed80 msgr2=0x7f96d41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 -- 172.21.15.116:0/330836519 shutdown_connections 2024-03-04T07:49:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.859+0000 7f96dc200640 1 -- 172.21.15.116:0/330836519 wait complete. 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.860+0000 7f96dc200640 1 Processor -- start 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96dc200640 1 -- start start 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96dc200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41037f0 0x7f96d4199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96dc200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96d41049e0 0x7f96d4199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96dc200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 0x7f96d41a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96dc200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d410f450 con 0x7f96d41053a0 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96dc200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d410f2d0 con 0x7f96d41049e0 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96dc200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d410f5d0 con 0x7f96d41037f0 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96d9800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41037f0 0x7f96d4199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96da200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 0x7f96d41a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96d9800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41037f0 0x7f96d4199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53752/0 (socket says 172.21.15.116:53752) 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96d9800640 1 -- 172.21.15.116:0/2870572633 learned_addr learned my addr 172.21.15.116:0/2870572633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.861+0000 7f96d8e00640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96d41049e0 0x7f96d4199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d9800640 1 -- 172.21.15.116:0/2870572633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96d41049e0 msgr2=0x7f96d4199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d9800640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96d41049e0 0x7f96d4199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d9800640 1 -- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 msgr2=0x7f96d41a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d9800640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 0x7f96d41a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d9800640 1 -- 172.21.15.116:0/2870572633 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d41a1680 con 0x7f96d41037f0 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96da200640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 0x7f96d41a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d8e00640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96d41049e0 0x7f96d4199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:57.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d9800640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41037f0 0x7f96d4199a60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f96c800e9c0 tx=0x7f96c800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d2000640 1 -- 172.21.15.116:0/2870572633 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c800ccc0 con 0x7f96d41037f0 2024-03-04T07:49:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d2000640 1 -- 172.21.15.116:0/2870572633 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96c80035f0 con 0x7f96d41037f0 2024-03-04T07:49:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d41a1970 con 0x7f96d41037f0 2024-03-04T07:49:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.862+0000 7f96d2000640 1 -- 172.21.15.116:0/2870572633 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c800ce60 con 0x7f96d41037f0 2024-03-04T07:49:57.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.863+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d41a1ec0 con 0x7f96d41037f0 2024-03-04T07:49:57.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.866+0000 7f96def8d740 1 Processor -- start 2024-03-04T07:49:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96def8d740 1 -- start start 2024-03-04T07:49:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96def8d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96def8d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96def8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96def8d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6992f7090 con 0x55a6993ca540 2024-03-04T07:49:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96d2000640 1 -- 172.21.15.116:0/2870572633 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f96c80037a0 con 0x7f96d41037f0 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96def8d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a69932ed30 con 0x55a6993c9b50 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96def8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6992d7a30 con 0x55a6993cf6b0 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96da200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96da200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41054/0 (socket says 172.21.15.116:41054) 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96da200640 1 -- 172.21.15.116:0/298606599 learned_addr learned my addr 172.21.15.116:0/298606599 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.867+0000 7f96d9800640 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96d8e00640 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96da200640 1 -- 172.21.15.116:0/298606599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 msgr2=0x55a6993d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96da200640 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96da200640 1 -- 172.21.15.116:0/298606599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 msgr2=0x55a6993cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96da200640 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96da200640 1 -- 172.21.15.116:0/298606599 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a69910d7d0 con 0x55a6993c9b50 2024-03-04T07:49:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96d8e00640 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96d9800640 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:49:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96da200640 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993c9f70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009be0 tx=0x7f96cc005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b09d9860b28a2fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96c6a00640 1 -- 172.21.15.116:0/298606599 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc043030 con 0x55a6993c9b50 2024-03-04T07:49:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96c6a00640 1 -- 172.21.15.116:0/298606599 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96cc003970 con 0x55a6993c9b50 2024-03-04T07:49:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.868+0000 7f96c6a00640 1 -- 172.21.15.116:0/298606599 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc02ccd0 con 0x55a6993c9b50 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96d2000640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96bc0a1d80 0x7f96bc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96d2000640 1 -- 172.21.15.116:0/2870572633 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f96c80ce0f0 con 0x7f96d41037f0 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 -- 172.21.15.116:0/298606599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 msgr2=0x55a6993c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993c9f70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009be0 tx=0x7f96cc005e30 comp rx=0 tx=0).stop 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96d9800640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96bc0a1d80 0x7f96bc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 -- 172.21.15.116:0/298606599 shutdown_connections 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 --2- 172.21.15.116:0/298606599 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993c9f70 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 -- 172.21.15.116:0/298606599 >> 172.21.15.116:0/298606599 conn(0x55a6993c50c0 msgr2=0x55a6993c54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 -- 172.21.15.116:0/298606599 shutdown_connections 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.869+0000 7f96def8d740 1 -- 172.21.15.116:0/298606599 wait complete. 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96d9800640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96bc0a1d80 0x7f96bc0a4240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f96c8097280 tx=0x7f96c8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 -- start start 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 Processor -- start 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993c9b50 0x55a6993d3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993ca540 0x55a6993d4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993cf6b0 0x55a6993db400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6992cd0c0 con 0x55a6993c9b50 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a69932add0 con 0x55a6993cf6b0 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6993d2600 con 0x55a6993ca540 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.870+0000 7f96def8d740 10 client.0 _ll_register_callbacks cb 0x55a6992ea860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96da200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993c9b50 0x55a6993d3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96da200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993c9b50 0x55a6993d3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46396/0 (socket says 172.21.15.116:46396) 2024-03-04T07:49:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96da200640 1 -- 172.21.15.116:0/1961352985 learned_addr learned my addr 172.21.15.116:0/1961352985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96d8e00640 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993ca540 0x55a6993d4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96da200640 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993cf6b0 0x55a6993db400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96d8e00640 1 -- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993cf6b0 msgr2=0x55a6993db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96d8e00640 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993cf6b0 0x55a6993db400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96d8e00640 1 -- 172.21.15.116:0/1961352985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993c9b50 msgr2=0x55a6993d3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96d8e00640 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993c9b50 0x55a6993d3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.871+0000 7f96d8e00640 1 -- 172.21.15.116:0/1961352985 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6990433f0 con 0x55a6993ca540 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96d8e00640 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993ca540 0x55a6993d4430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f96d419af70 tx=0x7f96c000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f67370c450e730 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96bb400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96bb400640 1 -- 172.21.15.116:0/1961352985 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c00120d0 con 0x55a6993ca540 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96bb400640 1 -- 172.21.15.116:0/1961352985 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f96c0004030 con 0x55a6993ca540 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96cc003970 con 0x55a6993ca540 2024-03-04T07:49:57.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96bb400640 1 -- 172.21.15.116:0/1961352985 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96c0012270 con 0x55a6993ca540 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96def8d740 10 client.27505 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.872+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a6990030f0 con 0x55a6993ca540 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96bb400640 1 -- 172.21.15.116:0/1961352985 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96cc003970 con 0x55a6993ca540 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96bb400640 1 -- 172.21.15.116:0/1961352985 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f96c001c460 con 0x55a6993ca540 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96def8d740 10 client.27505 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96def8d740 10 client.27505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96def8d740 20 client.27505 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96def8d740 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a6993df940 0x55a6993e1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96def8d740 4 client.27505 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a69933c270 con 0x55a6993df940 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.873+0000 7f96da200640 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a6993df940 0x55a6993e1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.874+0000 7f96da200640 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a6993df940 0x55a6993e1d80 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48eca9a3af390529 server_cookie=e4aa1f57273a50d6 in_seq=0 out_seq=0 2024-03-04T07:49:57.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.874+0000 7f96bb400640 10 client.27505 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96bb400640 1 -- 172.21.15.116:0/1961352985 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a69933c270 con 0x55a6993df940 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96bb400640 10 client.27505 handle_command_reply: tid=0 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 client.27505 shutdown 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993ca540 msgr2=0x55a6993d4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993ca540 0x55a6993d4430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f96d419af70 tx=0x7f96c000f790 comp rx=0 tx=0).stop 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 shutdown_connections 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a6993df940 0x55a6993e1d80 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993cf6b0 0x55a6993db400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993ca540 0x55a6993d4430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96bb400640 0 client.27505 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 --2- 172.21.15.116:0/1961352985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993c9b50 0x55a6993d3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 >> 172.21.15.116:0/1961352985 conn(0x55a6993c50c0 msgr2=0x55a6993c4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 shutdown_connections 2024-03-04T07:49:57.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 1 -- 172.21.15.116:0/1961352985 wait complete. 2024-03-04T07:49:57.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.875+0000 7f96def8d740 20 client.27505 trim_cache size 0 max 16384 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 Processor -- start 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 -- start start 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993c9b50 0x55a6993c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993ca540 0x55a6993db3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993cf6b0 0x55a6993c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41a8c00 con 0x55a6993cf6b0 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41ab1e0 con 0x55a6993ca540 2024-03-04T07:49:57.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.894+0000 7f96def8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41a9200 con 0x55a6993c9b50 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993ca540 0x55a6993db3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993ca540 0x55a6993db3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41072/0 (socket says 172.21.15.116:41072) 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96da200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993c9b50 0x55a6993c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 -- 172.21.15.116:0/455177096 learned_addr learned my addr 172.21.15.116:0/455177096 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96da200640 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993cf6b0 0x55a6993c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 -- 172.21.15.116:0/455177096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993c9b50 msgr2=0x55a6993c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993c9b50 0x55a6993c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 -- 172.21.15.116:0/455177096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993cf6b0 msgr2=0x55a6993c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993cf6b0 0x55a6993c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 -- 172.21.15.116:0/455177096 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6993ced50 con 0x55a6993ca540 2024-03-04T07:49:57.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.895+0000 7f96d8e00640 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993ca540 0x55a6993db3c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f96c00058c0 tx=0x7f96c00127f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=163dcd8e0e0d6d83 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.896+0000 7f96c6000640 1 -- 172.21.15.116:0/455177096 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c000fc90 con 0x55a6993ca540 2024-03-04T07:49:57.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.896+0000 7f96c6000640 1 -- 172.21.15.116:0/455177096 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96c00092d0 con 0x55a6993ca540 2024-03-04T07:49:57.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.896+0000 7f96c6000640 1 -- 172.21.15.116:0/455177096 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c00035f0 con 0x55a6993ca540 2024-03-04T07:49:57.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.896+0000 7f96def8d740 1 -- 172.21.15.116:0/455177096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993ca540 msgr2=0x55a6993db3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.896+0000 7f96def8d740 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993ca540 0x55a6993db3c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f96c00058c0 tx=0x7f96c00127f0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 -- 172.21.15.116:0/455177096 shutdown_connections 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993cf6b0 0x55a6993c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993ca540 0x55a6993db3c0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 --2- 172.21.15.116:0/455177096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993c9b50 0x55a6993c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 -- 172.21.15.116:0/455177096 >> 172.21.15.116:0/455177096 conn(0x55a6993c50c0 msgr2=0x55a6993c8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 -- 172.21.15.116:0/455177096 shutdown_connections 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 -- 172.21.15.116:0/455177096 wait complete. 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 -- start start 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.897+0000 7f96def8d740 1 Processor -- start 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96def8d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993db0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96def8d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993d96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96def8d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96def8d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41a9860 con 0x55a6993ca540 2024-03-04T07:49:57.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96def8d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41aab80 con 0x55a6993c9b50 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96def8d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41a9ec0 con 0x55a6993cf6b0 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96def8d740 10 client.0 _ll_register_callbacks cb 0x7f96c0012910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96da200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993db0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96d8e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993d96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96da200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993db0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41078/0 (socket says 172.21.15.116:41078) 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96da200640 1 -- 172.21.15.116:0/1648556466 learned_addr learned my addr 172.21.15.116:0/1648556466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96da200640 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96d8e00640 1 -- 172.21.15.116:0/1648556466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 msgr2=0x55a6993d9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96d8e00640 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96d8e00640 1 -- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 msgr2=0x55a6993db0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96d8e00640 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993db0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96d8e00640 1 -- 172.21.15.116:0/1648556466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6993da250 con 0x55a6993ca540 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.898+0000 7f96da200640 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993db0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96d8e00640 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993d96d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x55a6993d5190 tx=0x7f96c0009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1dc846be1b59bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:57.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96bb400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96bb400640 1 -- 172.21.15.116:0/1648556466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c001cac0 con 0x55a6993ca540 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96bb400640 1 -- 172.21.15.116:0/1648556466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f96c001cc60 con 0x55a6993ca540 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96c00092d0 con 0x55a6993ca540 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96bb400640 1 -- 172.21.15.116:0/1648556466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96c001ce60 con 0x55a6993ca540 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96def8d740 10 client.17499 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.899+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a69943c0b0 con 0x55a6993ca540 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96bb400640 1 -- 172.21.15.116:0/1648556466 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96c00092d0 con 0x55a6993ca540 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96bb400640 1 -- 172.21.15.116:0/1648556466 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f96c003c760 con 0x55a6993ca540 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96def8d740 10 client.17499 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96def8d740 10 client.17499 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96def8d740 20 client.17499 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96def8d740 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a69943c890 0x55a69943ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:57.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96def8d740 4 client.17499 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:49:57.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a6993365c0 con 0x55a69943c890 2024-03-04T07:49:57.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.900+0000 7f96da200640 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a69943c890 0x55a69943ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:57.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96bb400640 10 client.17499 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96da200640 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a69943c890 0x55a69943ecb0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c065c2d6b0216b58 server_cookie=e92df2e8ea9933ac in_seq=0 out_seq=0 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96bb400640 1 -- 172.21.15.116:0/1648556466 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a6993365c0 con 0x55a69943c890 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96bb400640 10 client.17499 handle_command_reply: tid=0 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 client.17499 shutdown 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 msgr2=0x55a6993d96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993d96d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x55a6993d5190 tx=0x7f96c0009b90 comp rx=0 tx=0).stop 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 shutdown_connections 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a69943c890 0x55a69943ecb0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a6993cf6b0 0x55a6993d9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a6993ca540 0x55a6993d96d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 --2- 172.21.15.116:0/1648556466 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a6993c9b50 0x55a6993db0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96bb400640 0 client.17499 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 >> 172.21.15.116:0/1648556466 conn(0x55a6993c50c0 msgr2=0x55a6993d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 shutdown_connections 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 1 -- 172.21.15.116:0/1648556466 wait complete. 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.901+0000 7f96def8d740 20 client.17499 trim_cache size 0 max 16384 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:49:57.906 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96bc0a1d80 msgr2=0x7f96bc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96bc0a1d80 0x7f96bc0a4240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f96c8097280 tx=0x7f96c8000bc0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41037f0 msgr2=0x7f96d4199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41037f0 0x7f96d4199a60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f96c800e9c0 tx=0x7f96c800edc0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 shutdown_connections 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96d41053a0 0x7f96d41a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96d41049e0 0x7f96d4199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96bc0a1d80 0x7f96bc0a4240 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 --2- 172.21.15.116:0/2870572633 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96d41037f0 0x7f96d4199a60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 >> 172.21.15.116:0/2870572633 conn(0x7f96d40fed80 msgr2=0x7f96d4107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 shutdown_connections 2024-03-04T07:49:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:57.902+0000 7f96dc200640 1 -- 172.21.15.116:0/2870572633 wait complete. 2024-03-04T07:49:57.920 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:49:57.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:49:57.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:57 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:49:57.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:57 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:57.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:57 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:49:57.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:57 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:57.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:57 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:49:58.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 -- 172.21.15.116:0/3811524939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 msgr2=0x7f1c28103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 --2- 172.21.15.116:0/3811524939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28103c30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009900 tx=0x7f1c1c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 -- 172.21.15.116:0/3811524939 shutdown_connections 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 --2- 172.21.15.116:0/3811524939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c281053c0 0x7f1c2810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 --2- 172.21.15.116:0/3811524939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c28104a00 0x7f1c28104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 --2- 172.21.15.116:0/3811524939 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28103c30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 -- 172.21.15.116:0/3811524939 >> 172.21.15.116:0/3811524939 conn(0x7f1c280fedc0 msgr2=0x7f1c281011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 -- 172.21.15.116:0/3811524939 shutdown_connections 2024-03-04T07:49:58.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.122+0000 7f1c30800640 1 -- 172.21.15.116:0/3811524939 wait complete. 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.123+0000 7f1c30800640 1 Processor -- start 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.123+0000 7f1c30800640 1 -- start start 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.123+0000 7f1c30800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.123+0000 7f1c30800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c28104a00 0x7f1c2819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.123+0000 7f1c30800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c281053c0 0x7f1c281a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c30800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2810f4b0 con 0x7f1c281053c0 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c30800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2810f330 con 0x7f1c28103810 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c30800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2810f630 con 0x7f1c28104a00 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41094/0 (socket says 172.21.15.116:41094) 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 -- 172.21.15.116:0/2739847319 learned_addr learned my addr 172.21.15.116:0/2739847319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2e800640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c281053c0 0x7f1c281a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2d400640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c28104a00 0x7f1c2819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 -- 172.21.15.116:0/2739847319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c28104a00 msgr2=0x7f1c2819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c28104a00 0x7f1c2819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 -- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c281053c0 msgr2=0x7f1c281a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c281053c0 0x7f1c281a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 -- 172.21.15.116:0/2739847319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c281a1770 con 0x7f1c28103810 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.124+0000 7f1c2de00640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28199b50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c000c00 tx=0x7f1c1c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.125+0000 7f1c22a00640 1 -- 172.21.15.116:0/2739847319 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c043020 con 0x7f1c28103810 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.125+0000 7f1c22a00640 1 -- 172.21.15.116:0/2739847319 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c1c0054e0 con 0x7f1c28103810 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.125+0000 7f1c22a00640 1 -- 172.21.15.116:0/2739847319 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c003bd0 con 0x7f1c28103810 2024-03-04T07:49:58.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.125+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c281a1a00 con 0x7f1c28103810 2024-03-04T07:49:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.125+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c281a1f50 con 0x7f1c28103810 2024-03-04T07:49:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.126+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c28068810 con 0x7f1c28103810 2024-03-04T07:49:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.128+0000 7f1c22a00640 1 -- 172.21.15.116:0/2739847319 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1c1c046020 con 0x7f1c28103810 2024-03-04T07:49:58.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.129+0000 7f1c22a00640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c100a1d80 0x7f1c100a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.129+0000 7f1c22a00640 1 -- 172.21.15.116:0/2739847319 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1c1c0efdb0 con 0x7f1c28103810 2024-03-04T07:49:58.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.129+0000 7f1c2d400640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c100a1d80 0x7f1c100a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.130+0000 7f1c2d400640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c100a1d80 0x7f1c100a4240 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1c2819b060 tx=0x7f1c18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.133+0000 7f1c22a00640 1 -- 172.21.15.116:0/2739847319 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1c1c04d020 con 0x7f1c28103810 2024-03-04T07:49:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:57 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:49:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:57 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:57 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:49:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:57 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:58.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:57 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:49:58.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:57 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:49:58.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:57 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:58.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:57 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:49:58.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:57 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:49:58.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:57 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:49:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.370+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c2810ab30 con 0x7f1c28103810 2024-03-04T07:49:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.371+0000 7f1c22a00640 1 -- 172.21.15.116:0/2739847319 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f1c1c0bd1e0 con 0x7f1c28103810 2024-03-04T07:49:58.372 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:49:58.372 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:49:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:49:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c100a1d80 msgr2=0x7f1c100a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c100a1d80 0x7f1c100a4240 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1c2819b060 tx=0x7f1c18005f90 comp rx=0 tx=0).stop 2024-03-04T07:49:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 msgr2=0x7f1c28199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28199b50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c000c00 tx=0x7f1c1c02c650 comp rx=0 tx=0).stop 2024-03-04T07:49:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 shutdown_connections 2024-03-04T07:49:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c281053c0 0x7f1c281a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1c100a1d80 0x7f1c100a4240 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c28104a00 0x7f1c2819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 --2- 172.21.15.116:0/2739847319 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1c28103810 0x7f1c28199b50 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 >> 172.21.15.116:0/2739847319 conn(0x7f1c280fedc0 msgr2=0x7f1c28107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.374+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 shutdown_connections 2024-03-04T07:49:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.375+0000 7f1c30800640 1 -- 172.21.15.116:0/2739847319 wait complete. 2024-03-04T07:49:58.394 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:49:58.394 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:49:58.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 -- 172.21.15.116:0/2321120176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94104a00 msgr2=0x7fca94104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 --2- 172.21.15.116:0/2321120176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94104a00 0x7fca94104e80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fca7c009900 tx=0x7fca7c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:49:58.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 -- 172.21.15.116:0/2321120176 shutdown_connections 2024-03-04T07:49:58.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 --2- 172.21.15.116:0/2321120176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca941053c0 0x7fca9410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 --2- 172.21.15.116:0/2321120176 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94104a00 0x7fca94104e80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 --2- 172.21.15.116:0/2321120176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca94103810 0x7fca94103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 -- 172.21.15.116:0/2321120176 >> 172.21.15.116:0/2321120176 conn(0x7fca940fed80 msgr2=0x7fca941011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 -- 172.21.15.116:0/2321120176 shutdown_connections 2024-03-04T07:49:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.613+0000 7fca99e00640 1 -- 172.21.15.116:0/2321120176 wait complete. 2024-03-04T07:49:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.614+0000 7fca99e00640 1 Processor -- start 2024-03-04T07:49:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.614+0000 7fca99e00640 1 -- start start 2024-03-04T07:49:58.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.614+0000 7fca99e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94103810 0x7fca94199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca99e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca94104a00 0x7fca9419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca99e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca941053c0 0x7fca941a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca99e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9410f470 con 0x7fca941053c0 2024-03-04T07:49:58.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca99e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9410f2f0 con 0x7fca94103810 2024-03-04T07:49:58.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca99e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9410f5f0 con 0x7fca94104a00 2024-03-04T07:49:58.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94103810 0x7fca94199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca93400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca941053c0 0x7fca941a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94103810 0x7fca94199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41122/0 (socket says 172.21.15.116:41122) 2024-03-04T07:49:58.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 -- 172.21.15.116:0/2447696159 learned_addr learned my addr 172.21.15.116:0/2447696159 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:58.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92000640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca94104a00 0x7fca9419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 -- 172.21.15.116:0/2447696159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca94104a00 msgr2=0x7fca9419a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca94104a00 0x7fca9419a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 -- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca941053c0 msgr2=0x7fca941a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca941053c0 0x7fca941a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.615+0000 7fca92a00640 1 -- 172.21.15.116:0/2447696159 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca941a1700 con 0x7fca94103810 2024-03-04T07:49:58.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.616+0000 7fca92a00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94103810 0x7fca94199ae0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fca8400e9c0 tx=0x7fca8400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.616+0000 7fca98a00640 1 -- 172.21.15.116:0/2447696159 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8400cde0 con 0x7fca94103810 2024-03-04T07:49:58.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.616+0000 7fca98a00640 1 -- 172.21.15.116:0/2447696159 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca84010910 con 0x7fca94103810 2024-03-04T07:49:58.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.616+0000 7fca98a00640 1 -- 172.21.15.116:0/2447696159 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca84018a00 con 0x7fca94103810 2024-03-04T07:49:58.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.616+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca941a19f0 con 0x7fca94103810 2024-03-04T07:49:58.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.616+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca941a1ec0 con 0x7fca94103810 2024-03-04T07:49:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.619+0000 7fca98a00640 1 -- 172.21.15.116:0/2447696159 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fca8401f020 con 0x7fca94103810 2024-03-04T07:49:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.619+0000 7fca9cc1b740 1 Processor -- start 2024-03-04T07:49:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.619+0000 7fca9cc1b740 1 -- start start 2024-03-04T07:49:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c0d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c0fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa7b6cd60 con 0x559fa7c0d800 2024-03-04T07:49:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa7b15cf0 con 0x559fa7c08690 2024-03-04T07:49:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa7b68e00 con 0x559fa7c07ca0 2024-03-04T07:49:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca92a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c0d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca92a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c0d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41126/0 (socket says 172.21.15.116:41126) 2024-03-04T07:49:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca92a00640 1 -- 172.21.15.116:0/4261241022 learned_addr learned my addr 172.21.15.116:0/4261241022 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca92000640 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c0fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.620+0000 7fca93400640 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca92a00640 1 -- 172.21.15.116:0/4261241022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 msgr2=0x559fa7c080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca92a00640 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca92a00640 1 -- 172.21.15.116:0/4261241022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 msgr2=0x559fa7c0fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca92a00640 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c0fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca92a00640 1 -- 172.21.15.116:0/4261241022 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fa78813f0 con 0x559fa7c08690 2024-03-04T07:49:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca98a00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca780a1e00 0x7fca780a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca92a00640 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c0d190 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fca840d6190 tx=0x7fca840d6280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=213363e0e4f765bf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca93400640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca780a1e00 0x7fca780a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca83e00640 1 -- 172.21.15.116:0/4261241022 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca84096f80 con 0x559fa7c08690 2024-03-04T07:49:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca98a00640 1 -- 172.21.15.116:0/2447696159 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fca840cdff0 con 0x7fca94103810 2024-03-04T07:49:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca83e00640 1 -- 172.21.15.116:0/4261241022 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca8409b3e0 con 0x559fa7c08690 2024-03-04T07:49:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.621+0000 7fca83e00640 1 -- 172.21.15.116:0/4261241022 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca840c9360 con 0x559fa7c08690 2024-03-04T07:49:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 -- 172.21.15.116:0/4261241022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 msgr2=0x559fa7c0d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca93400640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca780a1e00 0x7fca780a42c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x559fa7c08ef0 tx=0x7fca8c009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c0d190 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fca840d6190 tx=0x7fca840d6280 comp rx=0 tx=0).stop 2024-03-04T07:49:58.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 -- 172.21.15.116:0/4261241022 shutdown_connections 2024-03-04T07:49:58.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c0fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c0d190 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/4261241022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 -- 172.21.15.116:0/4261241022 >> 172.21.15.116:0/4261241022 conn(0x559fa7c03210 msgr2=0x559fa7c03610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 -- 172.21.15.116:0/4261241022 shutdown_connections 2024-03-04T07:49:58.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.622+0000 7fca9cc1b740 1 -- 172.21.15.116:0/4261241022 wait complete. 2024-03-04T07:49:58.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 -- start start 2024-03-04T07:49:58.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 Processor -- start 2024-03-04T07:49:58.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c12210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c12750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c19790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa7c107b0 con 0x559fa7c0d800 2024-03-04T07:49:58.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa7b0b220 con 0x559fa7c08690 2024-03-04T07:49:58.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa7c10930 con 0x559fa7c07ca0 2024-03-04T07:49:58.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.623+0000 7fca9cc1b740 10 client.0 _ll_register_callbacks cb 0x559fa7880be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:58.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c19790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca92000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c12750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c19790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46462/0 (socket says 172.21.15.116:46462) 2024-03-04T07:49:58.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 -- 172.21.15.116:0/2768788295 learned_addr learned my addr 172.21.15.116:0/2768788295 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:58.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca92a00640 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c12210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 -- 172.21.15.116:0/2768788295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 msgr2=0x559fa7c12210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c12210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 -- 172.21.15.116:0/2768788295 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 msgr2=0x559fa7c12750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c12750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 -- 172.21.15.116:0/2768788295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fa794b7d0 con 0x559fa7c0d800 2024-03-04T07:49:58.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca93400640 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c19790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0042b0 tx=0x7fca8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7101bc6a8e2ac16 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.624+0000 7fca80c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:49:58.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca80c00640 1 -- 172.21.15.116:0/2768788295 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c025070 con 0x559fa7c0d800 2024-03-04T07:49:58.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca80c00640 1 -- 172.21.15.116:0/2768788295 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fca8c011d60 con 0x559fa7c0d800 2024-03-04T07:49:58.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca8409b3e0 con 0x559fa7c0d800 2024-03-04T07:49:58.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca80c00640 1 -- 172.21.15.116:0/2768788295 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca8c031050 con 0x559fa7c0d800 2024-03-04T07:49:58.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca9cc1b740 10 client.17517 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:58.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559fa7c1cbd0 con 0x559fa7c0d800 2024-03-04T07:49:58.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca80c00640 1 -- 172.21.15.116:0/2768788295 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca8409b3e0 con 0x559fa7c0d800 2024-03-04T07:49:58.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.625+0000 7fca80c00640 1 -- 172.21.15.116:0/2768788295 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fca8c00f040 con 0x559fa7c0d800 2024-03-04T07:49:58.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca9cc1b740 10 client.17517 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:58.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca9cc1b740 10 client.17517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:58.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca9cc1b740 20 client.17517 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:58.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c1e2b0 0x559fa7c206e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca9cc1b740 4 client.17517 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:58.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fa7b7a2a0 con 0x559fa7c1e2b0 2024-03-04T07:49:58.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca92a00640 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c1e2b0 0x559fa7c206e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca92a00640 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c1e2b0 0x559fa7c206e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3af3db5b53bdfab8 server_cookie=fd9baf07f996cee in_seq=0 out_seq=0 2024-03-04T07:49:58.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.626+0000 7fca80c00640 10 client.17517 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:58.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca80c00640 1 -- 172.21.15.116:0/2768788295 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559fa7b7a2a0 con 0x559fa7c1e2b0 2024-03-04T07:49:58.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca80c00640 10 client.17517 handle_command_reply: tid=0 2024-03-04T07:49:58.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 client.17517 shutdown 2024-03-04T07:49:58.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 msgr2=0x559fa7c19790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c19790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0042b0 tx=0x7fca8c004a40 comp rx=0 tx=0).stop 2024-03-04T07:49:58.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 shutdown_connections 2024-03-04T07:49:58.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c1e2b0 0x559fa7c206e0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c0d800 0x559fa7c19790 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c08690 0x559fa7c12750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/2768788295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c12210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca80c00640 0 client.17517 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:58.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.627+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 >> 172.21.15.116:0/2768788295 conn(0x559fa7c03210 msgr2=0x559fa7c024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.628+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 shutdown_connections 2024-03-04T07:49:58.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.628+0000 7fca9cc1b740 1 -- 172.21.15.116:0/2768788295 wait complete. 2024-03-04T07:49:58.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.628+0000 7fca9cc1b740 20 client.17517 trim_cache size 0 max 16384 2024-03-04T07:49:58.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 Processor -- start 2024-03-04T07:49:58.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 -- start start 2024-03-04T07:49:58.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c07ca0 0x559fa7c09b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c0a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c0d800 0x559fa7c196b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941a8d60 con 0x559fa7c08690 2024-03-04T07:49:58.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941ab340 con 0x559fa7c07ca0 2024-03-04T07:49:58.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.641+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941a9360 con 0x559fa7c0d800 2024-03-04T07:49:58.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c07ca0 0x559fa7c09b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c07ca0 0x559fa7c09b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41148/0 (socket says 172.21.15.116:41148) 2024-03-04T07:49:58.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 -- 172.21.15.116:0/3052976932 learned_addr learned my addr 172.21.15.116:0/3052976932 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:58.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92000640 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c0a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca93400640 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c0d800 0x559fa7c196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 -- 172.21.15.116:0/3052976932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c0d800 msgr2=0x559fa7c196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c0d800 0x559fa7c196b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 -- 172.21.15.116:0/3052976932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 msgr2=0x559fa7c0a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c0a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 -- 172.21.15.116:0/3052976932 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fa7c1e090 con 0x559fa7c07ca0 2024-03-04T07:49:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca92a00640 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c07ca0 0x559fa7c09b90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fca840976b0 tx=0x7fca8400efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c663a56bd6772f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.642+0000 7fca83400640 1 -- 172.21.15.116:0/3052976932 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca840c8cc0 con 0x559fa7c07ca0 2024-03-04T07:49:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca83400640 1 -- 172.21.15.116:0/3052976932 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca8409b3e0 con 0x559fa7c07ca0 2024-03-04T07:49:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca83400640 1 -- 172.21.15.116:0/3052976932 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca840c8ee0 con 0x559fa7c07ca0 2024-03-04T07:49:58.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 -- 172.21.15.116:0/3052976932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c07ca0 msgr2=0x559fa7c09b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c07ca0 0x559fa7c09b90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fca840976b0 tx=0x7fca8400efb0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 -- 172.21.15.116:0/3052976932 shutdown_connections 2024-03-04T07:49:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c0d800 0x559fa7c196b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c0a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/3052976932 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c07ca0 0x559fa7c09b90 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 -- 172.21.15.116:0/3052976932 >> 172.21.15.116:0/3052976932 conn(0x559fa7c03210 msgr2=0x559fa7c10d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 -- 172.21.15.116:0/3052976932 shutdown_connections 2024-03-04T07:49:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.643+0000 7fca9cc1b740 1 -- 172.21.15.116:0/3052976932 wait complete. 2024-03-04T07:49:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 -- start start 2024-03-04T07:49:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 Processor -- start 2024-03-04T07:49:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c17410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c17950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c0d800 0x559fa7c188f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941a99c0 con 0x559fa7c08690 2024-03-04T07:49:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941aace0 con 0x559fa7c0d800 2024-03-04T07:49:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941aa020 con 0x559fa7c07ca0 2024-03-04T07:49:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca9cc1b740 10 client.0 _ll_register_callbacks cb 0x7fca8c009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:58.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c0d800 0x559fa7c188f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca92000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c17950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.644+0000 7fca92a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c17410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c0d800 0x559fa7c188f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41160/0 (socket says 172.21.15.116:41160) 2024-03-04T07:49:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 -- 172.21.15.116:0/1673726292 learned_addr learned my addr 172.21.15.116:0/1673726292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 -- 172.21.15.116:0/1673726292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 msgr2=0x559fa7c17410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c17410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 -- 172.21.15.116:0/1673726292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 msgr2=0x559fa7c17950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c17950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 -- 172.21.15.116:0/1673726292 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fa7c18ee0 con 0x559fa7c0d800 2024-03-04T07:49:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca92a00640 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c17410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca93400640 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c0d800 0x559fa7c188f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fca8c029650 tx=0x7fca8c005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7259dde30ab46df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca80c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca80c00640 1 -- 172.21.15.116:0/1673726292 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c039020 con 0x559fa7c0d800 2024-03-04T07:49:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca80c00640 1 -- 172.21.15.116:0/1673726292 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fca8c011dc0 con 0x559fa7c0d800 2024-03-04T07:49:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.645+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca8409b3e0 con 0x559fa7c0d800 2024-03-04T07:49:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca80c00640 1 -- 172.21.15.116:0/1673726292 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca8c025040 con 0x559fa7c0d800 2024-03-04T07:49:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 10 client.27656 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559fa7c82850 con 0x559fa7c0d800 2024-03-04T07:49:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca80c00640 1 -- 172.21.15.116:0/1673726292 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca8409b3e0 con 0x559fa7c0d800 2024-03-04T07:49:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca80c00640 1 -- 172.21.15.116:0/1673726292 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fca8c034070 con 0x559fa7c0d800 2024-03-04T07:49:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 10 client.27656 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 10 client.27656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 20 client.27656 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c83180 0x559fa7c855a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 4 client.27656 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:49:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559fa7b745f0 con 0x559fa7c83180 2024-03-04T07:49:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.646+0000 7fca92a00640 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c83180 0x559fa7c855a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca92a00640 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c83180 0x559fa7c855a0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2ce78d0d785aa9c server_cookie=476a6d2c1aa26292 in_seq=0 out_seq=0 2024-03-04T07:49:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca80c00640 10 client.27656 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca80c00640 1 -- 172.21.15.116:0/1673726292 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559fa7b745f0 con 0x559fa7c83180 2024-03-04T07:49:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca80c00640 10 client.27656 handle_command_reply: tid=0 2024-03-04T07:49:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 client.27656 shutdown 2024-03-04T07:49:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c0d800 msgr2=0x559fa7c188f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c0d800 0x559fa7c188f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fca8c029650 tx=0x7fca8c005ac0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 shutdown_connections 2024-03-04T07:49:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x559fa7c83180 0x559fa7c855a0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559fa7c0d800 0x559fa7c188f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559fa7c08690 0x559fa7c17950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 --2- 172.21.15.116:0/1673726292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559fa7c07ca0 0x559fa7c17410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca80c00640 0 client.27656 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 >> 172.21.15.116:0/1673726292 conn(0x559fa7c03210 msgr2=0x559fa7c121d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 shutdown_connections 2024-03-04T07:49:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 1 -- 172.21.15.116:0/1673726292 wait complete. 2024-03-04T07:49:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.647+0000 7fca9cc1b740 20 client.27656 trim_cache size 0 max 16384 2024-03-04T07:49:58.678 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:49:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca780a1e00 msgr2=0x7fca780a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca780a1e00 0x7fca780a42c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x559fa7c08ef0 tx=0x7fca8c009600 comp rx=0 tx=0).stop 2024-03-04T07:49:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94103810 msgr2=0x7fca94199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94103810 0x7fca94199ae0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fca8400e9c0 tx=0x7fca8400ee80 comp rx=0 tx=0).stop 2024-03-04T07:49:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 shutdown_connections 2024-03-04T07:49:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca941053c0 0x7fca941a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fca780a1e00 0x7fca780a42c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fca94104a00 0x7fca9419a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 --2- 172.21.15.116:0/2447696159 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fca94103810 0x7fca94199ae0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 >> 172.21.15.116:0/2447696159 conn(0x7fca940fed80 msgr2=0x7fca94107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 shutdown_connections 2024-03-04T07:49:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:58.648+0000 7fca99e00640 1 -- 172.21.15.116:0/2447696159 wait complete. 2024-03-04T07:49:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:58 smithi116 ceph-mon[32150]: pgmap v677: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:59.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:49:58 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2739847319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 -- 172.21.15.116:0/2318481707 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 msgr2=0x7f2980104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 --2- 172.21.15.116:0/2318481707 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f2980104ea0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2968009900 tx=0x7f296802bc40 comp rx=0 tx=0).stop 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 -- 172.21.15.116:0/2318481707 shutdown_connections 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 --2- 172.21.15.116:0/2318481707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29801053e0 0x7f298010bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 --2- 172.21.15.116:0/2318481707 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f2980104ea0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 --2- 172.21.15.116:0/2318481707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2980103830 0x7f2980103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 -- 172.21.15.116:0/2318481707 >> 172.21.15.116:0/2318481707 conn(0x7f29800fedc0 msgr2=0x7f29801011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 -- 172.21.15.116:0/2318481707 shutdown_connections 2024-03-04T07:49:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.182+0000 7f2986c00640 1 -- 172.21.15.116:0/2318481707 wait complete. 2024-03-04T07:49:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 Processor -- start 2024-03-04T07:49:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 -- start start 2024-03-04T07:49:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2980103830 0x7f2980199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f298019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29801053e0 0x7f29801a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298010f4e0 con 0x7f29801053e0 2024-03-04T07:49:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298010f360 con 0x7f2980104a20 2024-03-04T07:49:59.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2986c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298010f660 con 0x7f2980103830 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.184+0000 7f2977400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f298019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f298019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41180/0 (socket says 172.21.15.116:41180) 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 -- 172.21.15.116:0/1476166712 learned_addr learned my addr 172.21.15.116:0/1476166712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2984c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29801053e0 0x7f29801a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f297fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2980103830 0x7f2980199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 -- 172.21.15.116:0/1476166712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2980103830 msgr2=0x7f2980199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2980103830 0x7f2980199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 -- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29801053e0 msgr2=0x7f29801a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29801053e0 0x7f29801a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 -- 172.21.15.116:0/1476166712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29801a16e0 con 0x7f2980104a20 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2977400640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f298019a000 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f296802c8f0 tx=0x7f2968005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f297d600640 1 -- 172.21.15.116:0/1476166712 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968041070 con 0x7f2980104a20 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f297d600640 1 -- 172.21.15.116:0/1476166712 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29680054e0 con 0x7f2980104a20 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29801a1970 con 0x7f2980104a20 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.185+0000 7f297d600640 1 -- 172.21.15.116:0/1476166712 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968034440 con 0x7f2980104a20 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.186+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29801a1e80 con 0x7f2980104a20 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f297d600640 1 -- 172.21.15.116:0/1476166712 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2968044070 con 0x7f2980104a20 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 Processor -- start 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 -- start start 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4d4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da5a432370 con 0x55da5a4cd240 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da5a3db550 con 0x55da5a4d23b0 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2989983740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da5a42e410 con 0x55da5a4cc850 2024-03-04T07:49:59.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2984c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f297fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2984c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53880/0 (socket says 172.21.15.116:53880) 2024-03-04T07:49:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.189+0000 7f2984c00640 1 -- 172.21.15.116:0/3500087513 learned_addr learned my addr 172.21.15.116:0/3500087513 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2977400640 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4d4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2984c00640 1 -- 172.21.15.116:0/3500087513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 msgr2=0x55da5a4d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2984c00640 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4d4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2984c00640 1 -- 172.21.15.116:0/3500087513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 msgr2=0x55da5a4d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2984c00640 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2984c00640 1 -- 172.21.15.116:0/3500087513 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da5a3ef3c0 con 0x55da5a4cc850 2024-03-04T07:49:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2984c00640 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4ccc70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2978009f70 tx=0x7f2978010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4851705b140a5c15 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f297d600640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29640a1ea0 0x7f29640a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.190+0000 7f2975200640 1 -- 172.21.15.116:0/3500087513 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978016070 con 0x55da5a4cc850 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.191+0000 7f297d600640 1 -- 172.21.15.116:0/1476166712 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f29680ef9b0 con 0x7f2980104a20 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.191+0000 7f2975200640 1 -- 172.21.15.116:0/3500087513 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2978004d20 con 0x55da5a4cc850 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.191+0000 7f2975200640 1 -- 172.21.15.116:0/3500087513 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978012910 con 0x55da5a4cc850 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.191+0000 7f297fe00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29640a1ea0 0x7f29640a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.191+0000 7f2989983740 1 -- 172.21.15.116:0/3500087513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 msgr2=0x55da5a4ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.191+0000 7f2989983740 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4ccc70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2978009f70 tx=0x7f2978010800 comp rx=0 tx=0).stop 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.191+0000 7f297fe00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29640a1ea0 0x7f29640a4360 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x55da5a4cd640 tx=0x7f297000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 -- 172.21.15.116:0/3500087513 shutdown_connections 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4d4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 --2- 172.21.15.116:0/3500087513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4ccc70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 -- 172.21.15.116:0/3500087513 >> 172.21.15.116:0/3500087513 conn(0x55da5a4c7dc0 msgr2=0x55da5a4c81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 -- 172.21.15.116:0/3500087513 shutdown_connections 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 -- 172.21.15.116:0/3500087513 wait complete. 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.192+0000 7f2989983740 1 -- start start 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 1 Processor -- start 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4d6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cd240 0x55da5a4d70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4d23b0 0x55da5a4de130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da5a4d5360 con 0x55da5a4d23b0 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da5a3ee9b0 con 0x55da5a4cd240 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da5a4d54e0 con 0x55da5a4cc850 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2989983740 10 client.0 _ll_register_callbacks cb 0x55da5a42a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2984c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4d23b0 0x55da5a4de130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2977400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cd240 0x55da5a4d70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2984c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4d23b0 0x55da5a4de130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46512/0 (socket says 172.21.15.116:46512) 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f2984c00640 1 -- 172.21.15.116:0/3563789936 learned_addr learned my addr 172.21.15.116:0/3563789936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.193+0000 7f297fe00640 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4d6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2977400640 1 -- 172.21.15.116:0/3563789936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 msgr2=0x55da5a4d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2977400640 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4d6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2977400640 1 -- 172.21.15.116:0/3563789936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4d23b0 msgr2=0x55da5a4de130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2977400640 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4d23b0 0x55da5a4de130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2977400640 1 -- 172.21.15.116:0/3563789936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da5a147210 con 0x55da5a4cd240 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2977400640 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cd240 0x55da5a4d70f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f296802c180 tx=0x7f29680045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f3434befc15c057 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f296d600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f296d600640 1 -- 172.21.15.116:0/3563789936 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296804b030 con 0x55da5a4cd240 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f296d600640 1 -- 172.21.15.116:0/3563789936 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f29680dc0b0 con 0x55da5a4cd240 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2978004d20 con 0x55da5a4cd240 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f296d600640 1 -- 172.21.15.116:0/3563789936 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29680f5020 con 0x55da5a4cd240 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2989983740 10 client.27668 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.194+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55da5a213620 con 0x55da5a4cd240 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f296d600640 1 -- 172.21.15.116:0/3563789936 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2978004d20 con 0x55da5a4cd240 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f296d600640 1 -- 172.21.15.116:0/3563789936 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f29680a5ce0 con 0x55da5a4cd240 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f2989983740 10 client.27668 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f2989983740 10 client.27668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f2989983740 20 client.27668 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f2989983740 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a4e29b0 0x55da5a4e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f2989983740 4 client.27668 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da5a43f8b0 con 0x55da5a4e29b0 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.195+0000 7f297fe00640 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a4e29b0 0x55da5a4e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.196+0000 7f297fe00640 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a4e29b0 0x55da5a4e4da0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b1f5280de4846f3 server_cookie=c69c078b40f9ab7 in_seq=0 out_seq=0 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.196+0000 7f296d600640 10 client.27668 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.196+0000 7f296d600640 1 -- 172.21.15.116:0/3563789936 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55da5a43f8b0 con 0x55da5a4e29b0 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.196+0000 7f296d600640 10 client.27668 handle_command_reply: tid=0 2024-03-04T07:49:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.196+0000 7f2989983740 1 client.27668 shutdown 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cd240 msgr2=0x55da5a4d70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cd240 0x55da5a4d70f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f296802c180 tx=0x7f29680045d0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 shutdown_connections 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a4e29b0 0x55da5a4e4da0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4d23b0 0x55da5a4de130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cd240 0x55da5a4d70f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f296d600640 0 client.27668 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 --2- 172.21.15.116:0/3563789936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cc850 0x55da5a4d6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 >> 172.21.15.116:0/3563789936 conn(0x55da5a4c7dc0 msgr2=0x55da5a4c6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 shutdown_connections 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 1 -- 172.21.15.116:0/3563789936 wait complete. 2024-03-04T07:49:59.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.197+0000 7f2989983740 20 client.27668 trim_cache size 0 max 16384 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 Processor -- start 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 -- start start 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cc850 0x55da5a4cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cd240 0x55da5a4de0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29801a8ab0 con 0x55da5a4cc850 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29801ab090 con 0x55da5a4d23b0 2024-03-04T07:49:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2989983740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29801a90b0 con 0x55da5a4cd240 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2984c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2977400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cd240 0x55da5a4de0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2984c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4e96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41212/0 (socket says 172.21.15.116:41212) 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2977400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cd240 0x55da5a4de0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53902/0 (socket says 172.21.15.116:53902) 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f2984c00640 1 -- 172.21.15.116:0/2275412989 learned_addr learned my addr 172.21.15.116:0/2275412989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.213+0000 7f297fe00640 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cc850 0x55da5a4cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f2984c00640 1 -- 172.21.15.116:0/2275412989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cd240 msgr2=0x55da5a4de0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f2984c00640 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cd240 0x55da5a4de0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f2984c00640 1 -- 172.21.15.116:0/2275412989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cc850 msgr2=0x55da5a4cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f2984c00640 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cc850 0x55da5a4cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f2984c00640 1 -- 172.21.15.116:0/2275412989 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da5a4c3610 con 0x55da5a4d23b0 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f2977400640 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cd240 0x55da5a4de0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f2984c00640 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4e96a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2978006010 tx=0x7f2978009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=785898211f76e1e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f296fe00640 1 -- 172.21.15.116:0/2275412989 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978016050 con 0x55da5a4d23b0 2024-03-04T07:49:59.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f296fe00640 1 -- 172.21.15.116:0/2275412989 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2978004c00 con 0x55da5a4d23b0 2024-03-04T07:49:59.216 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.214+0000 7f296fe00640 1 -- 172.21.15.116:0/2275412989 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978012740 con 0x55da5a4d23b0 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 -- 172.21.15.116:0/2275412989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 msgr2=0x55da5a4e96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4e96a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2978006010 tx=0x7f2978009b90 comp rx=0 tx=0).stop 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 -- 172.21.15.116:0/2275412989 shutdown_connections 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4d23b0 0x55da5a4e96a0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4cd240 0x55da5a4de0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 --2- 172.21.15.116:0/2275412989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cc850 0x55da5a4cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 -- 172.21.15.116:0/2275412989 >> 172.21.15.116:0/2275412989 conn(0x55da5a4c7dc0 msgr2=0x55da5a4cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 -- 172.21.15.116:0/2275412989 shutdown_connections 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 -- 172.21.15.116:0/2275412989 wait complete. 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.215+0000 7f2989983740 1 -- start start 2024-03-04T07:49:59.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 1 Processor -- start 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cc850 0x55da5a4dbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4dc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4d23b0 0x55da5a4dd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29801a9710 con 0x55da5a4cd240 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29801aaa30 con 0x55da5a4cc850 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29801a9d70 con 0x55da5a4d23b0 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2989983740 10 client.0 _ll_register_callbacks cb 0x7f29680b8330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2977400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4dc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f297fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cc850 0x55da5a4dbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f297fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cc850 0x55da5a4dbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41214/0 (socket says 172.21.15.116:41214) 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f297fe00640 1 -- 172.21.15.116:0/887539178 learned_addr learned my addr 172.21.15.116:0/887539178 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.216+0000 7f2984c00640 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4d23b0 0x55da5a4dd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f297fe00640 1 -- 172.21.15.116:0/887539178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4d23b0 msgr2=0x55da5a4dd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f297fe00640 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4d23b0 0x55da5a4dd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f297fe00640 1 -- 172.21.15.116:0/887539178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 msgr2=0x55da5a4dc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f297fe00640 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4dc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f297fe00640 1 -- 172.21.15.116:0/887539178 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da5a4dd6e0 con 0x55da5a4cc850 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f297fe00640 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cc850 0x55da5a4dbba0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x55da5a4cdcb0 tx=0x7f29700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fcfa58cb3bdc683 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f296d600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f296d600640 1 -- 172.21.15.116:0/887539178 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297001eba0 con 0x55da5a4cc850 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f296d600640 1 -- 172.21.15.116:0/887539178 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f297001ed40 con 0x55da5a4cc850 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.217+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2978004c00 con 0x55da5a4cc850 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f296d600640 1 -- 172.21.15.116:0/887539178 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f297002a680 con 0x55da5a4cc850 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 10 client.27674 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55da5a53e6a0 con 0x55da5a4cc850 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f296d600640 1 -- 172.21.15.116:0/887539178 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2978004c00 con 0x55da5a4cc850 2024-03-04T07:49:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f296d600640 1 -- 172.21.15.116:0/887539178 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f29700338b0 con 0x55da5a4cc850 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 10 client.27674 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 10 client.27674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 20 client.27674 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a53f020 0x55da5a541440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 4 client.27674 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55da5a4d69a0 con 0x55da5a53f020 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.218+0000 7f2977400640 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a53f020 0x55da5a541440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.219+0000 7f2977400640 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a53f020 0x55da5a541440 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8789bf86ae4080 server_cookie=92d592db3be5281c in_seq=0 out_seq=0 2024-03-04T07:49:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.219+0000 7f296d600640 10 client.27674 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f296d600640 1 -- 172.21.15.116:0/887539178 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55da5a4d69a0 con 0x55da5a53f020 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f296d600640 10 client.27674 handle_command_reply: tid=0 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 client.27674 shutdown 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cc850 msgr2=0x55da5a4dbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cc850 0x55da5a4dbba0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x55da5a4cdcb0 tx=0x7f29700046c0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 shutdown_connections 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55da5a53f020 0x55da5a541440 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55da5a4d23b0 0x55da5a4dd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da5a4cd240 0x55da5a4dc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 --2- 172.21.15.116:0/887539178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55da5a4cc850 0x55da5a4dbba0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f296d600640 0 client.27674 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 >> 172.21.15.116:0/887539178 conn(0x55da5a4c7dc0 msgr2=0x55da5a4cb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 shutdown_connections 2024-03-04T07:49:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.233+0000 7f2989983740 1 -- 172.21.15.116:0/887539178 wait complete. 2024-03-04T07:49:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2989983740 20 client.27674 trim_cache size 0 max 16384 2024-03-04T07:49:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:49:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:49:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:49:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:49:59.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29640a1ea0 msgr2=0x7f29640a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29640a1ea0 0x7f29640a4360 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x55da5a4cd640 tx=0x7f297000b410 comp rx=0 tx=0).stop 2024-03-04T07:49:59.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 msgr2=0x7f298019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f298019a000 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f296802c8f0 tx=0x7f2968005e40 comp rx=0 tx=0).stop 2024-03-04T07:49:59.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 shutdown_connections 2024-03-04T07:49:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29801053e0 0x7f29801a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2980104a20 0x7f298019a000 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f29640a1ea0 0x7f29640a4360 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.234+0000 7f2986c00640 1 --2- 172.21.15.116:0/1476166712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2980103830 0x7f2980199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.235+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 >> 172.21.15.116:0/1476166712 conn(0x7f29800fedc0 msgr2=0x7f2980107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.235+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 shutdown_connections 2024-03-04T07:49:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:49:59.235+0000 7f2986c00640 1 -- 172.21.15.116:0/1476166712 wait complete. 2024-03-04T07:49:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:58 smithi152 ceph-mon[37154]: pgmap v677: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:49:58 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2739847319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:59.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:58 smithi133 ceph-mon[36505]: pgmap v677: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:49:59.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:49:58 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2739847319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:49:59.683 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:49:59.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:49:59.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:49:59.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 -- 172.21.15.116:0/2300853357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 msgr2=0x7fed54104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 --2- 172.21.15.116:0/2300853357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 0x7fed54104e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fed40009860 tx=0x7fed4002bb70 comp rx=0 tx=0).stop 2024-03-04T07:49:59.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 -- 172.21.15.116:0/2300853357 shutdown_connections 2024-03-04T07:49:59.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 --2- 172.21.15.116:0/2300853357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed541053a0 0x7fed5410bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 --2- 172.21.15.116:0/2300853357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 0x7fed54104e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 --2- 172.21.15.116:0/2300853357 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541037f0 0x7fed54103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 -- 172.21.15.116:0/2300853357 >> 172.21.15.116:0/2300853357 conn(0x7fed540fed80 msgr2=0x7fed541011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 -- 172.21.15.116:0/2300853357 shutdown_connections 2024-03-04T07:49:59.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.908+0000 7fed5c200640 1 -- 172.21.15.116:0/2300853357 wait complete. 2024-03-04T07:49:59.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.909+0000 7fed5c200640 1 Processor -- start 2024-03-04T07:49:59.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5c200640 1 -- start start 2024-03-04T07:49:59.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5c200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed541037f0 0x7fed54199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5c200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 0x7fed5419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5c200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541053a0 0x7fed541a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5c200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5410f450 con 0x7fed541037f0 2024-03-04T07:49:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5c200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5410f2d0 con 0x7fed541053a0 2024-03-04T07:49:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5c200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5410f5d0 con 0x7fed541049e0 2024-03-04T07:49:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5a200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541053a0 0x7fed541a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed59800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed541037f0 0x7fed54199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed58e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 0x7fed5419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5a200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541053a0 0x7fed541a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41232/0 (socket says 172.21.15.116:41232) 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.910+0000 7fed5a200640 1 -- 172.21.15.116:0/1825944260 learned_addr learned my addr 172.21.15.116:0/1825944260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5a200640 1 -- 172.21.15.116:0/1825944260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 msgr2=0x7fed5419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5a200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 0x7fed5419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5a200640 1 -- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed541037f0 msgr2=0x7fed54199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5a200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed541037f0 0x7fed54199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5a200640 1 -- 172.21.15.116:0/1825944260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed541a1710 con 0x7fed541053a0 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed58e00640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 0x7fed5419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed59800640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed541037f0 0x7fed54199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:49:59.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5a200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541053a0 0x7fed541a10a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009fd0 tx=0x7fed4c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed52000640 1 -- 172.21.15.116:0/1825944260 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c004b20 con 0x7fed541053a0 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed52000640 1 -- 172.21.15.116:0/1825944260 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed4c003e50 con 0x7fed541053a0 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed52000640 1 -- 172.21.15.116:0/1825944260 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00b630 con 0x7fed541053a0 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed541a1a00 con 0x7fed541053a0 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.911+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed541a1ed0 con 0x7fed541053a0 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.914+0000 7fed52000640 1 -- 172.21.15.116:0/1825944260 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fed4c01d020 con 0x7fed541053a0 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.915+0000 7fed5efbc740 1 Processor -- start 2024-03-04T07:49:59.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.915+0000 7fed5efbc740 1 -- start start 2024-03-04T07:49:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971b0d0 0x563f9971b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971bac0 0x563f997205c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f997230c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed5efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9962ee20 con 0x563f99720c30 2024-03-04T07:49:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed5efbc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f99680a20 con 0x563f9971bac0 2024-03-04T07:49:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed5efbc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f99629b00 con 0x563f9971b0d0 2024-03-04T07:49:59.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971bac0 0x563f997205c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971bac0 0x563f997205c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41234/0 (socket says 172.21.15.116:41234) 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 -- 172.21.15.116:0/3277043306 learned_addr learned my addr 172.21.15.116:0/3277043306 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed58e00640 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f997230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed52000640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed3c0a1d80 0x7fed3c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed58e00640 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971b0d0 0x563f9971b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 -- 172.21.15.116:0/3277043306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971b0d0 msgr2=0x563f9971b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971b0d0 0x563f9971b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 -- 172.21.15.116:0/3277043306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 msgr2=0x563f997230c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f997230c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed59800640 1 -- 172.21.15.116:0/3277043306 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f9963ee10 con 0x563f9971bac0 2024-03-04T07:49:59.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed58e00640 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971b0d0 0x563f9971b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:49:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.916+0000 7fed52000640 1 -- 172.21.15.116:0/1825944260 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed4c0cdd70 con 0x7fed541053a0 2024-03-04T07:49:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.917+0000 7fed5a200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed3c0a1d80 0x7fed3c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.917+0000 7fed59800640 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971bac0 0x563f997205c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fed480114c0 tx=0x7fed48011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9616ef7d1f574f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.917+0000 7fed46a00640 1 -- 172.21.15.116:0/3277043306 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48004020 con 0x563f9971bac0 2024-03-04T07:49:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.917+0000 7fed46a00640 1 -- 172.21.15.116:0/3277043306 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed48005500 con 0x563f9971bac0 2024-03-04T07:49:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.917+0000 7fed46a00640 1 -- 172.21.15.116:0/3277043306 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48004e10 con 0x563f9971bac0 2024-03-04T07:49:59.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 -- 172.21.15.116:0/3277043306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971bac0 msgr2=0x563f997205c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5a200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed3c0a1d80 0x7fed3c0a4240 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fed4c01c040 tx=0x7fed4c004170 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971bac0 0x563f997205c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fed480114c0 tx=0x7fed48011980 comp rx=0 tx=0).stop 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 -- 172.21.15.116:0/3277043306 shutdown_connections 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f997230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971bac0 0x563f997205c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3277043306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971b0d0 0x563f9971b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 -- 172.21.15.116:0/3277043306 >> 172.21.15.116:0/3277043306 conn(0x563f99716640 msgr2=0x563f99716a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 -- 172.21.15.116:0/3277043306 shutdown_connections 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 -- 172.21.15.116:0/3277043306 wait complete. 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.918+0000 7fed5efbc740 1 -- start start 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 1 Processor -- start 2024-03-04T07:49:59.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971b0d0 0x563f997254c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f99725a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f9972c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9963cf40 con 0x563f99720c30 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9967cac0 con 0x563f9971b0d0 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f99723be0 con 0x563f9971bac0 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed5efbc740 10 client.0 _ll_register_callbacks cb 0x563f9967baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed58e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971b0d0 0x563f997254c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed58e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f9972c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed58e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971b0d0 0x563f997254c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41240/0 (socket says 172.21.15.116:41240) 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed58e00640 1 -- 172.21.15.116:0/3839635506 learned_addr learned my addr 172.21.15.116:0/3839635506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.919+0000 7fed59800640 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f99725a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed58e00640 1 -- 172.21.15.116:0/3839635506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 msgr2=0x563f99725a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed58e00640 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f99725a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed58e00640 1 -- 172.21.15.116:0/3839635506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 msgr2=0x563f9972c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed58e00640 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f9972c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed58e00640 1 -- 172.21.15.116:0/3839635506 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f9945d7e0 con 0x563f9971b0d0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed58e00640 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971b0d0 0x563f997254c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed4002bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be48c1c23c315a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed3b400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed3b400640 1 -- 172.21.15.116:0/3839635506 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40039040 con 0x563f9971b0d0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed3b400640 1 -- 172.21.15.116:0/3839635506 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed4002c710 con 0x563f9971b0d0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed48005500 con 0x563f9971b0d0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.920+0000 7fed3b400640 1 -- 172.21.15.116:0/3839635506 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed40055050 con 0x563f9971b0d0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 10 client.27686 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563f994bdf70 con 0x563f9971b0d0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed3b400640 1 -- 172.21.15.116:0/3839635506 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed48005500 con 0x563f9971b0d0 2024-03-04T07:49:59.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed3b400640 1 -- 172.21.15.116:0/3839635506 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fed4003f070 con 0x563f9971b0d0 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 10 client.27686 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 10 client.27686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 20 client.27686 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f997311b0 0x563f99733610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 4 client.27686 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f9968df60 con 0x563f997311b0 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.921+0000 7fed59800640 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f997311b0 0x563f99733610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.922+0000 7fed59800640 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f997311b0 0x563f99733610 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d9c41d464dc1b22 server_cookie=c21334c8ed933304 in_seq=0 out_seq=0 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.922+0000 7fed3b400640 10 client.27686 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed3b400640 1 -- 172.21.15.116:0/3839635506 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563f9968df60 con 0x563f997311b0 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed3b400640 10 client.27686 handle_command_reply: tid=0 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 client.27686 shutdown 2024-03-04T07:49:59.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971b0d0 msgr2=0x563f997254c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971b0d0 0x563f997254c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed4002bb70 comp rx=0 tx=0).stop 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 shutdown_connections 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f997311b0 0x563f99733610 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f99720c30 0x563f9972c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f99725a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 --2- 172.21.15.116:0/3839635506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f9971b0d0 0x563f997254c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed3b400640 0 client.27686 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 >> 172.21.15.116:0/3839635506 conn(0x563f99716640 msgr2=0x563f99717890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 shutdown_connections 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 1 -- 172.21.15.116:0/3839635506 wait complete. 2024-03-04T07:49:59.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.923+0000 7fed5efbc740 20 client.27686 trim_cache size 0 max 16384 2024-03-04T07:49:59.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.940+0000 7fed5efbc740 1 Processor -- start 2024-03-04T07:49:59.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed5efbc740 1 -- start start 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f99723fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f99724520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed5efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541a8bb0 con 0x563f9971b0d0 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed5efbc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541ab190 con 0x563f99720c30 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed5efbc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541a91b0 con 0x563f9971bac0 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed58e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f99723fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed58e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f99723fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46578/0 (socket says 172.21.15.116:46578) 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed58e00640 1 -- 172.21.15.116:0/731984600 learned_addr learned my addr 172.21.15.116:0/731984600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed59800640 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.941+0000 7fed58e00640 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f99724520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed58e00640 1 -- 172.21.15.116:0/731984600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 msgr2=0x563f9972c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed58e00640 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed58e00640 1 -- 172.21.15.116:0/731984600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 msgr2=0x563f99724520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed58e00640 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f99724520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed58e00640 1 -- 172.21.15.116:0/731984600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f99355170 con 0x563f9971b0d0 2024-03-04T07:49:59.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed58e00640 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f99723fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fed40009250 tx=0x7fed400057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884c0f03d1ad78dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed46000640 1 -- 172.21.15.116:0/731984600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40039040 con 0x563f9971b0d0 2024-03-04T07:49:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed46000640 1 -- 172.21.15.116:0/731984600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed400044d0 con 0x563f9971b0d0 2024-03-04T07:49:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.942+0000 7fed46000640 1 -- 172.21.15.116:0/731984600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40000e60 con 0x563f9971b0d0 2024-03-04T07:49:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 -- 172.21.15.116:0/731984600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 msgr2=0x563f99723fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f99723fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fed40009250 tx=0x7fed400057c0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 -- 172.21.15.116:0/731984600 shutdown_connections 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f99724520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 --2- 172.21.15.116:0/731984600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f99723fe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 -- 172.21.15.116:0/731984600 >> 172.21.15.116:0/731984600 conn(0x563f99716640 msgr2=0x563f9971a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 -- 172.21.15.116:0/731984600 shutdown_connections 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 -- 172.21.15.116:0/731984600 wait complete. 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 -- start start 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.943+0000 7fed5efbc740 1 Processor -- start 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f9972aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed5efbc740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f9972bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed5efbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541a9810 con 0x563f9971b0d0 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed5efbc740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541aab30 con 0x563f99720c30 2024-03-04T07:49:59.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed5efbc740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541a9e70 con 0x563f9971bac0 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed5efbc740 10 client.0 _ll_register_callbacks cb 0x7fed4c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed59800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed58e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f9972aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed59800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53950/0 (socket says 172.21.15.116:53950) 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed59800640 1 -- 172.21.15.116:0/2531331673 learned_addr learned my addr 172.21.15.116:0/2531331673 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed58e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f9972aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46592/0 (socket says 172.21.15.116:46592) 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.944+0000 7fed58e00640 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f9972bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed59800640 1 -- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 msgr2=0x563f9972bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed59800640 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f9972bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed59800640 1 -- 172.21.15.116:0/2531331673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 msgr2=0x563f9972aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed59800640 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f9972aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed59800640 1 -- 172.21.15.116:0/2531331673 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f9972c5a0 con 0x563f9971bac0 2024-03-04T07:49:59.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed59800640 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972aff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fed48011e80 tx=0x7fed480096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c24f8c26dfb98e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed3b400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed3b400640 1 -- 172.21.15.116:0/2531331673 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4801d040 con 0x563f9971bac0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed400044d0 con 0x563f9971bac0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.945+0000 7fed3b400640 1 -- 172.21.15.116:0/2531331673 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fed48011020 con 0x563f9971bac0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed3b400640 1 -- 172.21.15.116:0/2531331673 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed48011220 con 0x563f9971bac0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 10 client.27568 fetch_fsmap learned FSMap version 19 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563f99795cb0 con 0x563f9971bac0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed3b400640 1 -- 172.21.15.116:0/2531331673 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed400044d0 con 0x563f9971bac0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed3b400640 1 -- 172.21.15.116:0/2531331673 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fed48026860 con 0x563f9971bac0 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 10 client.27568 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 10 client.27568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 20 client.27568 populate_metadata read hostname 'smithi116' 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f99796520 0x563f99798940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:49:59.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 4 client.27568 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:49:59.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563f996882b0 con 0x563f99796520 2024-03-04T07:49:59.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.946+0000 7fed58e00640 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f99796520 0x563f99798940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:49:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.947+0000 7fed58e00640 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f99796520 0x563f99798940 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc39a34367378a46 server_cookie=8b4d3950c8a91d33 in_seq=0 out_seq=0 2024-03-04T07:49:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.947+0000 7fed3b400640 10 client.27568 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed3b400640 1 -- 172.21.15.116:0/2531331673 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563f996882b0 con 0x563f99796520 2024-03-04T07:49:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed3b400640 10 client.27568 handle_command_reply: tid=0 2024-03-04T07:49:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 client.27568 shutdown 2024-03-04T07:49:59.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 msgr2=0x563f9972aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972aff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fed48011e80 tx=0x7fed480096c0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 shutdown_connections 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563f99796520 0x563f99798940 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563f99720c30 0x563f9972bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f9971bac0 0x563f9972aff0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed3b400640 0 client.27568 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 --2- 172.21.15.116:0/2531331673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f9971b0d0 0x563f9972aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 >> 172.21.15.116:0/2531331673 conn(0x563f99716640 msgr2=0x563f997151d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 shutdown_connections 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 1 -- 172.21.15.116:0/2531331673 wait complete. 2024-03-04T07:49:59.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.948+0000 7fed5efbc740 20 client.27568 trim_cache size 0 max 16384 2024-03-04T07:49:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:49:59.953 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:49:59.953 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "0ff5a084-b1e8-45c9-91a2-c02cf4b6a667", 2024-03-04T07:49:59.953 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:49:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:49:59.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed3c0a1d80 msgr2=0x7fed3c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed3c0a1d80 0x7fed3c0a4240 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fed4c01c040 tx=0x7fed4c004170 comp rx=0 tx=0).stop 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541053a0 msgr2=0x7fed541a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541053a0 0x7fed541a10a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009fd0 tx=0x7fed4c00c800 comp rx=0 tx=0).stop 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 shutdown_connections 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fed541053a0 0x7fed541a10a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fed3c0a1d80 0x7fed3c0a4240 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed541049e0 0x7fed5419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 --2- 172.21.15.116:0/1825944260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed541037f0 0x7fed54199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 >> 172.21.15.116:0/1825944260 conn(0x7fed540fed80 msgr2=0x7fed54107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:49:59.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 shutdown_connections 2024-03-04T07:49:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:49:59.949+0000 7fed5c200640 1 -- 172.21.15.116:0/1825944260 wait complete. 2024-03-04T07:49:59.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:49:59.968 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:50:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 -- 172.21.15.116:0/2014448710 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 msgr2=0x7fcacc103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 --2- 172.21.15.116:0/2014448710 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc103c30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009ca0 tx=0x7fcac402c0f0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 -- 172.21.15.116:0/2014448710 shutdown_connections 2024-03-04T07:50:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 --2- 172.21.15.116:0/2014448710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcacc1053c0 0x7fcacc10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 --2- 172.21.15.116:0/2014448710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcacc104a00 0x7fcacc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 --2- 172.21.15.116:0/2014448710 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc103c30 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 -- 172.21.15.116:0/2014448710 >> 172.21.15.116:0/2014448710 conn(0x7fcacc0fed80 msgr2=0x7fcacc1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 -- 172.21.15.116:0/2014448710 shutdown_connections 2024-03-04T07:50:00.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.180+0000 7fcad3000640 1 -- 172.21.15.116:0/2014448710 wait complete. 2024-03-04T07:50:00.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.181+0000 7fcad3000640 1 Processor -- start 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad3000640 1 -- start start 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad3000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad3000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcacc104a00 0x7fcacc19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad3000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcacc1053c0 0x7fcacc1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad3000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc10f460 con 0x7fcacc1053c0 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad3000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc10f2e0 con 0x7fcacc103810 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad3000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc10f5e0 con 0x7fcacc104a00 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcacbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcad1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcacc1053c0 0x7fcacc1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcacbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41278/0 (socket says 172.21.15.116:41278) 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcacbe00640 1 -- 172.21.15.116:0/19039233 learned_addr learned my addr 172.21.15.116:0/19039233 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.182+0000 7fcacb400640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcacc104a00 0x7fcacc19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcacbe00640 1 -- 172.21.15.116:0/19039233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcacc104a00 msgr2=0x7fcacc19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcacbe00640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcacc104a00 0x7fcacc19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcacbe00640 1 -- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcacc1053c0 msgr2=0x7fcacc1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcacbe00640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcacc1053c0 0x7fcacc1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcacbe00640 1 -- 172.21.15.116:0/19039233 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc1a1750 con 0x7fcacc103810 2024-03-04T07:50:00.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcacbe00640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc199b30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcac4000f80 tx=0x7fcac4032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcac8c00640 1 -- 172.21.15.116:0/19039233 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4041020 con 0x7fcacc103810 2024-03-04T07:50:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcac8c00640 1 -- 172.21.15.116:0/19039233 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcac4034cd0 con 0x7fcacc103810 2024-03-04T07:50:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcacc1a19e0 con 0x7fcacc103810 2024-03-04T07:50:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcac8c00640 1 -- 172.21.15.116:0/19039233 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4039040 con 0x7fcacc103810 2024-03-04T07:50:00.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.183+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcacc1a1ef0 con 0x7fcacc103810 2024-03-04T07:50:00.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.186+0000 7fcac8c00640 1 -- 172.21.15.116:0/19039233 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fcac4044020 con 0x7fcacc103810 2024-03-04T07:50:00.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.186+0000 7fcad5e29740 1 Processor -- start 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad5e29740 1 -- start start 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b0df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b13040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b136b0 0x556560b15b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad5e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556560a3b090 con 0x556560b0e540 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad5e29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556560a72d30 con 0x556560b136b0 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad5e29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556560a1ba30 con 0x556560b0db50 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcacbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b13040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcad1000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b0df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcacb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b136b0 0x556560b15b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcacb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b136b0 0x556560b15b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41284/0 (socket says 172.21.15.116:41284) 2024-03-04T07:50:00.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.187+0000 7fcacb400640 1 -- 172.21.15.116:0/541102828 learned_addr learned my addr 172.21.15.116:0/541102828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcacb400640 1 -- 172.21.15.116:0/541102828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 msgr2=0x556560b0df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcacb400640 1 --2- 172.21.15.116:0/541102828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b0df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcacb400640 1 -- 172.21.15.116:0/541102828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 msgr2=0x556560b13040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcacb400640 1 --2- 172.21.15.116:0/541102828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b13040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcacb400640 1 -- 172.21.15.116:0/541102828 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565608517d0 con 0x556560b136b0 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcac8c00640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcaac0a1e00 0x7fcaac0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcacb400640 1 --2- 172.21.15.116:0/541102828 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b136b0 0x556560b15b40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcab4009c20 tx=0x7fcab40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7e83c5480047e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:00.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcac8c00640 1 -- 172.21.15.116:0/19039233 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcac40efc30 con 0x7fcacc103810 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcad1000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcaac0a1e00 0x7fcaac0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcac1600640 1 -- 172.21.15.116:0/541102828 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4012450 con 0x556560b136b0 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcac1600640 1 -- 172.21.15.116:0/541102828 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab4012ad0 con 0x556560b136b0 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.188+0000 7fcac1600640 1 -- 172.21.15.116:0/541102828 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab400fbf0 con 0x556560b136b0 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 -- 172.21.15.116:0/541102828 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b136b0 msgr2=0x556560b15b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad1000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcaac0a1e00 0x7fcaac0a42c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x556560b0eda0 tx=0x7fcabc00b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 --2- 172.21.15.116:0/541102828 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b136b0 0x556560b15b40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcab4009c20 tx=0x7fcab40117b0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 -- 172.21.15.116:0/541102828 shutdown_connections 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 --2- 172.21.15.116:0/541102828 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b136b0 0x556560b15b40 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 --2- 172.21.15.116:0/541102828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b13040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 --2- 172.21.15.116:0/541102828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b0df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 -- 172.21.15.116:0/541102828 >> 172.21.15.116:0/541102828 conn(0x556560b090c0 msgr2=0x556560b094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:00.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 -- 172.21.15.116:0/541102828 shutdown_connections 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.189+0000 7fcad5e29740 1 -- 172.21.15.116:0/541102828 wait complete. 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 -- start start 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 Processor -- start 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b17ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0e540 0x556560b18410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 0x556560b1f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556560a110c0 con 0x556560b136b0 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556560a6edd0 con 0x556560b0db50 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556560b16660 con 0x556560b0e540 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad5e29740 10 client.0 _ll_register_callbacks cb 0x556560a2e860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcacbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b17ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.190+0000 7fcad1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 0x556560b1f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b17ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41298/0 (socket says 172.21.15.116:41298) 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 -- 172.21.15.116:0/2486682380 learned_addr learned my addr 172.21.15.116:0/2486682380 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:00.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacb400640 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0e540 0x556560b18410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 -- 172.21.15.116:0/2486682380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0e540 msgr2=0x556560b18410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0e540 0x556560b18410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 -- 172.21.15.116:0/2486682380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 msgr2=0x556560b1f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 0x556560b1f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 -- 172.21.15.116:0/2486682380 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565607873f0 con 0x556560b0db50 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcacbe00640 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b17ed0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcac4006010 tx=0x7fcac40f3260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4286d110b7f52b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcaba000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcaba000640 1 -- 172.21.15.116:0/2486682380 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac40b8bc0 con 0x556560b0db50 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcaba000640 1 -- 172.21.15.116:0/2486682380 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcac40c4c30 con 0x556560b0db50 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.191+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcab4012ad0 con 0x556560b0db50 2024-03-04T07:50:00.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcaba000640 1 -- 172.21.15.116:0/2486682380 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcac40bc280 con 0x556560b0db50 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 10 client.27704 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5565607470f0 con 0x556560b0db50 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcaba000640 1 -- 172.21.15.116:0/2486682380 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab4012ad0 con 0x556560b0db50 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcaba000640 1 -- 172.21.15.116:0/2486682380 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcac403c030 con 0x556560b0db50 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 10 client.27704 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 10 client.27704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 20 client.27704 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b23860 0x556560b25cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 4 client.27704 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556560a80270 con 0x556560b23860 2024-03-04T07:50:00.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.192+0000 7fcacb400640 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b23860 0x556560b25cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.193+0000 7fcacb400640 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b23860 0x556560b25cd0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a937de1c80d18481 server_cookie=9b47d7b2a6275d91 in_seq=0 out_seq=0 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.193+0000 7fcaba000640 10 client.27704 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcaba000640 1 -- 172.21.15.116:0/2486682380 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556560a80270 con 0x556560b23860 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcaba000640 10 client.27704 handle_command_reply: tid=0 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 client.27704 shutdown 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 msgr2=0x556560b17ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b17ed0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcac4006010 tx=0x7fcac40f3260 comp rx=0 tx=0).stop 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 shutdown_connections 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b23860 0x556560b25cd0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 0x556560b1f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0e540 0x556560b18410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcaba000640 0 client.27704 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2486682380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b17ed0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 >> 172.21.15.116:0/2486682380 conn(0x556560b090c0 msgr2=0x556560b08360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 shutdown_connections 2024-03-04T07:50:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 1 -- 172.21.15.116:0/2486682380 wait complete. 2024-03-04T07:50:00.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.194+0000 7fcad5e29740 20 client.27704 trim_cache size 0 max 16384 2024-03-04T07:50:00.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.212+0000 7fcad5e29740 1 Processor -- start 2024-03-04T07:50:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.212+0000 7fcad5e29740 1 -- start start 2024-03-04T07:50:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.212+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b18370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.212+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0e540 0x556560b1f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 0x556560b0ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcad5e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1a8cd0 con 0x556560b136b0 2024-03-04T07:50:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcad5e29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1ab2b0 con 0x556560b0e540 2024-03-04T07:50:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcad5e29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1a92d0 con 0x556560b0db50 2024-03-04T07:50:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0e540 0x556560b1f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0e540 0x556560b1f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41312/0 (socket says 172.21.15.116:41312) 2024-03-04T07:50:00.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 -- 172.21.15.116:0/2262757120 learned_addr learned my addr 172.21.15.116:0/2262757120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacbe00640 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b18370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 -- 172.21.15.116:0/2262757120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 msgr2=0x556560b18370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b18370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 -- 172.21.15.116:0/2262757120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 msgr2=0x556560b0ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 0x556560b0ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacb400640 1 -- 172.21.15.116:0/2262757120 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556560b0e920 con 0x556560b0e540 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.213+0000 7fcacbe00640 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b18370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.214+0000 7fcacb400640 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0e540 0x556560b1f410 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcab4005fd0 tx=0x7fcab4009420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae9d7ed0116dd86d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.214+0000 7fcac0c00640 1 -- 172.21.15.116:0/2262757120 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4009e70 con 0x556560b0e540 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.214+0000 7fcac0c00640 1 -- 172.21.15.116:0/2262757120 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab402f070 con 0x556560b0e540 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.214+0000 7fcac0c00640 1 -- 172.21.15.116:0/2262757120 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4029e00 con 0x556560b0e540 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.214+0000 7fcad5e29740 1 -- 172.21.15.116:0/2262757120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0e540 msgr2=0x556560b1f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.214+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0e540 0x556560b1f410 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcab4005fd0 tx=0x7fcab4009420 comp rx=0 tx=0).stop 2024-03-04T07:50:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 -- 172.21.15.116:0/2262757120 shutdown_connections 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b136b0 0x556560b0ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0e540 0x556560b1f410 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 --2- 172.21.15.116:0/2262757120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b0db50 0x556560b18370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 -- 172.21.15.116:0/2262757120 >> 172.21.15.116:0/2262757120 conn(0x556560b090c0 msgr2=0x556560b0c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 -- 172.21.15.116:0/2262757120 shutdown_connections 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 -- 172.21.15.116:0/2262757120 wait complete. 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 -- start start 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.215+0000 7fcad5e29740 1 Processor -- start 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b1ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b1f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad5e29740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b136b0 0x556560b1d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad5e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1a9930 con 0x556560b0e540 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad5e29740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1aac50 con 0x556560b0db50 2024-03-04T07:50:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad5e29740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1a9f90 con 0x556560b136b0 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad5e29740 10 client.0 _ll_register_callbacks cb 0x7fcac4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b1ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b1ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41320/0 (socket says 172.21.15.116:41320) 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacb400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b1f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 -- 172.21.15.116:0/800207451 learned_addr learned my addr 172.21.15.116:0/800207451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad1000640 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b136b0 0x556560b1d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 -- 172.21.15.116:0/800207451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b136b0 msgr2=0x556560b1d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b136b0 0x556560b1d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 -- 172.21.15.116:0/800207451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 msgr2=0x556560b1f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b1f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 -- 172.21.15.116:0/800207451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556560b1dd90 con 0x556560b0db50 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcad1000640 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b136b0 0x556560b1d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcacbe00640 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b1ee60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcac402c700 tx=0x7fcac40c4e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8d335afe18a55d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.216+0000 7fcaba000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcaba000640 1 -- 172.21.15.116:0/800207451 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac40bc460 con 0x556560b0db50 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcaba000640 1 -- 172.21.15.116:0/800207451 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcac40eddb0 con 0x556560b0db50 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcab402f070 con 0x556560b0db50 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcaba000640 1 -- 172.21.15.116:0/800207451 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcac40bc6f0 con 0x556560b0db50 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 10 client.27710 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556560b7fc80 con 0x556560b0db50 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcaba000640 1 -- 172.21.15.116:0/800207451 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcab402f070 con 0x556560b0db50 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcaba000640 1 -- 172.21.15.116:0/800207451 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fcac40a7070 con 0x556560b0db50 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 10 client.27710 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 10 client.27710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 20 client.27710 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b801b0 0x556560b825d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 4 client.27710 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:50:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556560a7a5c0 con 0x556560b801b0 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.217+0000 7fcacb400640 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b801b0 0x556560b825d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcacb400640 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b801b0 0x556560b825d0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebe78eeeaf17208e server_cookie=3d9eabe101d2948b in_seq=0 out_seq=0 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcaba000640 10 client.27710 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcaba000640 1 -- 172.21.15.116:0/800207451 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556560a7a5c0 con 0x556560b801b0 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcaba000640 10 client.27710 handle_command_reply: tid=0 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 client.27710 shutdown 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 msgr2=0x556560b1ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b1ee60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcac402c700 tx=0x7fcac40c4e60 comp rx=0 tx=0).stop 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 shutdown_connections 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556560b801b0 0x556560b825d0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556560b136b0 0x556560b1d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556560b0e540 0x556560b1f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 --2- 172.21.15.116:0/800207451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556560b0db50 0x556560b1ee60 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcaba000640 0 client.27710 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 >> 172.21.15.116:0/800207451 conn(0x556560b090c0 msgr2=0x556560b0cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.218+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 shutdown_connections 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad5e29740 1 -- 172.21.15.116:0/800207451 wait complete. 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad5e29740 20 client.27710 trim_cache size 0 max 16384 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stdout: "0ff5a084-b1e8-45c9-91a2-c02cf4b6a667": { 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:50:00.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "0ff5a084-b1e8-45c9-91a2-c02cf4b6a667", 2024-03-04T07:50:00.226 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:50:00.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:50:00.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:50:00.226 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcaac0a1e00 msgr2=0x7fcaac0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcaac0a1e00 0x7fcaac0a42c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x556560b0eda0 tx=0x7fcabc00b2e0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 msgr2=0x7fcacc199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc199b30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fcac4000f80 tx=0x7fcac4032ec0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 shutdown_connections 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcacc1053c0 0x7fcacc1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fcaac0a1e00 0x7fcaac0a42c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcacc104a00 0x7fcacc19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 --2- 172.21.15.116:0/19039233 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcacc103810 0x7fcacc199b30 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 >> 172.21.15.116:0/19039233 conn(0x7fcacc0fed80 msgr2=0x7fcacc107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 shutdown_connections 2024-03-04T07:50:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:00.219+0000 7fcad3000640 1 -- 172.21.15.116:0/19039233 wait complete. 2024-03-04T07:50:00.237 INFO:tasks.cephfs.filesystem:scrub status for tag:0ff5a084-b1e8-45c9-91a2-c02cf4b6a667 - {'path': '/', 'tag': '0ff5a084-b1e8-45c9-91a2-c02cf4b6a667', 'options': 'recursive,force'} 2024-03-04T07:50:01.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:00 smithi116 ceph-mon[32150]: pgmap v678: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:01.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:00 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:50:01.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:00 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:01.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:00 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:50:01.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:00 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:01.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:00 smithi116 ceph-mon[32150]: overall HEALTH_OK 2024-03-04T07:50:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:00 smithi152 ceph-mon[37154]: pgmap v678: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:00 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:50:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:00 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:00 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:50:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:00 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:00 smithi152 ceph-mon[37154]: overall HEALTH_OK 2024-03-04T07:50:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:00 smithi133 ceph-mon[36505]: pgmap v678: 97 pgs: 97 active+clean; 2.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:00 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:50:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:00 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:00 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:50:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:00 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:00 smithi133 ceph-mon[36505]: overall HEALTH_OK 2024-03-04T07:50:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:02 smithi116 ceph-mon[32150]: pgmap v679: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-03-04T07:50:03.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:02 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:50:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:02 smithi152 ceph-mon[37154]: pgmap v679: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-03-04T07:50:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:02 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:50:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:02 smithi133 ceph-mon[36505]: pgmap v679: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 KiB/s wr, 15 op/s 2024-03-04T07:50:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:02 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:50:05.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:04 smithi116 ceph-mon[32150]: pgmap v680: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:50:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:04 smithi152 ceph-mon[37154]: pgmap v680: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:50:05.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:04 smithi133 ceph-mon[36505]: pgmap v680: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-03-04T07:50:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:06 smithi116 ceph-mon[32150]: pgmap v681: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-04T07:50:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:06 smithi152 ceph-mon[37154]: pgmap v681: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-04T07:50:07.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:06 smithi133 ceph-mon[36505]: pgmap v681: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-03-04T07:50:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:08 smithi116 ceph-mon[32150]: pgmap v682: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:50:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:08 smithi152 ceph-mon[37154]: pgmap v682: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:50:09.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:08 smithi133 ceph-mon[36505]: pgmap v682: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:50:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:10 smithi116 ceph-mon[32150]: pgmap v683: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:50:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:10 smithi152 ceph-mon[37154]: pgmap v683: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:50:11.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:10 smithi133 ceph-mon[36505]: pgmap v683: 97 pgs: 97 active+clean; 3.0 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.374+0000 7fa6f3400640 1 -- 172.21.15.116:0/231350084 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec075f90 msgr2=0x7fa6ec0763b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.374+0000 7fa6f3400640 1 --2- 172.21.15.116:0/231350084 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec075f90 0x7fa6ec0763b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009900 tx=0x7fa6e002bc30 comp rx=0 tx=0).stop 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.374+0000 7fa6f3400640 1 -- 172.21.15.116:0/231350084 shutdown_connections 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.374+0000 7fa6f3400640 1 --2- 172.21.15.116:0/231350084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ec077b70 0x7fa6ec078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.374+0000 7fa6f3400640 1 --2- 172.21.15.116:0/231350084 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ec0771d0 0x7fa6ec077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.374+0000 7fa6f3400640 1 --2- 172.21.15.116:0/231350084 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec075f90 0x7fa6ec0763b0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.374+0000 7fa6f3400640 1 -- 172.21.15.116:0/231350084 >> 172.21.15.116:0/231350084 conn(0x7fa6ec0fed90 msgr2=0x7fa6ec1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.375+0000 7fa6f3400640 1 -- 172.21.15.116:0/231350084 shutdown_connections 2024-03-04T07:50:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.375+0000 7fa6f3400640 1 -- 172.21.15.116:0/231350084 wait complete. 2024-03-04T07:50:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.375+0000 7fa6f3400640 1 Processor -- start 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f3400640 1 -- start start 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ec075f90 0x7fa6ec19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ec0771d0 0x7fa6ec19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec077b70 0x7fa6ec1a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f3400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1138f0 con 0x7fa6ec0771d0 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f3400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec113770 con 0x7fa6ec077b70 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f3400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec113a70 con 0x7fa6ec075f90 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ec0771d0 0x7fa6ec19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec077b70 0x7fa6ec1a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec077b70 0x7fa6ec1a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58990/0 (socket says 172.21.15.116:58990) 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f1400640 1 -- 172.21.15.116:0/2635039897 learned_addr learned my addr 172.21.15.116:0/2635039897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.376+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ec075f90 0x7fa6ec19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f1400640 1 -- 172.21.15.116:0/2635039897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ec075f90 msgr2=0x7fa6ec19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f1400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ec075f90 0x7fa6ec19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f1400640 1 -- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ec0771d0 msgr2=0x7fa6ec19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f1400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ec0771d0 0x7fa6ec19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f1400640 1 -- 172.21.15.116:0/2635039897 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec1a5a80 con 0x7fa6ec077b70 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f1400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec077b70 0x7fa6ec1a5410 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400eb10 tx=0x7fa6e400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6e9600640 1 -- 172.21.15.116:0/2635039897 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4004b20 con 0x7fa6ec077b70 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6e9600640 1 -- 172.21.15.116:0/2635039897 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6e4004cc0 con 0x7fa6ec077b70 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec1a5d70 con 0x7fa6ec077b70 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6e9600640 1 -- 172.21.15.116:0/2635039897 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4010760 con 0x7fa6ec077b70 2024-03-04T07:50:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.377+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec1a6210 con 0x7fa6ec077b70 2024-03-04T07:50:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 Processor -- start 2024-03-04T07:50:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6e9600640 1 -- 172.21.15.116:0/2635039897 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa6e4005020 con 0x7fa6ec077b70 2024-03-04T07:50:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 -- start start 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7150cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a71511d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a71514840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a71472370 con 0x563a715123b0 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a7141b550 con 0x563a7150c850 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f619f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a7146e410 con 0x563a7150d240 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.382+0000 7fa6f0a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a71511d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a71511d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:47338/0 (socket says 172.21.15.116:47338) 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1625556039 learned_addr learned my addr 172.21.15.116:0/1625556039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7150cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a71514840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1625556039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 msgr2=0x563a7150cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7150cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1625556039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 msgr2=0x563a71514840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a71514840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1625556039 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a7142f3c0 con 0x563a7150d240 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a71511d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0000c00 tx=0x7fa6e0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5284205f76c39558 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.383+0000 7fa6e9600640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa6d40a1e20 0x7fa6d40a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6e9600640 1 -- 172.21.15.116:0/2635039897 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa6e40ce360 con 0x7fa6ec077b70 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6ddc00640 1 -- 172.21.15.116:0/1625556039 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0045070 con 0x563a7150d240 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6ddc00640 1 -- 172.21.15.116:0/1625556039 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6e0004040 con 0x563a7150d240 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa6d40a1e20 0x7fa6d40a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6ddc00640 1 -- 172.21.15.116:0/1625556039 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0004eb0 con 0x563a7150d240 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6f619f740 1 -- 172.21.15.116:0/1625556039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 msgr2=0x563a71511d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a71511d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0000c00 tx=0x7fa6e0009560 comp rx=0 tx=0).stop 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa6d40a1e20 0x7fa6d40a42e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x563a7150daa0 tx=0x7fa6d8004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6f619f740 1 -- 172.21.15.116:0/1625556039 shutdown_connections 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a71514840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a71511d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1625556039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7150cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.384+0000 7fa6f619f740 1 -- 172.21.15.116:0/1625556039 >> 172.21.15.116:0/1625556039 conn(0x563a71507dc0 msgr2=0x563a715081c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.385+0000 7fa6f619f740 1 -- 172.21.15.116:0/1625556039 shutdown_connections 2024-03-04T07:50:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.385+0000 7fa6f619f740 1 -- 172.21.15.116:0/1625556039 wait complete. 2024-03-04T07:50:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.385+0000 7fa6f619f740 1 -- start start 2024-03-04T07:50:12.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.385+0000 7fa6f619f740 1 Processor -- start 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a71516b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a715170d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7151e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a71515300 con 0x563a7150c850 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f619f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a7142e9b0 con 0x563a715123b0 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f619f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a71515480 con 0x563a7150d240 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f619f740 10 client.0 _ll_register_callbacks cb 0x563a7146a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7151e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a71516b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7151e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59020/0 (socket says 172.21.15.116:59020) 2024-03-04T07:50:12.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 -- 172.21.15.116:0/607777517 learned_addr learned my addr 172.21.15.116:0/607777517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a715170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 -- 172.21.15.116:0/607777517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 msgr2=0x563a715170d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a715170d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 -- 172.21.15.116:0/607777517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 msgr2=0x563a71516b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a71516b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f1400640 1 -- 172.21.15.116:0/607777517 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a71187210 con 0x563a715123b0 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.386+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a71516b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6f1400640 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7151e0a0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4097a60 tx=0x7fa6e409aab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b900f432f73f7ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6d2000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6d2000640 1 -- 172.21.15.116:0/607777517 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e40a35a0 con 0x563a715123b0 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6d2000640 1 -- 172.21.15.116:0/607777517 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa6e4005310 con 0x563a715123b0 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6e0004040 con 0x563a715123b0 2024-03-04T07:50:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6d2000640 1 -- 172.21.15.116:0/607777517 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6e409f1e0 con 0x563a715123b0 2024-03-04T07:50:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6f619f740 10 client.27722 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563a71253620 con 0x563a715123b0 2024-03-04T07:50:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6d2000640 1 -- 172.21.15.116:0/607777517 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6e0004040 con 0x563a715123b0 2024-03-04T07:50:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6d2000640 1 -- 172.21.15.116:0/607777517 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa6e4099f50 con 0x563a715123b0 2024-03-04T07:50:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6f619f740 10 client.27722 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.387+0000 7fa6f619f740 10 client.27722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.388+0000 7fa6f619f740 20 client.27722 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.388+0000 7fa6f619f740 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a71522970 0x563a71524da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.388+0000 7fa6f619f740 4 client.27722 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.388+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a7147f8b0 con 0x563a71522970 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.388+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a71522970 0x563a71524da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.388+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a71522970 0x563a71524da0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b926a5774bdac627 server_cookie=2418f78a35e3d437 in_seq=0 out_seq=0 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.388+0000 7fa6d2000640 10 client.27722 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.389+0000 7fa6d2000640 1 -- 172.21.15.116:0/607777517 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563a7147f8b0 con 0x563a71522970 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.389+0000 7fa6d2000640 10 client.27722 handle_command_reply: tid=0 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.389+0000 7fa6f619f740 1 client.27722 shutdown 2024-03-04T07:50:12.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.389+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 msgr2=0x563a7151e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.389+0000 7fa6f619f740 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7151e0a0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4097a60 tx=0x7fa6e409aab0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.389+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 shutdown_connections 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.389+0000 7fa6f619f740 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a71522970 0x563a71524da0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6f619f740 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7151e0a0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6f619f740 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a715170d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6f619f740 1 --2- 172.21.15.116:0/607777517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a71516b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6d2000640 0 client.27722 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 >> 172.21.15.116:0/607777517 conn(0x563a71507dc0 msgr2=0x563a71506fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 shutdown_connections 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6f619f740 1 -- 172.21.15.116:0/607777517 wait complete. 2024-03-04T07:50:12.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.390+0000 7fa6f619f740 20 client.27722 trim_cache size 0 max 16384 2024-03-04T07:50:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 Processor -- start 2024-03-04T07:50:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 -- start start 2024-03-04T07:50:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a7150b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7150b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1acf00 con 0x563a7150c850 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1af4e0 con 0x563a715123b0 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f619f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1ad500 con 0x563a7150d240 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a7150b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7150b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f0a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a7150b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45458/0 (socket says 172.21.15.116:45458) 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.405+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1507123686 learned_addr learned my addr 172.21.15.116:0/1507123686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1507123686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 msgr2=0x563a7151e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1507123686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 msgr2=0x563a7150b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7150b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6f0a00640 1 -- 172.21.15.116:0/1507123686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a71517420 con 0x563a7150c850 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6f1400640 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7150b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a7150b020 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005ab0 tx=0x7fa6e002df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52544cce6fad3a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6dd200640 1 -- 172.21.15.116:0/1507123686 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0045040 con 0x563a7150c850 2024-03-04T07:50:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6dd200640 1 -- 172.21.15.116:0/1507123686 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6e0004350 con 0x563a7150c850 2024-03-04T07:50:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.406+0000 7fa6dd200640 1 -- 172.21.15.116:0/1507123686 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0005410 con 0x563a7150c850 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 -- 172.21.15.116:0/1507123686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 msgr2=0x563a7150b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a7150b020 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005ab0 tx=0x7fa6e002df90 comp rx=0 tx=0).stop 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 -- 172.21.15.116:0/1507123686 shutdown_connections 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a715123b0 0x563a7150b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 --2- 172.21.15.116:0/1507123686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a7150c850 0x563a7150b020 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 -- 172.21.15.116:0/1507123686 >> 172.21.15.116:0/1507123686 conn(0x563a71507dc0 msgr2=0x563a7150e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 -- 172.21.15.116:0/1507123686 shutdown_connections 2024-03-04T07:50:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 -- 172.21.15.116:0/1507123686 wait complete. 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.407+0000 7fa6f619f740 1 -- start start 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 1 Processor -- start 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7151bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a7151d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1adb60 con 0x563a715123b0 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1aee80 con 0x563a7150c850 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1ae1c0 con 0x563a7150d240 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f619f740 10 client.0 _ll_register_callbacks cb 0x7fa6e40057d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7151bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f0a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7151bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59036/0 (socket says 172.21.15.116:59036) 2024-03-04T07:50:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f0a00640 1 -- 172.21.15.116:0/3895887872 learned_addr learned my addr 172.21.15.116:0/3895887872 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.408+0000 7fa6f1400640 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a7151d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f0a00640 1 -- 172.21.15.116:0/3895887872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 msgr2=0x563a7151c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f0a00640 1 -- 172.21.15.116:0/3895887872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 msgr2=0x563a7151d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a7151d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f0a00640 1 -- 172.21.15.116:0/3895887872 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a7151d880 con 0x563a7150c850 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f1400640 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a7151d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f0a00640 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7151bd90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005ec0 tx=0x7fa6e0004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=482f695b209f55f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6d2000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6d2000640 1 -- 172.21.15.116:0/3895887872 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e005f060 con 0x563a7150c850 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6e0004350 con 0x563a7150c850 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6d2000640 1 -- 172.21.15.116:0/3895887872 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa6e0005190 con 0x563a7150c850 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6d2000640 1 -- 172.21.15.116:0/3895887872 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6e0045040 con 0x563a7150c850 2024-03-04T07:50:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f619f740 10 client.27728 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.409+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x563a7157e870 con 0x563a7150c850 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6d2000640 1 -- 172.21.15.116:0/3895887872 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6e0004350 con 0x563a7150c850 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6d2000640 1 -- 172.21.15.116:0/3895887872 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa6e0064070 con 0x563a7150c850 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6f619f740 10 client.27728 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6f619f740 10 client.27728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6f619f740 20 client.27728 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6f619f740 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a7157ed70 0x563a71581160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6f619f740 4 client.27728 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563a71479c00 con 0x563a7157ed70 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.410+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a7157ed70 0x563a71581160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.411+0000 7fa6ebe00640 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a7157ed70 0x563a71581160 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d277dc00ac3d4205 server_cookie=f1ebeeb9a9603bc in_seq=0 out_seq=0 2024-03-04T07:50:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.411+0000 7fa6d2000640 10 client.27728 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6d2000640 1 -- 172.21.15.116:0/3895887872 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563a71479c00 con 0x563a7157ed70 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6d2000640 10 client.27728 handle_command_reply: tid=0 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 client.27728 shutdown 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 msgr2=0x563a7151bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7151bd90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005ec0 tx=0x7fa6e0004760 comp rx=0 tx=0).stop 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 shutdown_connections 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x563a7157ed70 0x563a71581160 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563a715123b0 0x563a7151d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563a7150d240 0x563a7151c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 --2- 172.21.15.116:0/3895887872 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x563a7150c850 0x563a7151bd90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6d2000640 0 client.27728 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 >> 172.21.15.116:0/3895887872 conn(0x563a71507dc0 msgr2=0x563a7150b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.432+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 shutdown_connections 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.433+0000 7fa6f619f740 1 -- 172.21.15.116:0/3895887872 wait complete. 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.433+0000 7fa6f619f740 20 client.27728 trim_cache size 0 max 16384 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:50:12.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:50:12.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.433+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa6d40a1e20 msgr2=0x7fa6d40a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.433+0000 7fa6f3400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa6d40a1e20 0x7fa6d40a42e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x563a7150daa0 tx=0x7fa6d8004730 comp rx=0 tx=0).stop 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec077b70 msgr2=0x7fa6ec1a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec077b70 0x7fa6ec1a5410 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400eb10 tx=0x7fa6e400c2d0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 shutdown_connections 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa6ec077b70 0x7fa6ec1a5410 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ec0771d0 0x7fa6ec19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa6d40a1e20 0x7fa6d40a42e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 --2- 172.21.15.116:0/2635039897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ec075f90 0x7fa6ec19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 >> 172.21.15.116:0/2635039897 conn(0x7fa6ec0fed90 msgr2=0x7fa6ec100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:12.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 shutdown_connections 2024-03-04T07:50:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:12.434+0000 7fa6f3400640 1 -- 172.21.15.116:0/2635039897 wait complete. 2024-03-04T07:50:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:12 smithi116 ceph-mon[32150]: pgmap v684: 97 pgs: 97 active+clean; 3.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:50:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:12 smithi152 ceph-mon[37154]: pgmap v684: 97 pgs: 97 active+clean; 3.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:50:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:12 smithi133 ceph-mon[36505]: pgmap v684: 97 pgs: 97 active+clean; 3.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-04T07:50:15.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:14 smithi116 ceph-mon[32150]: pgmap v685: 97 pgs: 97 active+clean; 3.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:50:15.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:14 smithi152 ceph-mon[37154]: pgmap v685: 97 pgs: 97 active+clean; 3.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:50:15.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:14 smithi133 ceph-mon[36505]: pgmap v685: 97 pgs: 97 active+clean; 3.1 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:50:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:16 smithi116 ceph-mon[32150]: pgmap v686: 97 pgs: 97 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-04T07:50:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:16 smithi152 ceph-mon[37154]: pgmap v686: 97 pgs: 97 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-04T07:50:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:16 smithi133 ceph-mon[36505]: pgmap v686: 97 pgs: 97 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 11 op/s 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:437/600: open fd = 3 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:438/600: open fd = 3 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:439/600: open fd = 3 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.551 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:440/600: open fd = 3 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:441/600: open fd = 3 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:442/600: open fd = 3 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.552 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:443/600: open fd = 3 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:444/600: open fd = 3 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:445/600: open fd = 3 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:446/600: open fd = 3 2024-03-04T07:50:17.553 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:447/600: open fd = 3 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:448/600: open fd = 3 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:449/600: open fd = 3 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.554 INFO:tasks.workunit.client.0.smithi116.stdout:450/600: open fd = 3 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:451/600: open fd = 3 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:452/600: open fd = 3 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:453/600: open fd = 3 2024-03-04T07:50:17.555 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:454/600: open fd = 3 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:455/600: open fd = 3 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:456/600: open fd = 3 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.556 INFO:tasks.workunit.client.0.smithi116.stdout:457/600: open fd = 3 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:458/600: open fd = 3 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:459/600: open fd = 3 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:460/600: open fd = 3 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.557 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:461/600: open fd = 3 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:462/600: open fd = 3 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:463/600: open fd = 3 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:464/600: open fd = 3 2024-03-04T07:50:17.558 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:465/600: open fd = 3 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:466/600: open fd = 3 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:467/600: open fd = 3 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:468/600: open fd = 3 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:469/600: open fd = 3 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:470/600: open fd = 3 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:471/600: open fd = 3 2024-03-04T07:50:17.560 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:472/600: open fd = 3 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:473/600: open fd = 3 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:474/600: open fd = 3 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.561 INFO:tasks.workunit.client.0.smithi116.stdout:475/600: open fd = 3 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:476/600: open fd = 3 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:477/600: open fd = 3 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:478/600: open fd = 3 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.562 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:479/600: open fd = 3 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:480/600: open fd = 3 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:481/600: open fd = 3 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:482/600: open fd = 3 2024-03-04T07:50:17.563 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:483/600: open fd = 3 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:484/600: open fd = 3 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:485/600: open fd = 3 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:486/600: open fd = 3 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:487/600: open fd = 3 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:488/600: open fd = 3 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:489/600: open fd = 3 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.565 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:490/600: open fd = 3 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:491/600: open fd = 3 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:492/600: open fd = 3 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:493/600: open fd = 3 2024-03-04T07:50:17.566 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:494/600: open fd = 3 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:495/600: open fd = 3 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:496/600: open fd = 3 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.567 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:497/600: open fd = 3 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:498/600: open fd = 3 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:50:17.568 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:50:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:18 smithi116 ceph-mon[32150]: pgmap v687: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:18 smithi152 ceph-mon[37154]: pgmap v687: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:19.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:18 smithi133 ceph-mon[36505]: pgmap v687: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:20 smithi116 ceph-mon[32150]: pgmap v688: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:20 smithi152 ceph-mon[37154]: pgmap v688: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:20 smithi133 ceph-mon[36505]: pgmap v688: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:22 smithi116 ceph-mon[32150]: pgmap v689: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:50:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:22 smithi152 ceph-mon[37154]: pgmap v689: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:50:23.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:22 smithi133 ceph-mon[36505]: pgmap v689: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:50:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:24 smithi116 ceph-mon[32150]: pgmap v690: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:50:25.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:24 smithi152 ceph-mon[37154]: pgmap v690: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:50:25.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:24 smithi133 ceph-mon[36505]: pgmap v690: 97 pgs: 97 active+clean; 3.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-04T07:50:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 -- 172.21.15.116:0/251558151 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96941097e0 msgr2=0x7f9694110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 --2- 172.21.15.116:0/251558151 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96941097e0 0x7f9694110070 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f968c009ca0 tx=0x7f968c02c100 comp rx=0 tx=0).stop 2024-03-04T07:50:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 -- 172.21.15.116:0/251558151 shutdown_connections 2024-03-04T07:50:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 --2- 172.21.15.116:0/251558151 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96941097e0 0x7f9694110070 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 --2- 172.21.15.116:0/251558151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9694108e20 0x7f96941092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 --2- 172.21.15.116:0/251558151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9694107c30 0x7f9694108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 -- 172.21.15.116:0/251558151 >> 172.21.15.116:0/251558151 conn(0x7f9694076be0 msgr2=0x7f9694079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 -- 172.21.15.116:0/251558151 shutdown_connections 2024-03-04T07:50:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.579+0000 7f969a800640 1 -- 172.21.15.116:0/251558151 wait complete. 2024-03-04T07:50:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.580+0000 7f969a800640 1 Processor -- start 2024-03-04T07:50:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.580+0000 7f969a800640 1 -- start start 2024-03-04T07:50:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f969a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9694107c30 0x7f969419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f969a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9694108e20 0x7f969419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f969a800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96941097e0 0x7f96941a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f969a800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941138b0 con 0x7f9694108e20 2024-03-04T07:50:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f969a800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9694113730 con 0x7f9694107c30 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f969a800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9694113a30 con 0x7f96941097e0 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9692a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9694108e20 0x7f969419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9694107c30 0x7f969419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9694107c30 0x7f969419de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49810/0 (socket says 172.21.15.116:49810) 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9692a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9694108e20 0x7f969419e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39068/0 (socket says 172.21.15.116:39068) 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 -- 172.21.15.116:0/1499033770 learned_addr learned my addr 172.21.15.116:0/1499033770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693e00640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96941097e0 0x7f96941a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 -- 172.21.15.116:0/1499033770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96941097e0 msgr2=0x7f96941a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96941097e0 0x7f96941a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 -- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9694108e20 msgr2=0x7f969419e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9694108e20 0x7f969419e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.581+0000 7f9693400640 1 -- 172.21.15.116:0/1499033770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96941a5a80 con 0x7f9694107c30 2024-03-04T07:50:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.582+0000 7f9693400640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9694107c30 0x7f969419de60 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f968400e9c0 tx=0x7f968400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.582+0000 7f968be00640 1 -- 172.21.15.116:0/1499033770 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400cd80 con 0x7f9694107c30 2024-03-04T07:50:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.582+0000 7f968be00640 1 -- 172.21.15.116:0/1499033770 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96840035f0 con 0x7f9694107c30 2024-03-04T07:50:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.582+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96941a5d70 con 0x7f9694107c30 2024-03-04T07:50:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.582+0000 7f968be00640 1 -- 172.21.15.116:0/1499033770 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684010690 con 0x7f9694107c30 2024-03-04T07:50:25.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.582+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96941a6200 con 0x7f9694107c30 2024-03-04T07:50:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.585+0000 7f969d5ab740 1 Processor -- start 2024-03-04T07:50:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.585+0000 7f968be00640 1 -- 172.21.15.116:0/1499033770 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f96840037a0 con 0x7f9694107c30 2024-03-04T07:50:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.585+0000 7f969d5ab740 1 -- start start 2024-03-04T07:50:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.585+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7faa850 0x5610d7faac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fab240 0x5610d7fafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fb2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f969d5ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610d7f10370 con 0x5610d7fb03b0 2024-03-04T07:50:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f969d5ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610d7eb9550 con 0x5610d7fab240 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f969d5ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610d7f0c410 con 0x5610d7faa850 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fab240 0x5610d7fafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9692a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fb2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7faa850 0x5610d7faac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fab240 0x5610d7fafd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49824/0 (socket says 172.21.15.116:49824) 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 -- 172.21.15.116:0/3104375963 learned_addr learned my addr 172.21.15.116:0/3104375963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7faa850 0x5610d7faac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:40282/0 (socket says 172.21.15.116:40282) 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 -- 172.21.15.116:0/3104375963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7faa850 msgr2=0x5610d7faac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 --2- 172.21.15.116:0/3104375963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7faa850 0x5610d7faac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 -- 172.21.15.116:0/3104375963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 msgr2=0x5610d7fb2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 --2- 172.21.15.116:0/3104375963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fb2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.586+0000 7f9693400640 1 -- 172.21.15.116:0/3104375963 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610d7ecd3c0 con 0x5610d7fab240 2024-03-04T07:50:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.587+0000 7f9693400640 1 --2- 172.21.15.116:0/3104375963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fab240 0x5610d7fafd40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f96840d1fd0 tx=0x7f96840d61a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7020ca39312ae62 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.587+0000 7f968be00640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96780a1ea0 0x7f96780a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.587+0000 7f9688c00640 1 -- 172.21.15.116:0/3104375963 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96840e1070 con 0x5610d7fab240 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.587+0000 7f9688c00640 1 -- 172.21.15.116:0/3104375963 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96840d02e0 con 0x5610d7fab240 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.587+0000 7f9688c00640 1 -- 172.21.15.116:0/3104375963 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96840998e0 con 0x5610d7fab240 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.587+0000 7f968be00640 1 -- 172.21.15.116:0/1499033770 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f96840ce0c0 con 0x7f9694107c30 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.587+0000 7f9693e00640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96780a1ea0 0x7f96780a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 -- 172.21.15.116:0/3104375963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fab240 msgr2=0x5610d7fafd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3104375963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fab240 0x5610d7fafd40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f96840d1fd0 tx=0x7f96840d61a0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 -- 172.21.15.116:0/3104375963 shutdown_connections 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3104375963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fb2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3104375963 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fab240 0x5610d7fafd40 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3104375963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7faa850 0x5610d7faac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 -- 172.21.15.116:0/3104375963 >> 172.21.15.116:0/3104375963 conn(0x5610d7fa5dc0 msgr2=0x5610d7fa61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 -- 172.21.15.116:0/3104375963 shutdown_connections 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f9693e00640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96780a1ea0 0x7f96780a4360 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f968c02c640 tx=0x7f968c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 -- 172.21.15.116:0/3104375963 wait complete. 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.588+0000 7f969d5ab740 1 -- start start 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 1 Processor -- start 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fb4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fb50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610d7fb3360 con 0x5610d7fb03b0 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610d7ecc9b0 con 0x5610d7faa850 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610d7fb34e0 con 0x5610d7fab240 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f969d5ab740 10 client.0 _ll_register_callbacks cb 0x5610d7f08340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fb4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fb4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49826/0 (socket says 172.21.15.116:49826) 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f9693e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f9693400640 1 -- 172.21.15.116:0/865546974 learned_addr learned my addr 172.21.15.116:0/865546974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.589+0000 7f9692a00640 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fb50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9693400640 1 -- 172.21.15.116:0/865546974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 msgr2=0x5610d7fb50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9693400640 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fb50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9693400640 1 -- 172.21.15.116:0/865546974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 msgr2=0x5610d7fbc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9693400640 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9693400640 1 -- 172.21.15.116:0/865546974 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610d7c25210 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9693400640 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fb4bb0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f96840a3020 tx=0x7f9684003e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9de77ba31930d547 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9681600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9681600640 1 -- 172.21.15.116:0/865546974 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684098800 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9681600640 1 -- 172.21.15.116:0/865546974 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9684003b10 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96840d02e0 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9681600640 1 -- 172.21.15.116:0/865546974 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9684098a50 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f969d5ab740 10 client.27740 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5610d7cf1620 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.590+0000 7f9681600640 1 -- 172.21.15.116:0/865546974 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96840d02e0 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f9681600640 1 -- 172.21.15.116:0/865546974 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f9684099700 con 0x5610d7faa850 2024-03-04T07:50:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f969d5ab740 10 client.27740 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f969d5ab740 10 client.27740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f969d5ab740 20 client.27740 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f969d5ab740 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d7fc0620 0x5610d7fc2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f969d5ab740 4 client.27740 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610d7f1d8b0 con 0x5610d7fc0620 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.591+0000 7f9692a00640 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d7fc0620 0x5610d7fc2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.592+0000 7f9692a00640 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d7fc0620 0x5610d7fc2a10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b0b9a44414d094a server_cookie=7f77279951b6c9dd in_seq=0 out_seq=0 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.592+0000 7f9681600640 10 client.27740 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.592+0000 7f9681600640 1 -- 172.21.15.116:0/865546974 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5610d7f1d8b0 con 0x5610d7fc0620 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.592+0000 7f9681600640 10 client.27740 handle_command_reply: tid=0 2024-03-04T07:50:25.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.592+0000 7f969d5ab740 1 client.27740 shutdown 2024-03-04T07:50:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.592+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 msgr2=0x5610d7fb4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.592+0000 7f969d5ab740 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fb4bb0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f96840a3020 tx=0x7f9684003e10 comp rx=0 tx=0).stop 2024-03-04T07:50:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 shutdown_connections 2024-03-04T07:50:25.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d7fc0620 0x5610d7fc2a10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fb50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 --2- 172.21.15.116:0/865546974 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fb4bb0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 >> 172.21.15.116:0/865546974 conn(0x5610d7fa5dc0 msgr2=0x5610d7fa4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f9681600640 0 client.27740 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 shutdown_connections 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 1 -- 172.21.15.116:0/865546974 wait complete. 2024-03-04T07:50:25.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.593+0000 7f969d5ab740 20 client.27740 trim_cache size 0 max 16384 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 Processor -- start 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 -- start start 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7faa850 0x5610d7fa4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fa53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941ad090 con 0x5610d7faa850 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941af670 con 0x5610d7fb03b0 2024-03-04T07:50:25.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f969d5ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941ad690 con 0x5610d7fab240 2024-03-04T07:50:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f9693400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7faa850 0x5610d7fa4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f9693e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f9693e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49842/0 (socket says 172.21.15.116:49842) 2024-03-04T07:50:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.608+0000 7f9693e00640 1 -- 172.21.15.116:0/3501631252 learned_addr learned my addr 172.21.15.116:0/3501631252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9692a00640 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fa53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9693e00640 1 -- 172.21.15.116:0/3501631252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 msgr2=0x5610d7fa53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9693e00640 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fa53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9693e00640 1 -- 172.21.15.116:0/3501631252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7faa850 msgr2=0x5610d7fa4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9693e00640 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7faa850 0x5610d7fa4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9693e00640 1 -- 172.21.15.116:0/3501631252 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610d7faa640 con 0x5610d7fb03b0 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9692a00640 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fa53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9693400640 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7faa850 0x5610d7fa4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9693e00640 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc0c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f968c02c710 tx=0x7f968c041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2417a6a9091f502c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9683e00640 1 -- 172.21.15.116:0/3501631252 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c039040 con 0x5610d7fb03b0 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9683e00640 1 -- 172.21.15.116:0/3501631252 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f968c004040 con 0x5610d7fb03b0 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.609+0000 7f9683e00640 1 -- 172.21.15.116:0/3501631252 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c04e7e0 con 0x5610d7fb03b0 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 -- 172.21.15.116:0/3501631252 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fb03b0 msgr2=0x5610d7fbc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc0c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f968c02c710 tx=0x7f968c041000 comp rx=0 tx=0).stop 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 -- 172.21.15.116:0/3501631252 shutdown_connections 2024-03-04T07:50:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7fb03b0 0x5610d7fbc0c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fab240 0x5610d7fa53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 --2- 172.21.15.116:0/3501631252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7faa850 0x5610d7fa4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 -- 172.21.15.116:0/3501631252 >> 172.21.15.116:0/3501631252 conn(0x5610d7fa5dc0 msgr2=0x5610d7fa4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 -- 172.21.15.116:0/3501631252 shutdown_connections 2024-03-04T07:50:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 -- 172.21.15.116:0/3501631252 wait complete. 2024-03-04T07:50:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.610+0000 7f969d5ab740 1 -- start start 2024-03-04T07:50:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 1 Processor -- start 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fbbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fab240 0x5610d7fbc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fb03b0 0x5610d7fba180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941adcf0 con 0x5610d7fab240 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941af010 con 0x5610d7faa850 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941ae350 con 0x5610d7fb03b0 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f969d5ab740 10 client.0 _ll_register_callbacks cb 0x7f96840df070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fbbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f9693400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fbbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:49846/0 (socket says 172.21.15.116:49846) 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f9693400640 1 -- 172.21.15.116:0/1808236337 learned_addr learned my addr 172.21.15.116:0/1808236337 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f9692a00640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fab240 0x5610d7fbc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.611+0000 7f9693e00640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fb03b0 0x5610d7fba180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9693400640 1 -- 172.21.15.116:0/1808236337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fb03b0 msgr2=0x5610d7fba180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9693400640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fb03b0 0x5610d7fba180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9693400640 1 -- 172.21.15.116:0/1808236337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fab240 msgr2=0x5610d7fbc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9693400640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fab240 0x5610d7fbc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9693400640 1 -- 172.21.15.116:0/1808236337 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610d7fba9b0 con 0x5610d7faa850 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9693e00640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fb03b0 0x5610d7fba180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9693400640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fbbb30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f968409ad40 tx=0x7f9684098e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66a67d195e25a5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9681600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9681600640 1 -- 172.21.15.116:0/1808236337 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684098c90 con 0x5610d7faa850 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9681600640 1 -- 172.21.15.116:0/1808236337 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f96840a30c0 con 0x5610d7faa850 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f968c004040 con 0x5610d7faa850 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9681600640 1 -- 172.21.15.116:0/1808236337 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9684098ef0 con 0x5610d7faa850 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f969d5ab740 10 client.27746 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5610d801c8b0 con 0x5610d7faa850 2024-03-04T07:50:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.612+0000 7f9681600640 1 -- 172.21.15.116:0/1808236337 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f968c004040 con 0x5610d7faa850 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f9681600640 1 -- 172.21.15.116:0/1808236337 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f968409b010 con 0x5610d7faa850 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f969d5ab740 10 client.27746 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f969d5ab740 10 client.27746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f969d5ab740 20 client.27746 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f969d5ab740 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d801cff0 0x5610d801f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f969d5ab740 4 client.27746 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5610d7f17c00 con 0x5610d801cff0 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f9692a00640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d801cff0 0x5610d801f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f9692a00640 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d801cff0 0x5610d801f410 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebcec1f0f70626ae server_cookie=cb71a965f88d8aa8 in_seq=0 out_seq=0 2024-03-04T07:50:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.613+0000 7f9681600640 10 client.27746 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.631+0000 7f9681600640 1 -- 172.21.15.116:0/1808236337 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5610d7f17c00 con 0x5610d801cff0 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.631+0000 7f9681600640 10 client.27746 handle_command_reply: tid=0 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.631+0000 7f969d5ab740 1 client.27746 shutdown 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 msgr2=0x5610d7fbbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fbbb30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f968409ad40 tx=0x7f9684098e50 comp rx=0 tx=0).stop 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 shutdown_connections 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5610d801cff0 0x5610d801f410 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610d7fb03b0 0x5610d7fba180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610d7fab240 0x5610d7fbc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f9681600640 0 client.27746 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 --2- 172.21.15.116:0/1808236337 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5610d7faa850 0x5610d7fbbb30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 >> 172.21.15.116:0/1808236337 conn(0x5610d7fa5dc0 msgr2=0x5610d7fa4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 shutdown_connections 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 1 -- 172.21.15.116:0/1808236337 wait complete. 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.632+0000 7f969d5ab740 20 client.27746 trim_cache size 0 max 16384 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:50:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96780a1ea0 msgr2=0x7f96780a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96780a1ea0 0x7f96780a4360 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f968c02c640 tx=0x7f968c032590 comp rx=0 tx=0).stop 2024-03-04T07:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9694107c30 msgr2=0x7f969419de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9694107c30 0x7f969419de60 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f968400e9c0 tx=0x7f968400ee80 comp rx=0 tx=0).stop 2024-03-04T07:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 shutdown_connections 2024-03-04T07:50:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f96780a1ea0 0x7f96780a4360 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96941097e0 0x7f96941a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9694108e20 0x7f969419e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 --2- 172.21.15.116:0/1499033770 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9694107c30 0x7f969419de60 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 >> 172.21.15.116:0/1499033770 conn(0x7f9694076be0 msgr2=0x7f969410c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 shutdown_connections 2024-03-04T07:50:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:25.633+0000 7f969a800640 1 -- 172.21.15.116:0/1499033770 wait complete. 2024-03-04T07:50:27.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:26 smithi116 ceph-mon[32150]: pgmap v691: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:50:27.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:26 smithi152 ceph-mon[37154]: pgmap v691: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:50:27.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:26 smithi133 ceph-mon[36505]: pgmap v691: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:50:29.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:28 smithi116 ceph-mon[32150]: pgmap v692: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:50:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:28 smithi152 ceph-mon[37154]: pgmap v692: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:50:29.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:28 smithi133 ceph-mon[36505]: pgmap v692: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:50:30.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:50:30.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:50:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 -- 172.21.15.116:0/2304250155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 msgr2=0x7f56f8110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 --2- 172.21.15.116:0/2304250155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f8110030 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009900 tx=0x7f56f002bc50 comp rx=0 tx=0).stop 2024-03-04T07:50:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 -- 172.21.15.116:0/2304250155 shutdown_connections 2024-03-04T07:50:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 --2- 172.21.15.116:0/2304250155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f8110030 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 --2- 172.21.15.116:0/2304250155 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 0x7f56f8109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 --2- 172.21.15.116:0/2304250155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56f8107bf0 0x7f56f8108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 -- 172.21.15.116:0/2304250155 >> 172.21.15.116:0/2304250155 conn(0x7f56f8076b90 msgr2=0x7f56f8078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 -- 172.21.15.116:0/2304250155 shutdown_connections 2024-03-04T07:50:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.461+0000 7f5700600640 1 -- 172.21.15.116:0/2304250155 wait complete. 2024-03-04T07:50:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.462+0000 7f5700600640 1 Processor -- start 2024-03-04T07:50:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.462+0000 7f5700600640 1 -- start start 2024-03-04T07:50:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f5700600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56f8107bf0 0x7f56f819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f5700600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 0x7f56f819e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f5700600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f81a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f5700600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81138a0 con 0x7f56f81097a0 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f5700600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8113720 con 0x7f56f8108de0 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f5700600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8113a20 con 0x7f56f8107bf0 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fe600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f81a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fd200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 0x7f56f819e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fe600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f81a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38210/0 (socket says 172.21.15.116:38210) 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fe600640 1 -- 172.21.15.116:0/1116507432 learned_addr learned my addr 172.21.15.116:0/1116507432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fd200640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 0x7f56f819e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58694/0 (socket says 172.21.15.116:58694) 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fdc00640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56f8107bf0 0x7f56f819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fd200640 1 -- 172.21.15.116:0/1116507432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56f8107bf0 msgr2=0x7f56f819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fd200640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56f8107bf0 0x7f56f819de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fd200640 1 -- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 msgr2=0x7f56f81a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fd200640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f81a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fd200640 1 -- 172.21.15.116:0/1116507432 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f81a5aa0 con 0x7f56f8108de0 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.463+0000 7f56fe600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f81a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.464+0000 7f56fd200640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 0x7f56f819e3c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f56e800eb00 tx=0x7f56e800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.464+0000 7f56f6000640 1 -- 172.21.15.116:0/1116507432 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8010430 con 0x7f56f8108de0 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.464+0000 7f56f6000640 1 -- 172.21.15.116:0/1116507432 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56e8010ab0 con 0x7f56f8108de0 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.464+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f81a5d90 con 0x7f56f8108de0 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.464+0000 7f56f6000640 1 -- 172.21.15.116:0/1116507432 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8017bb0 con 0x7f56f8108de0 2024-03-04T07:50:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.464+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f81a6260 con 0x7f56f8108de0 2024-03-04T07:50:30.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.467+0000 7f56f6000640 1 -- 172.21.15.116:0/1116507432 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f56e8010c60 con 0x7f56f8108de0 2024-03-04T07:50:30.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.467+0000 7f57033a2740 1 Processor -- start 2024-03-04T07:50:30.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.467+0000 7f57033a2740 1 -- start start 2024-03-04T07:50:30.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.467+0000 7f57033a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf449f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.467+0000 7f57033a2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf44f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.467+0000 7f57033a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf451b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.467+0000 7f57033a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cdf377090 con 0x556cdf44f6b0 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f57033a2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cdf3aed30 con 0x556cdf44a540 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f57033a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cdf357a30 con 0x556cdf449b50 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf44f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf44f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58700/0 (socket says 172.21.15.116:58700) 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 -- 172.21.15.116:0/3328133483 learned_addr learned my addr 172.21.15.116:0/3328133483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fd200640 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf451b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fe600640 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf449f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 -- 172.21.15.116:0/3328133483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 msgr2=0x556cdf449f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf449f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 -- 172.21.15.116:0/3328133483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 msgr2=0x556cdf451b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf451b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 -- 172.21.15.116:0/3328133483 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cdf18d7d0 con 0x556cdf44a540 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fd200640 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf451b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.468+0000 7f56fdc00640 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf44f040 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009ca0 tx=0x7f56ec011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dce81cdaa200ae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f56e6a00640 1 -- 172.21.15.116:0/3328133483 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec004220 con 0x556cdf44a540 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f56e6a00640 1 -- 172.21.15.116:0/3328133483 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56ec0043c0 con 0x556cdf44a540 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f56e6a00640 1 -- 172.21.15.116:0/3328133483 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec019930 con 0x556cdf44a540 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f56f6000640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56e00a1e00 0x7f56e00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f56f6000640 1 -- 172.21.15.116:0/1116507432 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f56e80ce6b0 con 0x7f56f8108de0 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f56fe600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56e00a1e00 0x7f56e00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f57033a2740 1 -- 172.21.15.116:0/3328133483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 msgr2=0x556cdf44f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f57033a2740 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf44f040 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009ca0 tx=0x7f56ec011810 comp rx=0 tx=0).stop 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f57033a2740 1 -- 172.21.15.116:0/3328133483 shutdown_connections 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f57033a2740 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf451b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f57033a2740 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf44f040 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f57033a2740 1 --2- 172.21.15.116:0/3328133483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf449f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.469+0000 7f57033a2740 1 -- 172.21.15.116:0/3328133483 >> 172.21.15.116:0/3328133483 conn(0x556cdf4450c0 msgr2=0x556cdf4454c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.470+0000 7f57033a2740 1 -- 172.21.15.116:0/3328133483 shutdown_connections 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.470+0000 7f57033a2740 1 -- 172.21.15.116:0/3328133483 wait complete. 2024-03-04T07:50:30.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.470+0000 7f56fe600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56e00a1e00 0x7f56e00a42c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f56f00096c0 tx=0x7f56f002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.470+0000 7f57033a2740 1 -- start start 2024-03-04T07:50:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.470+0000 7f57033a2740 1 Processor -- start 2024-03-04T07:50:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f57033a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf449b50 0x556cdf453f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f57033a2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf454460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f57033a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf44f6b0 0x556cdf45b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f57033a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cdf34d0c0 con 0x556cdf449b50 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f57033a2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cdf3aadd0 con 0x556cdf44a540 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f57033a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cdf452660 con 0x556cdf44f6b0 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f57033a2740 10 client.0 _ll_register_callbacks cb 0x556cdf36a860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f56fdc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf449b50 0x556cdf453f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f56fdc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf449b50 0x556cdf453f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38234/0 (socket says 172.21.15.116:38234) 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f56fd200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf454460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f56fdc00640 1 -- 172.21.15.116:0/3142793304 learned_addr learned my addr 172.21.15.116:0/3142793304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.471+0000 7f56fe600640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf44f6b0 0x556cdf45b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fdc00640 1 -- 172.21.15.116:0/3142793304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf44f6b0 msgr2=0x556cdf45b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fdc00640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf44f6b0 0x556cdf45b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fdc00640 1 -- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 msgr2=0x556cdf454460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fdc00640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf454460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fdc00640 1 -- 172.21.15.116:0/3142793304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556cdf0c33f0 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fd200640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf454460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fe600640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf44f6b0 0x556cdf45b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56fdc00640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf449b50 0x556cdf453f20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0096e0 tx=0x7f56ec003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d429bc28aff28c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56db400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56db400640 1 -- 172.21.15.116:0/3142793304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec028070 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56db400640 1 -- 172.21.15.116:0/3142793304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f56ec005040 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56ec0043c0 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f56db400640 1 -- 172.21.15.116:0/3142793304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56ec034050 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f57033a2740 10 client.17619 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.472+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556cdf0830f0 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f56db400640 1 -- 172.21.15.116:0/3142793304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56ec0043c0 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f56db400640 1 -- 172.21.15.116:0/3142793304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f56ec004aa0 con 0x556cdf449b50 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f57033a2740 10 client.17619 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f57033a2740 10 client.17619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f57033a2740 20 client.17619 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f57033a2740 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf45f770 0x556cdf461b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f57033a2740 4 client.17619 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556cdf3bc270 con 0x556cdf45f770 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.473+0000 7f56fd200640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf45f770 0x556cdf461b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.474+0000 7f56fd200640 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf45f770 0x556cdf461b60 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b05361d7a331a65 server_cookie=e662adf7178f6cda in_seq=0 out_seq=0 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.474+0000 7f56db400640 10 client.17619 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.474+0000 7f56db400640 1 -- 172.21.15.116:0/3142793304 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556cdf3bc270 con 0x556cdf45f770 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.474+0000 7f56db400640 10 client.17619 handle_command_reply: tid=0 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.474+0000 7f57033a2740 1 client.17619 shutdown 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf449b50 msgr2=0x556cdf453f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf449b50 0x556cdf453f20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0096e0 tx=0x7f56ec003820 comp rx=0 tx=0).stop 2024-03-04T07:50:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 shutdown_connections 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf45f770 0x556cdf461b60 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf44f6b0 0x556cdf45b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf454460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 --2- 172.21.15.116:0/3142793304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf449b50 0x556cdf453f20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f56db400640 0 client.17619 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 >> 172.21.15.116:0/3142793304 conn(0x556cdf4450c0 msgr2=0x556cdf444360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 shutdown_connections 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 1 -- 172.21.15.116:0/3142793304 wait complete. 2024-03-04T07:50:30.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.475+0000 7f57033a2740 20 client.17619 trim_cache size 0 max 16384 2024-03-04T07:50:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.492+0000 7f57033a2740 1 Processor -- start 2024-03-04T07:50:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.492+0000 7f57033a2740 1 -- start start 2024-03-04T07:50:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.492+0000 7f57033a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf466420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.492+0000 7f57033a2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf45b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f57033a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf466960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f57033a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81ad0d0 con 0x556cdf44f6b0 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f57033a2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81af6b0 con 0x556cdf44a540 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f57033a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81ad6d0 con 0x556cdf449b50 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf45b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf45b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58718/0 (socket says 172.21.15.116:58718) 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fe600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf466960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 -- 172.21.15.116:0/1071217481 learned_addr learned my addr 172.21.15.116:0/1071217481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fdc00640 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf466420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 -- 172.21.15.116:0/1071217481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 msgr2=0x556cdf466420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf466420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 -- 172.21.15.116:0/1071217481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 msgr2=0x556cdf466960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf466960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 -- 172.21.15.116:0/1071217481 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cdf445e30 con 0x556cdf44a540 2024-03-04T07:50:30.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.493+0000 7f56fd200640 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf45b3c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f56e8000f80 tx=0x7f56e80bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2be336ea74c88f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f56e6000640 1 -- 172.21.15.116:0/1071217481 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e809b3a0 con 0x556cdf44a540 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f56e6000640 1 -- 172.21.15.116:0/1071217481 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56e809f4f0 con 0x556cdf44a540 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f56e6000640 1 -- 172.21.15.116:0/1071217481 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e809f130 con 0x556cdf44a540 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 -- 172.21.15.116:0/1071217481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 msgr2=0x556cdf45b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf45b3c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f56e8000f80 tx=0x7f56e80bb040 comp rx=0 tx=0).stop 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 -- 172.21.15.116:0/1071217481 shutdown_connections 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf466960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf45b3c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 --2- 172.21.15.116:0/1071217481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf466420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 -- 172.21.15.116:0/1071217481 >> 172.21.15.116:0/1071217481 conn(0x556cdf4450c0 msgr2=0x556cdf448bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 -- 172.21.15.116:0/1071217481 shutdown_connections 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.494+0000 7f57033a2740 1 -- 172.21.15.116:0/1071217481 wait complete. 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 -- start start 2024-03-04T07:50:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 Processor -- start 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf458b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf459040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf45a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81add30 con 0x556cdf44f6b0 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81af050 con 0x556cdf44a540 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81ae390 con 0x556cdf449b50 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.495+0000 7f57033a2740 10 client.0 _ll_register_callbacks cb 0x7f56ec0042e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf459040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fe600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf45a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf459040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58728/0 (socket says 172.21.15.116:58728) 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fdc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf458b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 -- 172.21.15.116:0/249863087 learned_addr learned my addr 172.21.15.116:0/249863087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 -- 172.21.15.116:0/249863087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 msgr2=0x556cdf458b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf458b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 -- 172.21.15.116:0/249863087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 msgr2=0x556cdf45a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf45a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 -- 172.21.15.116:0/249863087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556cdf45a640 con 0x556cdf44a540 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fe600640 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf45a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56fd200640 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf459040 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f56e809e040 tx=0x7f56e8017f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b6880a4e60c09a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.496+0000 7f56db400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f56db400640 1 -- 172.21.15.116:0/249863087 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8084070 con 0x556cdf44a540 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f56db400640 1 -- 172.21.15.116:0/249863087 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f56e8097a10 con 0x556cdf44a540 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56e809f4f0 con 0x556cdf44a540 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f56db400640 1 -- 172.21.15.116:0/249863087 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56e80b6020 con 0x556cdf44a540 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 10 client.27764 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x556cdf4baff0 con 0x556cdf44a540 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f56db400640 1 -- 172.21.15.116:0/249863087 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56e809f4f0 con 0x556cdf44a540 2024-03-04T07:50:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f56db400640 1 -- 172.21.15.116:0/249863087 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f56e809a910 con 0x556cdf44a540 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 10 client.27764 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 10 client.27764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 20 client.27764 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf4bb730 0x556cdf4bdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 4 client.27764 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.497+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556cdf3b65c0 con 0x556cdf4bb730 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f56fdc00640 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf4bb730 0x556cdf4bdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f56fdc00640 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf4bb730 0x556cdf4bdb90 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df01a3b44d4224ea server_cookie=541f3bb44c95fc90 in_seq=0 out_seq=0 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f56db400640 10 client.27764 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f56db400640 1 -- 172.21.15.116:0/249863087 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556cdf3b65c0 con 0x556cdf4bb730 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f56db400640 10 client.27764 handle_command_reply: tid=0 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 client.27764 shutdown 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 msgr2=0x556cdf459040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf459040 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f56e809e040 tx=0x7f56e8017f30 comp rx=0 tx=0).stop 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 shutdown_connections 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x556cdf4bb730 0x556cdf4bdb90 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cdf44f6b0 0x556cdf45a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x556cdf44a540 0x556cdf459040 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 --2- 172.21.15.116:0/249863087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cdf449b50 0x556cdf458b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f56db400640 0 client.27764 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 >> 172.21.15.116:0/249863087 conn(0x556cdf4450c0 msgr2=0x556cdf443ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.498+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 shutdown_connections 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f57033a2740 1 -- 172.21.15.116:0/249863087 wait complete. 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f57033a2740 20 client.27764 trim_cache size 0 max 16384 2024-03-04T07:50:30.505 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:50:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:50:30.506 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:50:30.506 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:50:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56e00a1e00 msgr2=0x7f56e00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56e00a1e00 0x7f56e00a42c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f56f00096c0 tx=0x7f56f002e040 comp rx=0 tx=0).stop 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 msgr2=0x7f56f819e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 0x7f56f819e3c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f56e800eb00 tx=0x7f56e800c2d0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 shutdown_connections 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56f81097a0 0x7f56f81a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f56f8108de0 0x7f56f819e3c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f56e00a1e00 0x7f56e00a42c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 --2- 172.21.15.116:0/1116507432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f56f8107bf0 0x7f56f819de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 >> 172.21.15.116:0/1116507432 conn(0x7f56f8076b90 msgr2=0x7f56f810c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.499+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 shutdown_connections 2024-03-04T07:50:30.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.500+0000 7f5700600640 1 -- 172.21.15.116:0/1116507432 wait complete. 2024-03-04T07:50:30.520 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:50:30.520 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.731+0000 7f6a78000640 1 -- 172.21.15.116:0/1320524785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a701097c0 msgr2=0x7f6a70110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.731+0000 7f6a78000640 1 --2- 172.21.15.116:0/1320524785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a701097c0 0x7f6a70110050 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009ca0 tx=0x7f6a6802c100 comp rx=0 tx=0).stop 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.731+0000 7f6a78000640 1 -- 172.21.15.116:0/1320524785 shutdown_connections 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.731+0000 7f6a78000640 1 --2- 172.21.15.116:0/1320524785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a701097c0 0x7f6a70110050 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.731+0000 7f6a78000640 1 --2- 172.21.15.116:0/1320524785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a70108e00 0x7f6a70109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.732+0000 7f6a78000640 1 --2- 172.21.15.116:0/1320524785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a70107c10 0x7f6a70108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.732+0000 7f6a78000640 1 -- 172.21.15.116:0/1320524785 >> 172.21.15.116:0/1320524785 conn(0x7f6a70076be0 msgr2=0x7f6a70079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.732+0000 7f6a78000640 1 -- 172.21.15.116:0/1320524785 shutdown_connections 2024-03-04T07:50:30.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.732+0000 7f6a78000640 1 -- 172.21.15.116:0/1320524785 wait complete. 2024-03-04T07:50:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.732+0000 7f6a78000640 1 Processor -- start 2024-03-04T07:50:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a78000640 1 -- start start 2024-03-04T07:50:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a78000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a70107c10 0x7f6a7019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a78000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a70108e00 0x7f6a7019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a78000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a701097c0 0x7f6a701a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a78000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a70113870 con 0x7f6a70108e00 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a78000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701136f0 con 0x7f6a70107c10 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a78000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701139f0 con 0x7f6a701097c0 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a74c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a70108e00 0x7f6a7019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a75600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a70107c10 0x7f6a7019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a75600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a70107c10 0x7f6a7019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58734/0 (socket says 172.21.15.116:58734) 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a75600640 1 -- 172.21.15.116:0/745637946 learned_addr learned my addr 172.21.15.116:0/745637946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.733+0000 7f6a76000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a701097c0 0x7f6a701a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a75600640 1 -- 172.21.15.116:0/745637946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a701097c0 msgr2=0x7f6a701a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a75600640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a701097c0 0x7f6a701a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a75600640 1 -- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a70108e00 msgr2=0x7f6a7019e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a75600640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a70108e00 0x7f6a7019e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a75600640 1 -- 172.21.15.116:0/745637946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a701a5aa0 con 0x7f6a70107c10 2024-03-04T07:50:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a75600640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a70107c10 0x7f6a7019de80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400e9c0 tx=0x7f6a6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a6e000640 1 -- 172.21.15.116:0/745637946 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400cd80 con 0x7f6a70107c10 2024-03-04T07:50:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a6e000640 1 -- 172.21.15.116:0/745637946 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a640035f0 con 0x7f6a70107c10 2024-03-04T07:50:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a701a5d90 con 0x7f6a70107c10 2024-03-04T07:50:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a6e000640 1 -- 172.21.15.116:0/745637946 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64010690 con 0x7f6a70107c10 2024-03-04T07:50:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.734+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a701a6250 con 0x7f6a70107c10 2024-03-04T07:50:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.735+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7010ef30 con 0x7f6a70107c10 2024-03-04T07:50:30.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.737+0000 7f6a6e000640 1 -- 172.21.15.116:0/745637946 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f6a640037a0 con 0x7f6a70107c10 2024-03-04T07:50:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.738+0000 7f6a6e000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6a580a1e00 0x7f6a580a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.738+0000 7f6a6e000640 1 -- 172.21.15.116:0/745637946 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f6a640ce120 con 0x7f6a70107c10 2024-03-04T07:50:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.739+0000 7f6a74c00640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6a580a1e00 0x7f6a580a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:30.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.739+0000 7f6a74c00640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6a580a1e00 0x7f6a580a42c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6a7019f390 tx=0x7f6a5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:30.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.742+0000 7f6a6e000640 1 -- 172.21.15.116:0/745637946 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6a6409b510 con 0x7f6a70107c10 2024-03-04T07:50:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.980+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a7002cff0 con 0x7f6a70107c10 2024-03-04T07:50:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.981+0000 7f6a6e000640 1 -- 172.21.15.116:0/745637946 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f6a7002cff0 con 0x7f6a70107c10 2024-03-04T07:50:30.983 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:50:30.983 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:50:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.984+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6a580a1e00 msgr2=0x7f6a580a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.984+0000 7f6a78000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6a580a1e00 0x7f6a580a42c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6a7019f390 tx=0x7f6a5c00b410 comp rx=0 tx=0).stop 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.984+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a70107c10 msgr2=0x7f6a7019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.984+0000 7f6a78000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a70107c10 0x7f6a7019de80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400e9c0 tx=0x7f6a6400ee80 comp rx=0 tx=0).stop 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.984+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 shutdown_connections 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.984+0000 7f6a78000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f6a580a1e00 0x7f6a580a42c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.984+0000 7f6a78000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a701097c0 0x7f6a701a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.985+0000 7f6a78000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a70108e00 0x7f6a7019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.985+0000 7f6a78000640 1 --2- 172.21.15.116:0/745637946 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6a70107c10 0x7f6a7019de80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.985+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 >> 172.21.15.116:0/745637946 conn(0x7f6a70076be0 msgr2=0x7f6a7010c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.985+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 shutdown_connections 2024-03-04T07:50:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:30.985+0000 7f6a78000640 1 -- 172.21.15.116:0/745637946 wait complete. 2024-03-04T07:50:31.003 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:50:31.003 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 -- 172.21.15.116:0/457397778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8108de0 msgr2=0x7f32e8109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 --2- 172.21.15.116:0/457397778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8108de0 0x7f32e8109260 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009900 tx=0x7f32d002bc30 comp rx=0 tx=0).stop 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 -- 172.21.15.116:0/457397778 shutdown_connections 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 --2- 172.21.15.116:0/457397778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e81097a0 0x7f32e8110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 --2- 172.21.15.116:0/457397778 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8108de0 0x7f32e8109260 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 --2- 172.21.15.116:0/457397778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32e8107bf0 0x7f32e8108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 -- 172.21.15.116:0/457397778 >> 172.21.15.116:0/457397778 conn(0x7f32e8076b90 msgr2=0x7f32e8078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 -- 172.21.15.116:0/457397778 shutdown_connections 2024-03-04T07:50:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.207+0000 7f32ed000640 1 -- 172.21.15.116:0/457397778 wait complete. 2024-03-04T07:50:31.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.208+0000 7f32ed000640 1 Processor -- start 2024-03-04T07:50:31.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.208+0000 7f32ed000640 1 -- start start 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32ed000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8107bf0 0x7f32e819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32ed000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e8108de0 0x7f32e819e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32ed000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32e81097a0 0x7f32e81a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32ed000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81138c0 con 0x7f32e8108de0 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32ed000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e8113740 con 0x7f32e8107bf0 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32ed000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e8113a40 con 0x7f32e81097a0 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e8108de0 0x7f32e819e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32e5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8107bf0 0x7f32e819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e8108de0 0x7f32e819e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38270/0 (socket says 172.21.15.116:38270) 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32dfe00640 1 -- 172.21.15.116:0/1800681726 learned_addr learned my addr 172.21.15.116:0/1800681726 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32e6800640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32e81097a0 0x7f32e81a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32e5e00640 1 -- 172.21.15.116:0/1800681726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32e81097a0 msgr2=0x7f32e81a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32e5e00640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32e81097a0 0x7f32e81a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32e5e00640 1 -- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e8108de0 msgr2=0x7f32e819e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32e5e00640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e8108de0 0x7f32e819e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.209+0000 7f32e5e00640 1 -- 172.21.15.116:0/1800681726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e81a5b30 con 0x7f32e8107bf0 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.210+0000 7f32e5e00640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8107bf0 0x7f32e819df10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e9c0 tx=0x7f32d800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.210+0000 7f32dea00640 1 -- 172.21.15.116:0/1800681726 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800ccc0 con 0x7f32e8107bf0 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.210+0000 7f32dea00640 1 -- 172.21.15.116:0/1800681726 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32d80035f0 con 0x7f32e8107bf0 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.210+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e81a5e20 con 0x7f32e8107bf0 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.210+0000 7f32dea00640 1 -- 172.21.15.116:0/1800681726 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800ce60 con 0x7f32e8107bf0 2024-03-04T07:50:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.210+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e81a6230 con 0x7f32e8107bf0 2024-03-04T07:50:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.213+0000 7f32dea00640 1 -- 172.21.15.116:0/1800681726 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f32d80037a0 con 0x7f32e8107bf0 2024-03-04T07:50:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.213+0000 7f32efc31740 1 Processor -- start 2024-03-04T07:50:31.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.213+0000 7f32efc31740 1 -- start start 2024-03-04T07:50:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32efc31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d4ca0 0x55f84a4d50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32efc31740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4da190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32efc31740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4da800 0x55f84a4dcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32efc31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84a439d60 con 0x55f84a4d4ca0 2024-03-04T07:50:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32efc31740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84a3e2cf0 con 0x55f84a4d5690 2024-03-04T07:50:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32efc31740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84a435e00 con 0x55f84a4da800 2024-03-04T07:50:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d4ca0 0x55f84a4d50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32e5e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4da190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d4ca0 0x55f84a4d50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38272/0 (socket says 172.21.15.116:38272) 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32e6800640 1 -- 172.21.15.116:0/1819474579 learned_addr learned my addr 172.21.15.116:0/1819474579 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32dfe00640 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4da800 0x55f84a4dcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32e5e00640 1 -- 172.21.15.116:0/1819474579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4da800 msgr2=0x55f84a4dcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32e5e00640 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4da800 0x55f84a4dcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.214+0000 7f32e5e00640 1 -- 172.21.15.116:0/1819474579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d4ca0 msgr2=0x55f84a4d50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32e5e00640 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d4ca0 0x55f84a4d50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32e5e00640 1 -- 172.21.15.116:0/1819474579 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f84a14e3f0 con 0x55f84a4d5690 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32e6800640 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d4ca0 0x55f84a4d50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32dea00640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f32cc0a1d80 0x7f32cc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32e5e00640 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4da190 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f32d80cbd50 tx=0x7f32d80d6780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da5358f3429fc153 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32e6800640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f32cc0a1d80 0x7f32cc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32dea00640 1 -- 172.21.15.116:0/1800681726 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f32d80ce2f0 con 0x7f32e8107bf0 2024-03-04T07:50:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32d7400640 1 -- 172.21.15.116:0/1819474579 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8097280 con 0x55f84a4d5690 2024-03-04T07:50:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32d7400640 1 -- 172.21.15.116:0/1819474579 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32d809b6e0 con 0x55f84a4d5690 2024-03-04T07:50:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.215+0000 7f32d7400640 1 -- 172.21.15.116:0/1819474579 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d809a940 con 0x55f84a4d5690 2024-03-04T07:50:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 -- 172.21.15.116:0/1819474579 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 msgr2=0x55f84a4da190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4da190 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f32d80cbd50 tx=0x7f32d80d6780 comp rx=0 tx=0).stop 2024-03-04T07:50:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32e6800640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f32cc0a1d80 0x7f32cc0a4240 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55f84a4d5ef0 tx=0x7f32e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 -- 172.21.15.116:0/1819474579 shutdown_connections 2024-03-04T07:50:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4da800 0x55f84a4dcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4da190 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 --2- 172.21.15.116:0/1819474579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d4ca0 0x55f84a4d50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 -- 172.21.15.116:0/1819474579 >> 172.21.15.116:0/1819474579 conn(0x55f84a4d0210 msgr2=0x55f84a4d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 -- 172.21.15.116:0/1819474579 shutdown_connections 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 -- 172.21.15.116:0/1819474579 wait complete. 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.216+0000 7f32efc31740 1 -- start start 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 1 Processor -- start 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4df150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d5690 0x55f84a4df690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4da800 0x55f84a4e66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84a4dd750 con 0x55f84a4d5690 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84a3d8220 con 0x55f84a4da800 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84a4dd8d0 con 0x55f84a4d4ca0 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32efc31740 10 client.0 _ll_register_callbacks cb 0x55f84a14dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d5690 0x55f84a4df690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d5690 0x55f84a4df690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38288/0 (socket says 172.21.15.116:38288) 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32dfe00640 1 -- 172.21.15.116:0/433212267 learned_addr learned my addr 172.21.15.116:0/433212267 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32e6800640 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4da800 0x55f84a4e66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.217+0000 7f32e5e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4df150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32e6800640 1 -- 172.21.15.116:0/433212267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 msgr2=0x55f84a4df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32e6800640 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4df150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32e6800640 1 -- 172.21.15.116:0/433212267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d5690 msgr2=0x55f84a4df690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32e6800640 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d5690 0x55f84a4df690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32e6800640 1 -- 172.21.15.116:0/433212267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f84a2187d0 con 0x55f84a4da800 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32e6800640 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4da800 0x55f84a4e66d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005fd0 tx=0x7f32e0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=116b9ea19b48d2b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32cbe00640 1 -- 172.21.15.116:0/433212267 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0004b20 con 0x55f84a4da800 2024-03-04T07:50:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32cbe00640 1 -- 172.21.15.116:0/433212267 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f32e0004cc0 con 0x55f84a4da800 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.218+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32d809b6e0 con 0x55f84a4da800 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32cbe00640 1 -- 172.21.15.116:0/433212267 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e0004250 con 0x55f84a4da800 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 10 client.27782 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f84a4e9ab0 con 0x55f84a4da800 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32cbe00640 1 -- 172.21.15.116:0/433212267 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32d809b6e0 con 0x55f84a4da800 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32cbe00640 1 -- 172.21.15.116:0/433212267 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f32e0026070 con 0x55f84a4da800 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 10 client.27782 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 10 client.27782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 20 client.27782 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a4eb0d0 0x55f84a4ed550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 4 client.27782 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.219+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f84a4472a0 con 0x55f84a4eb0d0 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.220+0000 7f32e5e00640 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a4eb0d0 0x55f84a4ed550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.220+0000 7f32e5e00640 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a4eb0d0 0x55f84a4ed550 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe820ba61d9168d4 server_cookie=a99ddeb6d09c7f6d in_seq=0 out_seq=0 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.220+0000 7f32cbe00640 10 client.27782 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32cbe00640 1 -- 172.21.15.116:0/433212267 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f32d8079e50 con 0x55f84a4eb0d0 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32cbe00640 10 client.27782 handle_command_reply: tid=0 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 client.27782 shutdown 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4da800 msgr2=0x55f84a4e66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4da800 0x55f84a4e66d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005fd0 tx=0x7f32e0004ab0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 shutdown_connections 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a4eb0d0 0x55f84a4ed550 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4da800 0x55f84a4e66d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4d5690 0x55f84a4df690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 --2- 172.21.15.116:0/433212267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4df150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32cbe00640 0 client.27782 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 >> 172.21.15.116:0/433212267 conn(0x55f84a4d0210 msgr2=0x55f84a4cf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 shutdown_connections 2024-03-04T07:50:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 1 -- 172.21.15.116:0/433212267 wait complete. 2024-03-04T07:50:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.221+0000 7f32efc31740 20 client.27782 trim_cache size 0 max 16384 2024-03-04T07:50:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.234+0000 7f32efc31740 1 Processor -- start 2024-03-04T07:50:31.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.234+0000 7f32efc31740 1 -- start start 2024-03-04T07:50:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32efc31740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32efc31740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32efc31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32efc31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81ad080 con 0x55f84a4da800 2024-03-04T07:50:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32efc31740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81af660 con 0x55f84a4d5690 2024-03-04T07:50:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32efc31740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81ad680 con 0x55f84a4d4ca0 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38304/0 (socket says 172.21.15.116:38304) 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 -- 172.21.15.116:0/3929402097 learned_addr learned my addr 172.21.15.116:0/3929402097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e5e00640 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 -- 172.21.15.116:0/3929402097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 msgr2=0x55f84a4cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 -- 172.21.15.116:0/3929402097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 msgr2=0x55f84a4e6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e6800640 1 -- 172.21.15.116:0/3929402097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f84a4d5a70 con 0x55f84a4da800 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.235+0000 7f32e5e00640 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.236+0000 7f32e6800640 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4cf160 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005ee0 tx=0x7f32e0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3018913579690e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.236+0000 7f32d6a00640 1 -- 172.21.15.116:0/3929402097 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e001e430 con 0x55f84a4da800 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.236+0000 7f32d6a00640 1 -- 172.21.15.116:0/3929402097 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e001eab0 con 0x55f84a4da800 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.236+0000 7f32d6a00640 1 -- 172.21.15.116:0/3929402097 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0031a00 con 0x55f84a4da800 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.236+0000 7f32efc31740 1 -- 172.21.15.116:0/3929402097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 msgr2=0x55f84a4cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.236+0000 7f32efc31740 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4cf160 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005ee0 tx=0x7f32e0004600 comp rx=0 tx=0).stop 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 -- 172.21.15.116:0/3929402097 shutdown_connections 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4cf160 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 --2- 172.21.15.116:0/3929402097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 -- 172.21.15.116:0/3929402097 >> 172.21.15.116:0/3929402097 conn(0x55f84a4d0210 msgr2=0x55f84a4d05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 -- 172.21.15.116:0/3929402097 shutdown_connections 2024-03-04T07:50:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 -- 172.21.15.116:0/3929402097 wait complete. 2024-03-04T07:50:31.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 -- start start 2024-03-04T07:50:31.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.237+0000 7f32efc31740 1 Processor -- start 2024-03-04T07:50:31.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32efc31740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4e6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32efc31740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32efc31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4e4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32efc31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81adce0 con 0x55f84a4da800 2024-03-04T07:50:31.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32efc31740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81af000 con 0x55f84a4d5690 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32efc31740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81ae340 con 0x55f84a4d4ca0 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32efc31740 10 client.0 _ll_register_callbacks cb 0x7f32e0005df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4e4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38314/0 (socket says 172.21.15.116:38314) 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32e6800640 1 -- 172.21.15.116:0/2795965468 learned_addr learned my addr 172.21.15.116:0/2795965468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32e5e00640 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4e6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32dfe00640 1 -- 172.21.15.116:0/2795965468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 msgr2=0x55f84a4e6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32dfe00640 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4e6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32dfe00640 1 -- 172.21.15.116:0/2795965468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 msgr2=0x55f84a4e4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32dfe00640 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4e4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32dfe00640 1 -- 172.21.15.116:0/2795965468 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f84a4e4e90 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.238+0000 7f32e6800640 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4e4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32dfe00640 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e65c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x55f84a4cbd60 tx=0x7f32d0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c8fbb6a4e884d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32cbe00640 1 -- 172.21.15.116:0/2795965468 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00044a0 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32cbe00640 1 -- 172.21.15.116:0/2795965468 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f32d00048d0 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32e001eab0 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32cbe00640 1 -- 172.21.15.116:0/2795965468 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32d0004ad0 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32efc31740 10 client.27788 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.239+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f84a5474c0 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32cbe00640 1 -- 172.21.15.116:0/2795965468 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e001eab0 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32cbe00640 1 -- 172.21.15.116:0/2795965468 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f32d0003830 con 0x55f84a4d5690 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32efc31740 10 client.27788 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32efc31740 10 client.27788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:31.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32efc31740 20 client.27788 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32efc31740 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a547ea0 0x55f84a54a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32efc31740 4 client.27788 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f84a4d66a0 con 0x55f84a547ea0 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32e5e00640 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a547ea0 0x55f84a54a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32e5e00640 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a547ea0 0x55f84a54a2c0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e14baa02d7e22574 server_cookie=bff265dd8634de41 in_seq=0 out_seq=0 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.240+0000 7f32cbe00640 10 client.27788 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32cbe00640 1 -- 172.21.15.116:0/2795965468 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f84a4d66a0 con 0x55f84a547ea0 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32cbe00640 10 client.27788 handle_command_reply: tid=0 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 client.27788 shutdown 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 msgr2=0x55f84a4e65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e65c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x55f84a4cbd60 tx=0x7f32d0004270 comp rx=0 tx=0).stop 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 shutdown_connections 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f84a547ea0 0x55f84a54a2c0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f84a4da800 0x55f84a4e4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32cbe00640 0 client.27788 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f84a4d5690 0x55f84a4e65c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 --2- 172.21.15.116:0/2795965468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f84a4d4ca0 0x55f84a4e6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 >> 172.21.15.116:0/2795965468 conn(0x55f84a4d0210 msgr2=0x55f84a4d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 shutdown_connections 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 1 -- 172.21.15.116:0/2795965468 wait complete. 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.241+0000 7f32efc31740 20 client.27788 trim_cache size 0 max 16384 2024-03-04T07:50:31.247 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:50:31.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f32cc0a1d80 msgr2=0x7f32cc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f32cc0a1d80 0x7f32cc0a4240 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55f84a4d5ef0 tx=0x7f32e0009500 comp rx=0 tx=0).stop 2024-03-04T07:50:31.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8107bf0 msgr2=0x7f32e819df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:31.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8107bf0 0x7f32e819df10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e9c0 tx=0x7f32d800ee80 comp rx=0 tx=0).stop 2024-03-04T07:50:31.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 shutdown_connections 2024-03-04T07:50:31.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f32cc0a1d80 0x7f32cc0a4240 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32e81097a0 0x7f32e81a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e8108de0 0x7f32e819e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 --2- 172.21.15.116:0/1800681726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f32e8107bf0 0x7f32e819df10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:31.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 >> 172.21.15.116:0/1800681726 conn(0x7f32e8076b90 msgr2=0x7f32e810c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:31.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 shutdown_connections 2024-03-04T07:50:31.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:31.242+0000 7f32ed000640 1 -- 172.21.15.116:0/1800681726 wait complete. 2024-03-04T07:50:31.257 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:30 smithi116 ceph-mon[32150]: pgmap v693: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:50:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:30 smithi152 ceph-mon[37154]: pgmap v693: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:50:31.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:30 smithi133 ceph-mon[36505]: pgmap v693: 97 pgs: 97 active+clean; 3.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-03-04T07:50:32.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:31 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/745637946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:50:32.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:31 smithi116 ceph-mon[32150]: pgmap v694: 97 pgs: 97 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:50:32.260 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:50:32.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:50:32.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:50:32.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:31 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/745637946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:50:32.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:31 smithi152 ceph-mon[37154]: pgmap v694: 97 pgs: 97 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:50:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:31 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/745637946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:50:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:31 smithi133 ceph-mon[36505]: pgmap v694: 97 pgs: 97 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.487+0000 7f6959400640 1 -- 172.21.15.116:0/862035112 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69541053c0 msgr2=0x7f695410bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.487+0000 7f6959400640 1 --2- 172.21.15.116:0/862035112 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69541053c0 0x7f695410bc50 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f694c009900 tx=0x7f694c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.488+0000 7f6959400640 1 -- 172.21.15.116:0/862035112 shutdown_connections 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.488+0000 7f6959400640 1 --2- 172.21.15.116:0/862035112 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f69541053c0 0x7f695410bc50 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.488+0000 7f6959400640 1 --2- 172.21.15.116:0/862035112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6954104a00 0x7f6954104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.488+0000 7f6959400640 1 --2- 172.21.15.116:0/862035112 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6954103810 0x7f6954103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.488+0000 7f6959400640 1 -- 172.21.15.116:0/862035112 >> 172.21.15.116:0/862035112 conn(0x7f69540fed80 msgr2=0x7f69541011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.488+0000 7f6959400640 1 -- 172.21.15.116:0/862035112 shutdown_connections 2024-03-04T07:50:32.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.488+0000 7f6959400640 1 -- 172.21.15.116:0/862035112 wait complete. 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.489+0000 7f6959400640 1 Processor -- start 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.489+0000 7f6959400640 1 -- start start 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.489+0000 7f6959400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6954103810 0x7f6954199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6959400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6954104a00 0x7f695419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6959400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69541053c0 0x7f69541a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6959400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695410f4e0 con 0x7f6954104a00 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6959400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695410f360 con 0x7f6954103810 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6959400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695410f660 con 0x7f69541053c0 2024-03-04T07:50:32.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6951e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6954104a00 0x7f695419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6954103810 0x7f6954199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6954103810 0x7f6954199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58816/0 (socket says 172.21.15.116:58816) 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 -- 172.21.15.116:0/1233145624 learned_addr learned my addr 172.21.15.116:0/1233145624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6953200640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69541053c0 0x7f69541a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 -- 172.21.15.116:0/1233145624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69541053c0 msgr2=0x7f69541a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69541053c0 0x7f69541a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 -- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6954104a00 msgr2=0x7f695419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6954104a00 0x7f695419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.490+0000 7f6952800640 1 -- 172.21.15.116:0/1233145624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69541a1770 con 0x7f6954103810 2024-03-04T07:50:32.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.491+0000 7f6952800640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6954103810 0x7f6954199b50 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f694400eb70 tx=0x7f694400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.491+0000 7f694b400640 1 -- 172.21.15.116:0/1233145624 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944003b70 con 0x7f6954103810 2024-03-04T07:50:32.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.491+0000 7f694b400640 1 -- 172.21.15.116:0/1233145624 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f694400ee70 con 0x7f6954103810 2024-03-04T07:50:32.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.491+0000 7f694b400640 1 -- 172.21.15.116:0/1233145624 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944010830 con 0x7f6954103810 2024-03-04T07:50:32.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.491+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69541a1a60 con 0x7f6954103810 2024-03-04T07:50:32.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.491+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69541a1f60 con 0x7f6954103810 2024-03-04T07:50:32.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.494+0000 7f694b400640 1 -- 172.21.15.116:0/1233145624 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f6944004020 con 0x7f6954103810 2024-03-04T07:50:32.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.494+0000 7f695c07c740 1 Processor -- start 2024-03-04T07:50:32.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.494+0000 7f695c07c740 1 -- start start 2024-03-04T07:50:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f695c07c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114f0d0 0x55910114f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f695c07c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x5591011545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f695c07c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x5591011570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f695c07c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101062e20 con 0x559101154c30 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f695c07c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591010b4a20 con 0x55910114f0d0 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f695c07c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55910105db00 con 0x55910114fac0 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f6953200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114f0d0 0x55910114f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f6953200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114f0d0 0x55910114f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58826/0 (socket says 172.21.15.116:58826) 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f6953200640 1 -- 172.21.15.116:0/2567994750 learned_addr learned my addr 172.21.15.116:0/2567994750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f6952800640 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x5591011545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.495+0000 7f6951e00640 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x5591011570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f6953200640 1 -- 172.21.15.116:0/2567994750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 msgr2=0x5591011545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f6953200640 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x5591011545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f6953200640 1 -- 172.21.15.116:0/2567994750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 msgr2=0x5591011570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f6953200640 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x5591011570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f6953200640 1 -- 172.21.15.116:0/2567994750 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559101072e10 con 0x55910114f0d0 2024-03-04T07:50:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f694b400640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69380a1e00 0x7f69380a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f6953200640 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114f0d0 0x55910114f4f0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f694c005ab0 tx=0x7f694c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0b0667d1772fdaf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f694b400640 1 -- 172.21.15.116:0/1233145624 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f69440ce4a0 con 0x7f6954103810 2024-03-04T07:50:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f6952800640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69380a1e00 0x7f69380a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f693f400640 1 -- 172.21.15.116:0/2567994750 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c042070 con 0x55910114f0d0 2024-03-04T07:50:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f693f400640 1 -- 172.21.15.116:0/2567994750 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f694c02cb90 con 0x55910114f0d0 2024-03-04T07:50:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.496+0000 7f693f400640 1 -- 172.21.15.116:0/2567994750 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c004920 con 0x55910114f0d0 2024-03-04T07:50:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 -- 172.21.15.116:0/2567994750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114f0d0 msgr2=0x55910114f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f6952800640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69380a1e00 0x7f69380a42c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x55910114fec0 tx=0x7f69440d1ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114f0d0 0x55910114f4f0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f694c005ab0 tx=0x7f694c009560 comp rx=0 tx=0).stop 2024-03-04T07:50:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 -- 172.21.15.116:0/2567994750 shutdown_connections 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x5591011570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x5591011545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 --2- 172.21.15.116:0/2567994750 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114f0d0 0x55910114f4f0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 -- 172.21.15.116:0/2567994750 >> 172.21.15.116:0/2567994750 conn(0x55910114a640 msgr2=0x55910114aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 -- 172.21.15.116:0/2567994750 shutdown_connections 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.497+0000 7f695c07c740 1 -- 172.21.15.116:0/2567994750 wait complete. 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 -- start start 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 Processor -- start 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x5591011594c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x559101159a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x559101160a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101070f40 con 0x55910114f0d0 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591010b0ac0 con 0x559101154c30 2024-03-04T07:50:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101157b80 con 0x55910114fac0 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.498+0000 7f695c07c740 10 client.0 _ll_register_callbacks cb 0x5591010afaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x559101159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6953200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x5591011594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x559101159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53220/0 (socket says 172.21.15.116:53220) 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 -- 172.21.15.116:0/289532277 learned_addr learned my addr 172.21.15.116:0/289532277 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6953200640 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x559101160a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 -- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 msgr2=0x559101160a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x559101160a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 -- 172.21.15.116:0/289532277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 msgr2=0x5591011594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x5591011594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6951e00640 1 -- 172.21.15.116:0/289532277 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559100e917e0 con 0x55910114fac0 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.499+0000 7f6953200640 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x5591011594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f6951e00640 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x559101159a00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6940012700 tx=0x7f6940012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27b0ee3083085788 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f6933e00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f6933e00640 1 -- 172.21.15.116:0/289532277 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694000f8c0 con 0x55910114fac0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f6933e00640 1 -- 172.21.15.116:0/289532277 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f694000fa60 con 0x55910114fac0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f694c02cb90 con 0x55910114fac0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f6933e00640 1 -- 172.21.15.116:0/289532277 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f694000fc60 con 0x55910114fac0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f695c07c740 10 client.27667 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x559100ef1f70 con 0x55910114fac0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.500+0000 7f6933e00640 1 -- 172.21.15.116:0/289532277 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f694c02cb90 con 0x55910114fac0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f6933e00640 1 -- 172.21.15.116:0/289532277 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6940005490 con 0x55910114fac0 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f695c07c740 10 client.27667 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f695c07c740 10 client.27667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f695c07c740 20 client.27667 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f695c07c740 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011653b0 0x5591011677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f695c07c740 4 client.27667 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:32.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591010c1f60 con 0x5591011653b0 2024-03-04T07:50:32.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f6953200640 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011653b0 0x5591011677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f6953200640 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011653b0 0x5591011677f0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c825b2d0a569bf8f server_cookie=368370bc04af858f in_seq=0 out_seq=0 2024-03-04T07:50:32.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.501+0000 7f6933e00640 10 client.27667 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f6933e00640 1 -- 172.21.15.116:0/289532277 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5591010c1f60 con 0x5591011653b0 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f6933e00640 10 client.27667 handle_command_reply: tid=0 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 client.27667 shutdown 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 msgr2=0x559101159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x559101159a00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6940012700 tx=0x7f6940012bc0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 shutdown_connections 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011653b0 0x5591011677f0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x559101160a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x559101159a00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f6933e00640 0 client.27667 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 --2- 172.21.15.116:0/289532277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x5591011594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.502+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 >> 172.21.15.116:0/289532277 conn(0x55910114a640 msgr2=0x55910114b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.503+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 shutdown_connections 2024-03-04T07:50:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.503+0000 7f695c07c740 1 -- 172.21.15.116:0/289532277 wait complete. 2024-03-04T07:50:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.503+0000 7f695c07c740 20 client.27667 trim_cache size 0 max 16384 2024-03-04T07:50:32.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 Processor -- start 2024-03-04T07:50:32.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 -- start start 2024-03-04T07:50:32.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114f0d0 0x559101150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114fac0 0x559101151500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x559101160a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69541a8c80 con 0x559101154c30 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69541ab260 con 0x55910114fac0 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.520+0000 7f695c07c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69541a9280 con 0x55910114f0d0 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114fac0 0x559101151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114fac0 0x559101151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58844/0 (socket says 172.21.15.116:58844) 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 -- 172.21.15.116:0/1439715992 learned_addr learned my addr 172.21.15.116:0/1439715992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6953200640 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114f0d0 0x559101150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6953200640 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x559101160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 -- 172.21.15.116:0/1439715992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114f0d0 msgr2=0x559101150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114f0d0 0x559101150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 -- 172.21.15.116:0/1439715992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 msgr2=0x559101160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x559101160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 -- 172.21.15.116:0/1439715992 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559100d89170 con 0x55910114fac0 2024-03-04T07:50:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6953200640 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x559101160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:50:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.521+0000 7f6951e00640 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114fac0 0x559101151500 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6940005800 tx=0x7f694000ff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d2a7158264779d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f693ea00640 1 -- 172.21.15.116:0/1439715992 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694002b020 con 0x55910114fac0 2024-03-04T07:50:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f693ea00640 1 -- 172.21.15.116:0/1439715992 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f694001c5b0 con 0x55910114fac0 2024-03-04T07:50:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f693ea00640 1 -- 172.21.15.116:0/1439715992 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400053c0 con 0x55910114fac0 2024-03-04T07:50:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 -- 172.21.15.116:0/1439715992 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114fac0 msgr2=0x559101151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114fac0 0x559101151500 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6940005800 tx=0x7f694000ff90 comp rx=0 tx=0).stop 2024-03-04T07:50:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 -- 172.21.15.116:0/1439715992 shutdown_connections 2024-03-04T07:50:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559101154c30 0x559101160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55910114fac0 0x559101151500 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 --2- 172.21.15.116:0/1439715992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114f0d0 0x559101150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 -- 172.21.15.116:0/1439715992 >> 172.21.15.116:0/1439715992 conn(0x55910114a640 msgr2=0x55910114aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 -- 172.21.15.116:0/1439715992 shutdown_connections 2024-03-04T07:50:32.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.522+0000 7f695c07c740 1 -- 172.21.15.116:0/1439715992 wait complete. 2024-03-04T07:50:32.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 -- start start 2024-03-04T07:50:32.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 Processor -- start 2024-03-04T07:50:32.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x55910115e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x55910115ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x55910115f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69541a98e0 con 0x55910114f0d0 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69541aac00 con 0x559101154c30 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69541a9f40 con 0x55910114fac0 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.523+0000 7f695c07c740 10 client.0 _ll_register_callbacks cb 0x7f6940009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6953200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x55910115e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6953200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x55910115e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38360/0 (socket says 172.21.15.116:38360) 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6953200640 1 -- 172.21.15.116:0/1487869745 learned_addr learned my addr 172.21.15.116:0/1487869745 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6951e00640 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x55910115ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6953200640 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x55910115f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6951e00640 1 -- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 msgr2=0x55910115f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6951e00640 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x55910115f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6951e00640 1 -- 172.21.15.116:0/1487869745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 msgr2=0x55910115e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6951e00640 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x55910115e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.524+0000 7f6951e00640 1 -- 172.21.15.116:0/1487869745 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55910115fee0 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f6951e00640 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x55910115ea60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6940012420 tx=0x7f6940005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88429b46f8b88ed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f6933e00640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f6933e00640 1 -- 172.21.15.116:0/1487869745 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940034020 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f6933e00640 1 -- 172.21.15.116:0/1487869745 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f694001c970 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f694001c5b0 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f6933e00640 1 -- 172.21.15.116:0/1487869745 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f694001cbb0 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f695c07c740 10 client.27673 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5591011c9680 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f6933e00640 1 -- 172.21.15.116:0/1487869745 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f694001c5b0 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.525+0000 7f6933e00640 1 -- 172.21.15.116:0/1487869745 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f6940027460 con 0x55910114fac0 2024-03-04T07:50:32.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f695c07c740 10 client.27673 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f695c07c740 10 client.27673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f695c07c740 20 client.27673 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f695c07c740 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011c9ef0 0x5591011cc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f695c07c740 4 client.27673 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5591010bc2b0 con 0x5591011c9ef0 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f6953200640 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011c9ef0 0x5591011cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f6953200640 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011c9ef0 0x5591011cc310 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f5115fc6bb2ef0a server_cookie=e5755a446120a128 in_seq=0 out_seq=0 2024-03-04T07:50:32.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.526+0000 7f6933e00640 10 client.27673 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f6933e00640 1 -- 172.21.15.116:0/1487869745 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5591010bc2b0 con 0x5591011c9ef0 2024-03-04T07:50:32.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f6933e00640 10 client.27673 handle_command_reply: tid=0 2024-03-04T07:50:32.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 client.27673 shutdown 2024-03-04T07:50:32.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 msgr2=0x55910115ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x55910115ea60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6940012420 tx=0x7f6940005370 comp rx=0 tx=0).stop 2024-03-04T07:50:32.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 shutdown_connections 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5591011c9ef0 0x5591011cc310 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x559101154c30 0x55910115f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55910114fac0 0x55910115ea60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 --2- 172.21.15.116:0/1487869745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55910114f0d0 0x55910115e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f6933e00640 0 client.27673 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 >> 172.21.15.116:0/1487869745 conn(0x55910114a640 msgr2=0x559101157fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 shutdown_connections 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 1 -- 172.21.15.116:0/1487869745 wait complete. 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.528+0000 7f695c07c740 20 client.27673 trim_cache size 0 max 16384 2024-03-04T07:50:32.536 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:50:32.537 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:50:32.537 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "f99ff21a-28fb-46f7-ad39-2ec86c7e1143", 2024-03-04T07:50:32.537 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:50:32.537 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69380a1e00 msgr2=0x7f69380a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69380a1e00 0x7f69380a42c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x55910114fec0 tx=0x7f69440d1ee0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6954103810 msgr2=0x7f6954199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6954103810 0x7f6954199b50 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f694400eb70 tx=0x7f694400c6a0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 shutdown_connections 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f69380a1e00 0x7f69380a42c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69541053c0 0x7f69541a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6954104a00 0x7f695419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 --2- 172.21.15.116:0/1233145624 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6954103810 0x7f6954199b50 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 >> 172.21.15.116:0/1233145624 conn(0x7f69540fed80 msgr2=0x7f6954107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 shutdown_connections 2024-03-04T07:50:32.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.529+0000 7f6959400640 1 -- 172.21.15.116:0/1233145624 wait complete. 2024-03-04T07:50:32.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:50:32.549 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:50:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.765+0000 7fad4a600640 1 -- 172.21.15.116:0/4202161737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 msgr2=0x7fad44104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.765+0000 7fad4a600640 1 --2- 172.21.15.116:0/4202161737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad44104e80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009900 tx=0x7fad2c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:50:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.766+0000 7fad4a600640 1 -- 172.21.15.116:0/4202161737 shutdown_connections 2024-03-04T07:50:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.766+0000 7fad4a600640 1 --2- 172.21.15.116:0/4202161737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad441053c0 0x7fad4410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.766+0000 7fad4a600640 1 --2- 172.21.15.116:0/4202161737 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad44104e80 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.766+0000 7fad4a600640 1 --2- 172.21.15.116:0/4202161737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad44103810 0x7fad44103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.766+0000 7fad4a600640 1 -- 172.21.15.116:0/4202161737 >> 172.21.15.116:0/4202161737 conn(0x7fad440fed80 msgr2=0x7fad441011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.766+0000 7fad4a600640 1 -- 172.21.15.116:0/4202161737 shutdown_connections 2024-03-04T07:50:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.766+0000 7fad4a600640 1 -- 172.21.15.116:0/4202161737 wait complete. 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 Processor -- start 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 -- start start 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad44103810 0x7fad44199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad4419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad441053c0 0x7fad441a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4410f4e0 con 0x7fad441053c0 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4410f360 con 0x7fad44104a00 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad4a600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4410f660 con 0x7fad44103810 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad3be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad4419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad43e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad441053c0 0x7fad441a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad3be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad4419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58868/0 (socket says 172.21.15.116:58868) 2024-03-04T07:50:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad3be00640 1 -- 172.21.15.116:0/2435808266 learned_addr learned my addr 172.21.15.116:0/2435808266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.767+0000 7fad43400640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad44103810 0x7fad44199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad3be00640 1 -- 172.21.15.116:0/2435808266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad44103810 msgr2=0x7fad44199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad3be00640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad44103810 0x7fad44199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad3be00640 1 -- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad441053c0 msgr2=0x7fad441a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad3be00640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad441053c0 0x7fad441a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad3be00640 1 -- 172.21.15.116:0/2435808266 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad441a17b0 con 0x7fad44104a00 2024-03-04T07:50:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad3be00640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad4419a0d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fad2c02c8f0 tx=0x7fad2c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad40c00640 1 -- 172.21.15.116:0/2435808266 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c041070 con 0x7fad44104a00 2024-03-04T07:50:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad40c00640 1 -- 172.21.15.116:0/2435808266 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad2c004450 con 0x7fad44104a00 2024-03-04T07:50:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad40c00640 1 -- 172.21.15.116:0/2435808266 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c003df0 con 0x7fad44104a00 2024-03-04T07:50:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad441a1a40 con 0x7fad44104a00 2024-03-04T07:50:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.768+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad441a1ee0 con 0x7fad44104a00 2024-03-04T07:50:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.771+0000 7fad4d417740 1 Processor -- start 2024-03-04T07:50:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad4d417740 1 -- start start 2024-03-04T07:50:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad40c00640 1 -- 172.21.15.116:0/2435808266 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fad2c044070 con 0x7fad44104a00 2024-03-04T07:50:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad4d417740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad4d417740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0fa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad4d417740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad4d417740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41c022090 con 0x55b41c0fa6b0 2024-03-04T07:50:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad4d417740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41c059d30 con 0x55b41c0f5540 2024-03-04T07:50:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad4d417740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41c002a30 con 0x55b41c0f4b50 2024-03-04T07:50:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad43400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad43e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad3be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad43400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0fa040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58874/0 (socket says 172.21.15.116:58874) 2024-03-04T07:50:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad43e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53256/0 (socket says 172.21.15.116:53256) 2024-03-04T07:50:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.772+0000 7fad43400640 1 -- 172.21.15.116:0/3307354015 learned_addr learned my addr 172.21.15.116:0/3307354015 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43e00640 1 -- 172.21.15.116:0/3307354015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 msgr2=0x55b41c0fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43e00640 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43e00640 1 -- 172.21.15.116:0/3307354015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 msgr2=0x55b41c0fcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43e00640 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0fcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43e00640 1 -- 172.21.15.116:0/3307354015 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b41be387d0 con 0x55b41c0f4b50 2024-03-04T07:50:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43400640 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43e00640 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0f4f70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009f10 tx=0x7fad3c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc2fa1e4c5a0d003 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad40c00640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fad240a1e00 0x7fad240a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad38c00640 1 -- 172.21.15.116:0/3307354015 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c004d20 con 0x55b41c0f4b50 2024-03-04T07:50:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad40c00640 1 -- 172.21.15.116:0/2435808266 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fad2c0ef890 con 0x7fad44104a00 2024-03-04T07:50:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad43400640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fad240a1e00 0x7fad240a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad38c00640 1 -- 172.21.15.116:0/3307354015 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad3c004030 con 0x55b41c0f4b50 2024-03-04T07:50:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.773+0000 7fad38c00640 1 -- 172.21.15.116:0/3307354015 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c005260 con 0x55b41c0f4b50 2024-03-04T07:50:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad43400640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fad240a1e00 0x7fad240a42c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55b41c0f5940 tx=0x7fad34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 -- 172.21.15.116:0/3307354015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 msgr2=0x55b41c0f4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0f4f70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009f10 tx=0x7fad3c011810 comp rx=0 tx=0).stop 2024-03-04T07:50:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 -- 172.21.15.116:0/3307354015 shutdown_connections 2024-03-04T07:50:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 --2- 172.21.15.116:0/3307354015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0f4f70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 -- 172.21.15.116:0/3307354015 >> 172.21.15.116:0/3307354015 conn(0x55b41c0f00c0 msgr2=0x55b41c0f04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 -- 172.21.15.116:0/3307354015 shutdown_connections 2024-03-04T07:50:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.774+0000 7fad4d417740 1 -- 172.21.15.116:0/3307354015 wait complete. 2024-03-04T07:50:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 -- start start 2024-03-04T07:50:32.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 Processor -- start 2024-03-04T07:50:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0ff450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41bff80c0 con 0x55b41c0fa6b0 2024-03-04T07:50:32.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41c055dd0 con 0x55b41c0f5540 2024-03-04T07:50:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.775+0000 7fad4d417740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41c0fd660 con 0x55b41c0f4b50 2024-03-04T07:50:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad4d417740 10 client.0 _ll_register_callbacks cb 0x55b41c015860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0ff450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0ff450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58886/0 (socket says 172.21.15.116:58886) 2024-03-04T07:50:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 -- 172.21.15.116:0/1204864032 learned_addr learned my addr 172.21.15.116:0/1204864032 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad43400640 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad43e00640 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 -- 172.21.15.116:0/1204864032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 msgr2=0x55b41c0fef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 -- 172.21.15.116:0/1204864032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 msgr2=0x55b41c106490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c106490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 -- 172.21.15.116:0/1204864032 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b41bd6e3f0 con 0x55b41c0f5540 2024-03-04T07:50:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad3be00640 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0ff450 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0b8f50 tx=0x7fad2c005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86fbd774640ef988 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad31600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad31600640 1 -- 172.21.15.116:0/1204864032 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c04b070 con 0x55b41c0f5540 2024-03-04T07:50:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.776+0000 7fad31600640 1 -- 172.21.15.116:0/1204864032 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fad2c0dc0c0 con 0x55b41c0f5540 2024-03-04T07:50:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad3c004030 con 0x55b41c0f5540 2024-03-04T07:50:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad31600640 1 -- 172.21.15.116:0/1204864032 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad2c0f5020 con 0x55b41c0f5540 2024-03-04T07:50:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 10 client.27818 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b41bd2e0f0 con 0x55b41c0f5540 2024-03-04T07:50:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad31600640 1 -- 172.21.15.116:0/1204864032 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad3c004030 con 0x55b41c0f5540 2024-03-04T07:50:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad31600640 1 -- 172.21.15.116:0/1204864032 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fad2c0b8bf0 con 0x55b41c0f5540 2024-03-04T07:50:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 10 client.27818 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 10 client.27818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 20 client.27818 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c10a980 0x55b41c10cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 4 client.27818 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b41c067270 con 0x55b41c10a980 2024-03-04T07:50:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.777+0000 7fad43400640 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c10a980 0x55b41c10cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.778+0000 7fad43400640 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c10a980 0x55b41c10cd70 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e546812f5ab017 server_cookie=210eb1a6932832d6 in_seq=0 out_seq=0 2024-03-04T07:50:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.778+0000 7fad31600640 10 client.27818 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad31600640 1 -- 172.21.15.116:0/1204864032 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b41c067270 con 0x55b41c10a980 2024-03-04T07:50:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad31600640 10 client.27818 handle_command_reply: tid=0 2024-03-04T07:50:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 client.27818 shutdown 2024-03-04T07:50:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 msgr2=0x55b41c0ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0ff450 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0b8f50 tx=0x7fad2c005b50 comp rx=0 tx=0).stop 2024-03-04T07:50:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 shutdown_connections 2024-03-04T07:50:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c10a980 0x55b41c10cd70 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c0ff450 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 --2- 172.21.15.116:0/1204864032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 >> 172.21.15.116:0/1204864032 conn(0x55b41c0f00c0 msgr2=0x55b41c0ef360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad31600640 0 client.27818 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 shutdown_connections 2024-03-04T07:50:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 1 -- 172.21.15.116:0/1204864032 wait complete. 2024-03-04T07:50:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.779+0000 7fad4d417740 20 client.27818 trim_cache size 0 max 16384 2024-03-04T07:50:32.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.796+0000 7fad4d417740 1 Processor -- start 2024-03-04T07:50:32.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.796+0000 7fad4d417740 1 -- start start 2024-03-04T07:50:32.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad4d417740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fe530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad4d417740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c106450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad4d417740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0eead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad4d417740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441a8c80 con 0x55b41c0fa6b0 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad4d417740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441ab260 con 0x55b41c0f5540 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad4d417740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441a9280 con 0x55b41c0f4b50 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c106450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad43e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c106450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58890/0 (socket says 172.21.15.116:58890) 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 -- 172.21.15.116:0/1790466348 learned_addr learned my addr 172.21.15.116:0/1790466348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad43400640 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fe530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 -- 172.21.15.116:0/1790466348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 msgr2=0x55b41c0fe530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fe530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 -- 172.21.15.116:0/1790466348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 msgr2=0x55b41c0eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.797+0000 7fad3be00640 1 -- 172.21.15.116:0/1790466348 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b41c0f5dd0 con 0x55b41c0f5540 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad3be00640 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c106450 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fad2c02c180 tx=0x7fad2c003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66976a012304d8c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad33e00640 1 -- 172.21.15.116:0/1790466348 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c04b070 con 0x55b41c0f5540 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad33e00640 1 -- 172.21.15.116:0/1790466348 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad2c0f2da0 con 0x55b41c0f5540 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad33e00640 1 -- 172.21.15.116:0/1790466348 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c0b8820 con 0x55b41c0f5540 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 -- 172.21.15.116:0/1790466348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 msgr2=0x55b41c106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c106450 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fad2c02c180 tx=0x7fad2c003f90 comp rx=0 tx=0).stop 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 -- 172.21.15.116:0/1790466348 shutdown_connections 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0fa6b0 0x55b41c0eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0f5540 0x55b41c106450 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 --2- 172.21.15.116:0/1790466348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c0fe530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 -- 172.21.15.116:0/1790466348 >> 172.21.15.116:0/1790466348 conn(0x55b41c0f00c0 msgr2=0x55b41c0f6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.798+0000 7fad4d417740 1 -- 172.21.15.116:0/1790466348 shutdown_connections 2024-03-04T07:50:32.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 -- 172.21.15.116:0/1790466348 wait complete. 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 -- start start 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 Processor -- start 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0f5540 0x55b41c1045b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0fa6b0 0x55b41c104af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441a98e0 con 0x55b41c0f5540 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441aac00 con 0x55b41c0fa6b0 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441a9f40 con 0x55b41c0f4b50 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.799+0000 7fad4d417740 10 client.0 _ll_register_callbacks cb 0x7fad2c0f5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad3be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0f5540 0x55b41c1045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad43e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0fa6b0 0x55b41c104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad43e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0fa6b0 0x55b41c104af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58906/0 (socket says 172.21.15.116:58906) 2024-03-04T07:50:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad43e00640 1 -- 172.21.15.116:0/914451481 learned_addr learned my addr 172.21.15.116:0/914451481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad43400640 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad3be00640 1 -- 172.21.15.116:0/914451481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 msgr2=0x55b41c105fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad3be00640 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c105fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad3be00640 1 -- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0fa6b0 msgr2=0x55b41c104af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad3be00640 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0fa6b0 0x55b41c104af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad3be00640 1 -- 172.21.15.116:0/914451481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b41c105030 con 0x55b41c0f5540 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad43e00640 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0fa6b0 0x55b41c104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad3be00640 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0f5540 0x55b41c1045b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fad2c02e040 tx=0x7fad2c02cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d6b5fc4733528 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.800+0000 7fad31600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad31600640 1 -- 172.21.15.116:0/914451481 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c0d4030 con 0x55b41c0f5540 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad31600640 1 -- 172.21.15.116:0/914451481 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fad2c004c60 con 0x55b41c0f5540 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad2c0f2da0 con 0x55b41c0f5540 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad31600640 1 -- 172.21.15.116:0/914451481 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad2c04b070 con 0x55b41c0f5540 2024-03-04T07:50:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 10 client.17682 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55b41c16f6d0 con 0x55b41c0f5540 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad31600640 1 -- 172.21.15.116:0/914451481 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad2c0f2da0 con 0x55b41c0f5540 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad31600640 1 -- 172.21.15.116:0/914451481 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fad2c0bb9d0 con 0x55b41c0f5540 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 10 client.17682 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 10 client.17682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 20 client.17682 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c16fd10 0x55b41c172130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 4 client.17682 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.801+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b41c0fed00 con 0x55b41c16fd10 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad43400640 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c16fd10 0x55b41c172130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad43400640 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c16fd10 0x55b41c172130 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c9c3a3716f44e14 server_cookie=cc8815d8c2877c5e in_seq=0 out_seq=0 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad31600640 10 client.17682 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad31600640 1 -- 172.21.15.116:0/914451481 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b41c0fed00 con 0x55b41c16fd10 2024-03-04T07:50:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad31600640 10 client.17682 handle_command_reply: tid=0 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad4d417740 1 client.17682 shutdown 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0f5540 msgr2=0x55b41c1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.802+0000 7fad4d417740 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0f5540 0x55b41c1045b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fad2c02e040 tx=0x7fad2c02cf90 comp rx=0 tx=0).stop 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 shutdown_connections 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55b41c16fd10 0x55b41c172130 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55b41c0fa6b0 0x55b41c104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b41c0f5540 0x55b41c1045b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad31600640 0 client.17682 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 --2- 172.21.15.116:0/914451481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b41c0f4b50 0x55b41c105fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 >> 172.21.15.116:0/914451481 conn(0x55b41c0f00c0 msgr2=0x55b41c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 shutdown_connections 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 1 -- 172.21.15.116:0/914451481 wait complete. 2024-03-04T07:50:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4d417740 20 client.17682 trim_cache size 0 max 16384 2024-03-04T07:50:32.813 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:50:32.813 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:50:32.813 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:50:32.813 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fad240a1e00 msgr2=0x7fad240a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4a600640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fad240a1e00 0x7fad240a42c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55b41c0f5940 tx=0x7fad34005f90 comp rx=0 tx=0).stop 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 msgr2=0x7fad4419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.803+0000 7fad4a600640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad4419a0d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fad2c02c8f0 tx=0x7fad2c02c660 comp rx=0 tx=0).stop 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 shutdown_connections 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad441053c0 0x7fad441a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fad44104a00 0x7fad4419a0d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fad240a1e00 0x7fad240a42c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 --2- 172.21.15.116:0/2435808266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad44103810 0x7fad44199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 >> 172.21.15.116:0/2435808266 conn(0x7fad440fed80 msgr2=0x7fad44107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 shutdown_connections 2024-03-04T07:50:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:32.804+0000 7fad4a600640 1 -- 172.21.15.116:0/2435808266 wait complete. 2024-03-04T07:50:32.822 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:50:32.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:50:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 -- 172.21.15.116:0/2619434360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c107c10 msgr2=0x7f133c108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 --2- 172.21.15.116:0/2619434360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c107c10 0x7f133c108030 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1330009900 tx=0x7f133002bc30 comp rx=0 tx=0).stop 2024-03-04T07:50:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 -- 172.21.15.116:0/2619434360 shutdown_connections 2024-03-04T07:50:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 --2- 172.21.15.116:0/2619434360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 0x7f133c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 --2- 172.21.15.116:0/2619434360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f133c108e00 0x7f133c109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 --2- 172.21.15.116:0/2619434360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c107c10 0x7f133c108030 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 -- 172.21.15.116:0/2619434360 >> 172.21.15.116:0/2619434360 conn(0x7f133c076be0 msgr2=0x7f133c079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 -- 172.21.15.116:0/2619434360 shutdown_connections 2024-03-04T07:50:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.019+0000 7f1344800640 1 -- 172.21.15.116:0/2619434360 wait complete. 2024-03-04T07:50:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.020+0000 7f1344800640 1 Processor -- start 2024-03-04T07:50:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1344800640 1 -- start start 2024-03-04T07:50:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1344800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f133c107c10 0x7f133c19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1344800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c108e00 0x7f133c19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1344800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 0x7f133c1a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1344800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c1138d0 con 0x7f133c108e00 2024-03-04T07:50:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1344800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c113750 con 0x7f133c1097c0 2024-03-04T07:50:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1344800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c113a50 con 0x7f133c107c10 2024-03-04T07:50:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1341400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c108e00 0x7f133c19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1342800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 0x7f133c1a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1341400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c108e00 0x7f133c19e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38414/0 (socket says 172.21.15.116:38414) 2024-03-04T07:50:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1342800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 0x7f133c1a54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58934/0 (socket says 172.21.15.116:58934) 2024-03-04T07:50:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1341e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f133c107c10 0x7f133c19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.021+0000 7f1341400640 1 -- 172.21.15.116:0/97345470 learned_addr learned my addr 172.21.15.116:0/97345470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1342800640 1 -- 172.21.15.116:0/97345470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f133c107c10 msgr2=0x7f133c19df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1342800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f133c107c10 0x7f133c19df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1342800640 1 -- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c108e00 msgr2=0x7f133c19e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1342800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c108e00 0x7f133c19e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1342800640 1 -- 172.21.15.116:0/97345470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c1a5b60 con 0x7f133c1097c0 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1342800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 0x7f133c1a54f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f133800ec50 tx=0x7f133800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1336a00640 1 -- 172.21.15.116:0/97345470 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338003b70 con 0x7f133c1097c0 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1336a00640 1 -- 172.21.15.116:0/97345470 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1338003d10 con 0x7f133c1097c0 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c1a5e50 con 0x7f133c1097c0 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1336a00640 1 -- 172.21.15.116:0/97345470 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338010890 con 0x7f133c1097c0 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.022+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c1a62c0 con 0x7f133c1097c0 2024-03-04T07:50:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.024+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c10ef30 con 0x7f133c1097c0 2024-03-04T07:50:33.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.025+0000 7f1336a00640 1 -- 172.21.15.116:0/97345470 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1338003fc0 con 0x7f133c1097c0 2024-03-04T07:50:33.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.026+0000 7f1336a00640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13240a1d80 0x7f13240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.027+0000 7f1336a00640 1 -- 172.21.15.116:0/97345470 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f13380ce310 con 0x7f133c1097c0 2024-03-04T07:50:33.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.027+0000 7f1341e00640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13240a1d80 0x7f13240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.028+0000 7f1341e00640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13240a1d80 0x7f13240a4240 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f133002c170 tx=0x7f133002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.031+0000 7f1336a00640 1 -- 172.21.15.116:0/97345470 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f133809b740 con 0x7f133c1097c0 2024-03-04T07:50:33.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:32 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:50:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:32 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:32 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:50:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:32 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:33.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:32 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:50:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.269+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f133c02cff0 con 0x7f133c1097c0 2024-03-04T07:50:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:32 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:50:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:32 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:32 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:50:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:32 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:33.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:32 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:50:33.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.270+0000 7f1336a00640 1 -- 172.21.15.116:0/97345470 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f133809b040 con 0x7f133c1097c0 2024-03-04T07:50:33.272 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:50:33.272 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:50:33.272 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:50:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13240a1d80 msgr2=0x7f13240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13240a1d80 0x7f13240a4240 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f133002c170 tx=0x7f133002df90 comp rx=0 tx=0).stop 2024-03-04T07:50:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 msgr2=0x7f133c1a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 0x7f133c1a54f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f133800ec50 tx=0x7f133800c6a0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 shutdown_connections 2024-03-04T07:50:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f133c1097c0 0x7f133c1a54f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f133c108e00 0x7f133c19e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f13240a1d80 0x7f13240a4240 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 --2- 172.21.15.116:0/97345470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f133c107c10 0x7f133c19df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 >> 172.21.15.116:0/97345470 conn(0x7f133c076be0 msgr2=0x7f133c10c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.273+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 shutdown_connections 2024-03-04T07:50:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.274+0000 7f1344800640 1 -- 172.21.15.116:0/97345470 wait complete. 2024-03-04T07:50:33.294 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:50:33.294 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:50:33.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:32 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:50:33.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:32 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:33.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:32 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:50:33.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:32 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:33.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:32 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:50:33.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 -- 172.21.15.116:0/46263162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 msgr2=0x7f333c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 --2- 172.21.15.116:0/46263162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 0x7f333c104e60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3324009900 tx=0x7f332402bc30 comp rx=0 tx=0).stop 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 -- 172.21.15.116:0/46263162 shutdown_connections 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 --2- 172.21.15.116:0/46263162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f333c1053a0 0x7f333c10bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 --2- 172.21.15.116:0/46263162 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 0x7f333c104e60 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 --2- 172.21.15.116:0/46263162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 0x7f333c103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 -- 172.21.15.116:0/46263162 >> 172.21.15.116:0/46263162 conn(0x7f333c0fed80 msgr2=0x7f333c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 -- 172.21.15.116:0/46263162 shutdown_connections 2024-03-04T07:50:33.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.501+0000 7f3341400640 1 -- 172.21.15.116:0/46263162 wait complete. 2024-03-04T07:50:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.502+0000 7f3341400640 1 Processor -- start 2024-03-04T07:50:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.502+0000 7f3341400640 1 -- start start 2024-03-04T07:50:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f3341400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 0x7f333c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f3341400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 0x7f333c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f3341400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f333c1053a0 0x7f333c1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f3341400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c10f470 con 0x7f333c1037f0 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f3341400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c10f2f0 con 0x7f333c1049e0 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f3341400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c10f5f0 con 0x7f333c1053a0 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f333a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 0x7f333c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f333a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 0x7f333c199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38440/0 (socket says 172.21.15.116:38440) 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f333a800640 1 -- 172.21.15.116:0/4069780196 learned_addr learned my addr 172.21.15.116:0/4069780196 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f3333e00640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 0x7f333c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.503+0000 7f333b200640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f333c1053a0 0x7f333c1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f333a800640 1 -- 172.21.15.116:0/4069780196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f333c1053a0 msgr2=0x7f333c1a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f333a800640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f333c1053a0 0x7f333c1a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f333a800640 1 -- 172.21.15.116:0/4069780196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 msgr2=0x7f333c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f333a800640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 0x7f333c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f333a800640 1 -- 172.21.15.116:0/4069780196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c1a1730 con 0x7f333c1037f0 2024-03-04T07:50:33.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f3333e00640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 0x7f333c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:33.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f333a800640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 0x7f333c199b10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f332c00e9c0 tx=0x7f332c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f3333400640 1 -- 172.21.15.116:0/4069780196 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c00cd80 con 0x7f333c1037f0 2024-03-04T07:50:33.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c1a1a20 con 0x7f333c1037f0 2024-03-04T07:50:33.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f3333400640 1 -- 172.21.15.116:0/4069780196 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f332c0035f0 con 0x7f333c1037f0 2024-03-04T07:50:33.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f3333400640 1 -- 172.21.15.116:0/4069780196 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c018630 con 0x7f333c1037f0 2024-03-04T07:50:33.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.504+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333c1a1f90 con 0x7f333c1037f0 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.507+0000 7f3333400640 1 -- 172.21.15.116:0/4069780196 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f332c0037a0 con 0x7f333c1037f0 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.507+0000 7f33441aa740 1 Processor -- start 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f33441aa740 1 -- start start 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f33441aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc75fca0 0x555cfc7600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f33441aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc765190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f33441aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc767c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f33441aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cfc6c4d60 con 0x555cfc765800 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f33441aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cfc66dcf0 con 0x555cfc760690 2024-03-04T07:50:33.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f33441aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cfc6c0e00 con 0x555cfc75fca0 2024-03-04T07:50:33.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f3333e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc767c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f333a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc765190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f333b200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc75fca0 0x555cfc7600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f333b200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc75fca0 0x555cfc7600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53308/0 (socket says 172.21.15.116:53308) 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f333a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc765190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58966/0 (socket says 172.21.15.116:58966) 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.508+0000 7f333b200640 1 -- 172.21.15.116:0/2200110813 learned_addr learned my addr 172.21.15.116:0/2200110813 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f333a800640 1 -- 172.21.15.116:0/2200110813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc75fca0 msgr2=0x555cfc7600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f333a800640 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc75fca0 0x555cfc7600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f333a800640 1 -- 172.21.15.116:0/2200110813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 msgr2=0x555cfc767c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f333a800640 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc767c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f333a800640 1 -- 172.21.15.116:0/2200110813 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cfc3d93f0 con 0x555cfc760690 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f333b200640 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc75fca0 0x555cfc7600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f333a800640 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc765190 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f332c0cbb60 tx=0x7f332c0d61a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40aee31cb0c684af server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f3333400640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33200a1d80 0x7f33200a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.509+0000 7f332b400640 1 -- 172.21.15.116:0/2200110813 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c0e1070 con 0x555cfc760690 2024-03-04T07:50:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.510+0000 7f332b400640 1 -- 172.21.15.116:0/2200110813 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f332c09b4f0 con 0x555cfc760690 2024-03-04T07:50:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.510+0000 7f332b400640 1 -- 172.21.15.116:0/2200110813 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c09a750 con 0x555cfc760690 2024-03-04T07:50:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.510+0000 7f3333400640 1 -- 172.21.15.116:0/4069780196 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f332c0ce100 con 0x7f333c1037f0 2024-03-04T07:50:33.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.510+0000 7f333b200640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33200a1d80 0x7f33200a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.510+0000 7f33441aa740 1 -- 172.21.15.116:0/2200110813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 msgr2=0x555cfc765190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.510+0000 7f333b200640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33200a1d80 0x7f33200a4240 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x555cfc760ef0 tx=0x7f3334009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.510+0000 7f33441aa740 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc765190 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f332c0cbb60 tx=0x7f332c0d61a0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 -- 172.21.15.116:0/2200110813 shutdown_connections 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc767c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc765190 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 --2- 172.21.15.116:0/2200110813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc75fca0 0x555cfc7600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 -- 172.21.15.116:0/2200110813 >> 172.21.15.116:0/2200110813 conn(0x555cfc75b210 msgr2=0x555cfc75b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 -- 172.21.15.116:0/2200110813 shutdown_connections 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 -- 172.21.15.116:0/2200110813 wait complete. 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 -- start start 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.511+0000 7f33441aa740 1 Processor -- start 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f33441aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc76a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f33441aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc76a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f33441aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc765800 0x555cfc771710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f33441aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cfc7687b0 con 0x555cfc75fca0 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f33441aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cfc663220 con 0x555cfc765800 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f33441aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cfc768930 con 0x555cfc760690 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f33441aa740 10 client.0 _ll_register_callbacks cb 0x555cfc3d8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc76a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc765800 0x555cfc771710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc765800 0x555cfc771710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58972/0 (socket says 172.21.15.116:58972) 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 -- 172.21.15.116:0/403747475 learned_addr learned my addr 172.21.15.116:0/403747475 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f3333e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc76a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc76a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38456/0 (socket says 172.21.15.116:38456) 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 -- 172.21.15.116:0/403747475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 msgr2=0x555cfc76a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc76a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 -- 172.21.15.116:0/403747475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 msgr2=0x555cfc76a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc76a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.512+0000 7f333b200640 1 -- 172.21.15.116:0/403747475 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cfc4a37d0 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f333b200640 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc765800 0x555cfc771710 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3334003bc0 tx=0x7f3334004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3de290a0d3bb6c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f331fe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f331fe00640 1 -- 172.21.15.116:0/403747475 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3334004310 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f332c09b4f0 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f331fe00640 1 -- 172.21.15.116:0/403747475 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f33340044b0 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f331fe00640 1 -- 172.21.15.116:0/403747475 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33340046b0 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f33441aa740 10 client.27842 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.513+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555cfc774b40 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f331fe00640 1 -- 172.21.15.116:0/403747475 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f332c09b4f0 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f331fe00640 1 -- 172.21.15.116:0/403747475 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3334026070 con 0x555cfc765800 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f33441aa740 10 client.27842 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f33441aa740 10 client.27842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f33441aa740 20 client.27842 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f33441aa740 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc776180 0x555cfc7785e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f33441aa740 4 client.27842 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555cfc6d22a0 con 0x555cfc776180 2024-03-04T07:50:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.514+0000 7f333a800640 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc776180 0x555cfc7785e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f333a800640 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc776180 0x555cfc7785e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cea21414ab397c0 server_cookie=99ed69e4fbd334a1 in_seq=0 out_seq=0 2024-03-04T07:50:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f331fe00640 10 client.27842 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:33.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f331fe00640 1 -- 172.21.15.116:0/403747475 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555cfc6d22a0 con 0x555cfc776180 2024-03-04T07:50:33.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f331fe00640 10 client.27842 handle_command_reply: tid=0 2024-03-04T07:50:33.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 client.27842 shutdown 2024-03-04T07:50:33.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc765800 msgr2=0x555cfc771710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc765800 0x555cfc771710 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3334003bc0 tx=0x7f3334004ab0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 shutdown_connections 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc776180 0x555cfc7785e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc765800 0x555cfc771710 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc76a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 --2- 172.21.15.116:0/403747475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc76a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f331fe00640 0 client.27842 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.515+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 >> 172.21.15.116:0/403747475 conn(0x555cfc75b210 msgr2=0x555cfc75a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.516+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 shutdown_connections 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.516+0000 7f33441aa740 1 -- 172.21.15.116:0/403747475 wait complete. 2024-03-04T07:50:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.516+0000 7f33441aa740 20 client.27842 trim_cache size 0 max 16384 2024-03-04T07:50:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 Processor -- start 2024-03-04T07:50:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 -- start start 2024-03-04T07:50:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc75fca0 0x555cfc75a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc77cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1a8d60 con 0x555cfc765800 2024-03-04T07:50:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1ab340 con 0x555cfc75fca0 2024-03-04T07:50:33.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.529+0000 7f33441aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1a9360 con 0x555cfc760690 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc75fca0 0x555cfc75a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc75fca0 0x555cfc75a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58980/0 (socket says 172.21.15.116:58980) 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333a800640 1 -- 172.21.15.116:0/2431720774 learned_addr learned my addr 172.21.15.116:0/2431720774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333b200640 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc77cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f3333e00640 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333a800640 1 -- 172.21.15.116:0/2431720774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 msgr2=0x555cfc7716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333a800640 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333a800640 1 -- 172.21.15.116:0/2431720774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 msgr2=0x555cfc77cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.530+0000 7f333a800640 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc77cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.531+0000 7f333a800640 1 -- 172.21.15.116:0/2431720774 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cfc75b020 con 0x555cfc75fca0 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.531+0000 7f333b200640 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc77cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.531+0000 7f3333e00640 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.531+0000 7f333a800640 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc75fca0 0x555cfc75a330 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f332c0058c0 tx=0x7f332c0d1b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ab841da6335be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.531+0000 7f332aa00640 1 -- 172.21.15.116:0/2431720774 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c013b60 con 0x555cfc75fca0 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.531+0000 7f332aa00640 1 -- 172.21.15.116:0/2431720774 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f332c09b4f0 con 0x555cfc75fca0 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.531+0000 7f332aa00640 1 -- 172.21.15.116:0/2431720774 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c013d50 con 0x555cfc75fca0 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 -- 172.21.15.116:0/2431720774 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc75fca0 msgr2=0x555cfc75a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc75fca0 0x555cfc75a330 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f332c0058c0 tx=0x7f332c0d1b40 comp rx=0 tx=0).stop 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 -- 172.21.15.116:0/2431720774 shutdown_connections 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc765800 0x555cfc77cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 --2- 172.21.15.116:0/2431720774 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc75fca0 0x555cfc75a330 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 -- 172.21.15.116:0/2431720774 >> 172.21.15.116:0/2431720774 conn(0x555cfc75b210 msgr2=0x555cfc759c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 -- 172.21.15.116:0/2431720774 shutdown_connections 2024-03-04T07:50:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 -- 172.21.15.116:0/2431720774 wait complete. 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 -- start start 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.532+0000 7f33441aa740 1 Processor -- start 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f33441aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc771190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f33441aa740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f33441aa740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc765800 0x555cfc76f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f33441aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1a99c0 con 0x555cfc75fca0 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f33441aa740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1aace0 con 0x555cfc760690 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f33441aa740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1aa020 con 0x555cfc765800 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f33441aa740 10 client.0 _ll_register_callbacks cb 0x7f3334011e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f333a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc771190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f3333e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f3333e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:58996/0 (socket says 172.21.15.116:58996) 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f333a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc771190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38474/0 (socket says 172.21.15.116:38474) 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f3333e00640 1 -- 172.21.15.116:0/3516264658 learned_addr learned my addr 172.21.15.116:0/3516264658 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.533+0000 7f333b200640 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc765800 0x555cfc76f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f3333e00640 1 -- 172.21.15.116:0/3516264658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc765800 msgr2=0x555cfc76f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f3333e00640 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc765800 0x555cfc76f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f3333e00640 1 -- 172.21.15.116:0/3516264658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 msgr2=0x555cfc771190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f3333e00640 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc771190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f3333e00640 1 -- 172.21.15.116:0/3516264658 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cfc7700c0 con 0x555cfc760690 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f333b200640 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc765800 0x555cfc76f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:33.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f3333e00640 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc7716d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f3324009900 tx=0x7f3324005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13f668da5ce274ca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f331fe00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f331fe00640 1 -- 172.21.15.116:0/3516264658 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324005d50 con 0x555cfc760690 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f331fe00640 1 -- 172.21.15.116:0/3516264658 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f33240046c0 con 0x555cfc760690 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f332c09b4f0 con 0x555cfc760690 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f331fe00640 1 -- 172.21.15.116:0/3516264658 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33240048c0 con 0x555cfc760690 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f33441aa740 10 client.27848 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.534+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555cfc7daf40 con 0x555cfc760690 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f331fe00640 1 -- 172.21.15.116:0/3516264658 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f332c09b4f0 con 0x555cfc760690 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f331fe00640 1 -- 172.21.15.116:0/3516264658 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3324034440 con 0x555cfc760690 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f33441aa740 10 client.27848 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f33441aa740 10 client.27848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f33441aa740 20 client.27848 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:33.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f33441aa740 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc7db920 0x555cfc7ddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f33441aa740 4 client.27848 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555cfc6cc5f0 con 0x555cfc7db920 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.535+0000 7f333a800640 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc7db920 0x555cfc7ddd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f333a800640 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc7db920 0x555cfc7ddd40 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85cd4cb2bb99d7e3 server_cookie=56e96cca922671c8 in_seq=0 out_seq=0 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f331fe00640 10 client.27848 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f331fe00640 1 -- 172.21.15.116:0/3516264658 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555cfc6cc5f0 con 0x555cfc7db920 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f331fe00640 10 client.27848 handle_command_reply: tid=0 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 client.27848 shutdown 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 msgr2=0x555cfc7716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc7716d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f3324009900 tx=0x7f3324005270 comp rx=0 tx=0).stop 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 shutdown_connections 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555cfc7db920 0x555cfc7ddd40 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555cfc765800 0x555cfc76f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555cfc760690 0x555cfc7716d0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f331fe00640 0 client.27848 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 --2- 172.21.15.116:0/3516264658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555cfc75fca0 0x555cfc771190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 >> 172.21.15.116:0/3516264658 conn(0x555cfc75b210 msgr2=0x555cfc75a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 shutdown_connections 2024-03-04T07:50:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 1 -- 172.21.15.116:0/3516264658 wait complete. 2024-03-04T07:50:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.536+0000 7f33441aa740 20 client.27848 trim_cache size 0 max 16384 2024-03-04T07:50:33.543 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33200a1d80 msgr2=0x7f33200a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33200a1d80 0x7f33200a4240 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x555cfc760ef0 tx=0x7f3334009500 comp rx=0 tx=0).stop 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 msgr2=0x7f333c199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 0x7f333c199b10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f332c00e9c0 tx=0x7f332c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 shutdown_connections 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f33200a1d80 0x7f33200a4240 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f333c1053a0 0x7f333c1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f333c1049e0 0x7f333c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 --2- 172.21.15.116:0/4069780196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c1037f0 0x7f333c199b10 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 >> 172.21.15.116:0/4069780196 conn(0x7f333c0fed80 msgr2=0x7f333c107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 shutdown_connections 2024-03-04T07:50:33.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:33.537+0000 7f3341400640 1 -- 172.21.15.116:0/4069780196 wait complete. 2024-03-04T07:50:34.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:33 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/97345470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:50:34.382 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:33 smithi116 ceph-mon[32150]: pgmap v695: 97 pgs: 97 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:34.383 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:33 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/97345470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:50:34.383 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:33 smithi133 ceph-mon[36505]: pgmap v695: 97 pgs: 97 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:34.383 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:33 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/97345470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:50:34.383 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:33 smithi152 ceph-mon[37154]: pgmap v695: 97 pgs: 97 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:34.556 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:50:34.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:50:34.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:50:34.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 -- 172.21.15.116:0/2342193329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 msgr2=0x7f17f0108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 --2- 172.21.15.116:0/2342193329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f0108010 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009900 tx=0x7f17e002bc40 comp rx=0 tx=0).stop 2024-03-04T07:50:34.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 -- 172.21.15.116:0/2342193329 shutdown_connections 2024-03-04T07:50:34.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 --2- 172.21.15.116:0/2342193329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17f01097a0 0x7f17f0110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 --2- 172.21.15.116:0/2342193329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17f0108de0 0x7f17f0109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 --2- 172.21.15.116:0/2342193329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f0108010 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 -- 172.21.15.116:0/2342193329 >> 172.21.15.116:0/2342193329 conn(0x7f17f0076b90 msgr2=0x7f17f0078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:34.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 -- 172.21.15.116:0/2342193329 shutdown_connections 2024-03-04T07:50:34.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.782+0000 7f17f6400640 1 -- 172.21.15.116:0/2342193329 wait complete. 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 Processor -- start 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 -- start start 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17f0108de0 0x7f17f019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17f01097a0 0x7f17f01a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0113890 con 0x7f17f01097a0 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0113710 con 0x7f17f0107bf0 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.783+0000 7f17f6400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0113a10 con 0x7f17f0108de0 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f019df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59022/0 (socket says 172.21.15.116:59022) 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 -- 172.21.15.116:0/104946667 learned_addr learned my addr 172.21.15.116:0/104946667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17efe00640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17f01097a0 0x7f17f01a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17e7e00640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17f0108de0 0x7f17f019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 -- 172.21.15.116:0/104946667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17f0108de0 msgr2=0x7f17f019e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17f0108de0 0x7f17f019e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 -- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17f01097a0 msgr2=0x7f17f01a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17f01097a0 0x7f17f01a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 -- 172.21.15.116:0/104946667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f01a5b20 con 0x7f17f0107bf0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17efe00640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17f01097a0 0x7f17f01a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.784+0000 7f17ef400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f019df00 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f17e002efd0 tx=0x7f17e0005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.785+0000 7f17ecc00640 1 -- 172.21.15.116:0/104946667 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0041070 con 0x7f17f0107bf0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.785+0000 7f17ecc00640 1 -- 172.21.15.116:0/104946667 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17e00044d0 con 0x7f17f0107bf0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.785+0000 7f17ecc00640 1 -- 172.21.15.116:0/104946667 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0034440 con 0x7f17f0107bf0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.785+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f01a5db0 con 0x7f17f0107bf0 2024-03-04T07:50:34.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.785+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f01a6200 con 0x7f17f0107bf0 2024-03-04T07:50:34.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17ecc00640 1 -- 172.21.15.116:0/104946667 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f17e0044070 con 0x7f17f0107bf0 2024-03-04T07:50:34.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 Processor -- start 2024-03-04T07:50:34.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 -- start start 2024-03-04T07:50:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf7b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf805c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589bde8ee20 con 0x5589bdf80c30 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589bdee0a20 con 0x5589bdf7bac0 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.788+0000 7f17f91a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589bde89b00 con 0x5589bdf7b0d0 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf805c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf805c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59038/0 (socket says 172.21.15.116:59038) 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 -- 172.21.15.116:0/4009327493 learned_addr learned my addr 172.21.15.116:0/4009327493 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17efe00640 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf7b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 -- 172.21.15.116:0/4009327493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 msgr2=0x5589bdf7b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf7b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 -- 172.21.15.116:0/4009327493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 msgr2=0x5589bdf830c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf830c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 -- 172.21.15.116:0/4009327493 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589bde9ee10 con 0x5589bdf7bac0 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17e7e00640 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf830c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ecc00640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f17d40a1e00 0x7f17d40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.789+0000 7f17ef400640 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf805c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f17e00f5a20 tx=0x7f17e00f5b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d16ce591431df0d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17ecc00640 1 -- 172.21.15.116:0/104946667 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f17e00ef9e0 con 0x7f17f0107bf0 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17efe00640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f17d40a1e00 0x7f17d40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17e4c00640 1 -- 172.21.15.116:0/4009327493 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e00b89b0 con 0x5589bdf7bac0 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17e4c00640 1 -- 172.21.15.116:0/4009327493 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17e00bce10 con 0x5589bdf7bac0 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17e4c00640 1 -- 172.21.15.116:0/4009327493 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e00bc070 con 0x5589bdf7bac0 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17f91a8740 1 -- 172.21.15.116:0/4009327493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 msgr2=0x5589bdf805c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17f91a8740 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf805c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f17e00f5a20 tx=0x7f17e00f5b10 comp rx=0 tx=0).stop 2024-03-04T07:50:34.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17efe00640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f17d40a1e00 0x7f17d40a42c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x5589bdf7c320 tx=0x7f17e8009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17f91a8740 1 -- 172.21.15.116:0/4009327493 shutdown_connections 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17f91a8740 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf830c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17f91a8740 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf805c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17f91a8740 1 --2- 172.21.15.116:0/4009327493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf7b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.790+0000 7f17f91a8740 1 -- 172.21.15.116:0/4009327493 >> 172.21.15.116:0/4009327493 conn(0x5589bdf76640 msgr2=0x5589bdf76a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.791+0000 7f17f91a8740 1 -- 172.21.15.116:0/4009327493 shutdown_connections 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.791+0000 7f17f91a8740 1 -- 172.21.15.116:0/4009327493 wait complete. 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.791+0000 7f17f91a8740 1 -- start start 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.791+0000 7f17f91a8740 1 Processor -- start 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf851b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf856f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf8c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17f91a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589bde9cf40 con 0x5589bdf80c30 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17f91a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589bdedcac0 con 0x5589bdf7bac0 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17f91a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589bdf839a0 con 0x5589bdf7b0d0 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17f91a8740 10 client.0 _ll_register_callbacks cb 0x5589bdedbaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf8c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17ef400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf851b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf856f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf8c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38516/0 (socket says 172.21.15.116:38516) 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 -- 172.21.15.116:0/238904419 learned_addr learned my addr 172.21.15.116:0/238904419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 -- 172.21.15.116:0/238904419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 msgr2=0x5589bdf851b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf851b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 -- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 msgr2=0x5589bdf856f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf856f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17efe00640 1 -- 172.21.15.116:0/238904419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589bdcbd7e0 con 0x5589bdf80c30 2024-03-04T07:50:34.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.792+0000 7f17e7e00640 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf856f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17efe00640 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf8c730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f17e8011c80 tx=0x7f17e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf7f137361ce7f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17dd600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17dd600640 1 -- 172.21.15.116:0/238904419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8020420 con 0x5589bdf80c30 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17dd600640 1 -- 172.21.15.116:0/238904419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f17e80205c0 con 0x5589bdf80c30 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17e00bce10 con 0x5589bdf80c30 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17dd600640 1 -- 172.21.15.116:0/238904419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17e80207c0 con 0x5589bdf80c30 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17f91a8740 10 client.17718 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.793+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5589bdd1df70 con 0x5589bdf80c30 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17dd600640 1 -- 172.21.15.116:0/238904419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17e00bce10 con 0x5589bdf80c30 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17dd600640 1 -- 172.21.15.116:0/238904419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f17e800f040 con 0x5589bdf80c30 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17f91a8740 10 client.17718 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17f91a8740 10 client.17718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:34.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17f91a8740 20 client.17718 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17f91a8740 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdf911d0 0x5589bdf935f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17f91a8740 4 client.17718 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589bdeedf60 con 0x5589bdf911d0 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.794+0000 7f17ef400640 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdf911d0 0x5589bdf935f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.795+0000 7f17ef400640 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdf911d0 0x5589bdf935f0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb76ef680d1faad server_cookie=92c1b3170aa4ebba in_seq=0 out_seq=0 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.795+0000 7f17dd600640 10 client.17718 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.795+0000 7f17dd600640 1 -- 172.21.15.116:0/238904419 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5589bdeedf60 con 0x5589bdf911d0 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.795+0000 7f17dd600640 10 client.17718 handle_command_reply: tid=0 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.795+0000 7f17f91a8740 1 client.17718 shutdown 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 msgr2=0x5589bdf8c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf8c730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f17e8011c80 tx=0x7f17e8004a40 comp rx=0 tx=0).stop 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 shutdown_connections 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdf911d0 0x5589bdf935f0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf80c30 0x5589bdf8c730 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf7bac0 0x5589bdf856f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 --2- 172.21.15.116:0/238904419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7b0d0 0x5589bdf851b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17dd600640 0 client.17718 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:34.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 >> 172.21.15.116:0/238904419 conn(0x5589bdf76640 msgr2=0x5589bdf77890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:34.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 shutdown_connections 2024-03-04T07:50:34.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 1 -- 172.21.15.116:0/238904419 wait complete. 2024-03-04T07:50:34.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.796+0000 7f17f91a8740 20 client.17718 trim_cache size 0 max 16384 2024-03-04T07:50:34.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.813+0000 7f17f91a8740 1 Processor -- start 2024-03-04T07:50:34.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17f91a8740 1 -- start start 2024-03-04T07:50:34.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf75a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf79fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17f91a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f01ad020 con 0x5589bdf7b0d0 2024-03-04T07:50:34.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17f91a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f01af600 con 0x5589bdf80c30 2024-03-04T07:50:34.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17f91a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f01ad620 con 0x5589bdf7bac0 2024-03-04T07:50:34.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17efe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf79fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53410/0 (socket says 172.21.15.116:53410) 2024-03-04T07:50:34.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17e7e00640 1 -- 172.21.15.116:0/3681400322 learned_addr learned my addr 172.21.15.116:0/3681400322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:34.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.814+0000 7f17ef400640 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf75a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17efe00640 1 -- 172.21.15.116:0/3681400322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 msgr2=0x5589bdf8c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17efe00640 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17efe00640 1 -- 172.21.15.116:0/3681400322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 msgr2=0x5589bdf75a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17efe00640 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf75a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17efe00640 1 -- 172.21.15.116:0/3681400322 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589bdbb5170 con 0x5589bdf80c30 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17efe00640 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf79fb0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f17e801dba0 tx=0x7f17e8005f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5fd28d3c71a429 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17dfe00640 1 -- 172.21.15.116:0/3681400322 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8037070 con 0x5589bdf80c30 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17dfe00640 1 -- 172.21.15.116:0/3681400322 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17e801fb50 con 0x5589bdf80c30 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.815+0000 7f17dfe00640 1 -- 172.21.15.116:0/3681400322 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e801d7f0 con 0x5589bdf80c30 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 -- 172.21.15.116:0/3681400322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 msgr2=0x5589bdf79fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf79fb0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f17e801dba0 tx=0x7f17e8005f50 comp rx=0 tx=0).stop 2024-03-04T07:50:34.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 -- 172.21.15.116:0/3681400322 shutdown_connections 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf79fb0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 --2- 172.21.15.116:0/3681400322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf75a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 -- 172.21.15.116:0/3681400322 >> 172.21.15.116:0/3681400322 conn(0x5589bdf76640 msgr2=0x5589bdf798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 -- 172.21.15.116:0/3681400322 shutdown_connections 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 -- 172.21.15.116:0/3681400322 wait complete. 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.816+0000 7f17f91a8740 1 -- start start 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 1 Processor -- start 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf8a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf8b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f01adc80 con 0x5589bdf7b0d0 2024-03-04T07:50:34.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f01aefa0 con 0x5589bdf80c30 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f01ae2e0 con 0x5589bdf7bac0 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17f91a8740 10 client.0 _ll_register_callbacks cb 0x7f17e8009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17ef400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf8a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17ef400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf8a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38530/0 (socket says 172.21.15.116:38530) 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17ef400640 1 -- 172.21.15.116:0/2361435807 learned_addr learned my addr 172.21.15.116:0/2361435807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17efe00640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf8b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.817+0000 7f17e7e00640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17efe00640 1 -- 172.21.15.116:0/2361435807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 msgr2=0x5589bdf8a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17efe00640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17efe00640 1 -- 172.21.15.116:0/2361435807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 msgr2=0x5589bdf8a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17efe00640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf8a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17efe00640 1 -- 172.21.15.116:0/2361435807 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589bdf8be40 con 0x5589bdf80c30 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17ef400640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf8a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17efe00640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf8b850 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f17e801daa0 tx=0x7f17e8011ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25c30e003a8cb608 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:34.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17dd600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:34.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17dd600640 1 -- 172.21.15.116:0/2361435807 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8037040 con 0x5589bdf80c30 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17e801fb50 con 0x5589bdf80c30 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.818+0000 7f17dd600640 1 -- 172.21.15.116:0/2361435807 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f17e8004c00 con 0x5589bdf80c30 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17dd600640 1 -- 172.21.15.116:0/2361435807 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17e8004e40 con 0x5589bdf80c30 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 10 client.27866 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5589bdfecfc0 con 0x5589bdf80c30 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17dd600640 1 -- 172.21.15.116:0/2361435807 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17e801fb50 con 0x5589bdf80c30 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17dd600640 1 -- 172.21.15.116:0/2361435807 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f17e8005500 con 0x5589bdf80c30 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 10 client.27866 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 10 client.27866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 20 client.27866 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdfed7b0 0x5589bdfefbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 4 client.27866 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5589bdee82b0 con 0x5589bdfed7b0 2024-03-04T07:50:34.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.819+0000 7f17ef400640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdfed7b0 0x5589bdfefbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.820+0000 7f17dd600640 10 client.27866 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.820+0000 7f17ef400640 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdfed7b0 0x5589bdfefbd0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bedd5aa0715f24dd server_cookie=5da74ca52e963f01 in_seq=0 out_seq=0 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17dd600640 1 -- 172.21.15.116:0/2361435807 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5589bdee82b0 con 0x5589bdfed7b0 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17dd600640 10 client.27866 handle_command_reply: tid=0 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 client.27866 shutdown 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 msgr2=0x5589bdf8b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf8b850 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f17e801daa0 tx=0x7f17e8011ee0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 shutdown_connections 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5589bdfed7b0 0x5589bdfefbd0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5589bdf80c30 0x5589bdf8b850 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589bdf7bac0 0x5589bdf8a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 --2- 172.21.15.116:0/2361435807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5589bdf7b0d0 0x5589bdf8a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 >> 172.21.15.116:0/2361435807 conn(0x5589bdf76640 msgr2=0x5589bdf798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:34.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17dd600640 0 client.27866 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 shutdown_connections 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.821+0000 7f17f91a8740 1 -- 172.21.15.116:0/2361435807 wait complete. 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f91a8740 20 client.27866 trim_cache size 0 max 16384 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "3293ac7f-cf44-42fb-8e58-94559bc5671a", 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:50:34.826 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:50:34.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f17d40a1e00 msgr2=0x7f17d40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f17d40a1e00 0x7f17d40a42c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x5589bdf7c320 tx=0x7f17e8009800 comp rx=0 tx=0).stop 2024-03-04T07:50:34.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 msgr2=0x7f17f019df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:34.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f019df00 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f17e002efd0 tx=0x7f17e0005d50 comp rx=0 tx=0).stop 2024-03-04T07:50:34.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 shutdown_connections 2024-03-04T07:50:34.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17f01097a0 0x7f17f01a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f17d40a1e00 0x7f17d40a42c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17f0108de0 0x7f17f019e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 --2- 172.21.15.116:0/104946667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17f0107bf0 0x7f17f019df00 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:34.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 >> 172.21.15.116:0/104946667 conn(0x7f17f0076b90 msgr2=0x7f17f010c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:34.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 shutdown_connections 2024-03-04T07:50:34.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:34.822+0000 7f17f6400640 1 -- 172.21.15.116:0/104946667 wait complete. 2024-03-04T07:50:34.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:50:34.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:50:35.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 -- 172.21.15.116:0/3272544055 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0075f90 msgr2=0x7f94d00763b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3272544055 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0075f90 0x7f94d00763b0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009860 tx=0x7f94c002bb70 comp rx=0 tx=0).stop 2024-03-04T07:50:35.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 -- 172.21.15.116:0/3272544055 shutdown_connections 2024-03-04T07:50:35.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3272544055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d0077b70 0x7f94d0078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3272544055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d00771d0 0x7f94d0077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3272544055 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0075f90 0x7f94d00763b0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 -- 172.21.15.116:0/3272544055 >> 172.21.15.116:0/3272544055 conn(0x7f94d00fed80 msgr2=0x7f94d01011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:35.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 -- 172.21.15.116:0/3272544055 shutdown_connections 2024-03-04T07:50:35.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.047+0000 7f94d5c00640 1 -- 172.21.15.116:0/3272544055 wait complete. 2024-03-04T07:50:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.048+0000 7f94d5c00640 1 Processor -- start 2024-03-04T07:50:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.048+0000 7f94d5c00640 1 -- start start 2024-03-04T07:50:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94d5c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d0075f90 0x7f94d019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94d5c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d00771d0 0x7f94d019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94d5c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0077b70 0x7f94d01a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94d5c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01138a0 con 0x7f94d00771d0 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94d5c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d0113720 con 0x7f94d0077b70 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94d5c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d0113a20 con 0x7f94d0075f90 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d00771d0 0x7f94d019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d0075f90 0x7f94d019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d0075f90 0x7f94d019df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53428/0 (socket says 172.21.15.116:53428) 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 -- 172.21.15.116:0/3311599331 learned_addr learned my addr 172.21.15.116:0/3311599331 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cf400640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0077b70 0x7f94d01a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 -- 172.21.15.116:0/3311599331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0077b70 msgr2=0x7f94d01a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0077b70 0x7f94d01a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 -- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d00771d0 msgr2=0x7f94d019e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d00771d0 0x7f94d019e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.049+0000 7f94cea00640 1 -- 172.21.15.116:0/3311599331 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d01a5b70 con 0x7f94d0075f90 2024-03-04T07:50:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.050+0000 7f94cea00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d0075f90 0x7f94d019df50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f94c00349e0 tx=0x7f94c0032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.050+0000 7f94c7400640 1 -- 172.21.15.116:0/3311599331 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c0041030 con 0x7f94d0075f90 2024-03-04T07:50:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.050+0000 7f94c7400640 1 -- 172.21.15.116:0/3311599331 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94c002c510 con 0x7f94d0075f90 2024-03-04T07:50:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.050+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d01a5e00 con 0x7f94d0075f90 2024-03-04T07:50:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.050+0000 7f94c7400640 1 -- 172.21.15.116:0/3311599331 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c0039040 con 0x7f94d0075f90 2024-03-04T07:50:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.050+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d01a6200 con 0x7f94d0075f90 2024-03-04T07:50:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.053+0000 7f94c7400640 1 -- 172.21.15.116:0/3311599331 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f94c0044070 con 0x7f94d0075f90 2024-03-04T07:50:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 Processor -- start 2024-03-04T07:50:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 -- start start 2024-03-04T07:50:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91cd6b0 0x5644e91cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e90f5090 con 0x5644e91cd6b0 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e912cd30 con 0x5644e91c8540 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94d89bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e90d5a30 con 0x5644e91c7b50 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94cea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91cd6b0 0x5644e91cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94cf400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94cea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59088/0 (socket says 172.21.15.116:59088) 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94cea00640 1 -- 172.21.15.116:0/1639183426 learned_addr learned my addr 172.21.15.116:0/1639183426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94cf400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:53436/0 (socket says 172.21.15.116:53436) 2024-03-04T07:50:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.054+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91cd6b0 0x5644e91cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38560/0 (socket says 172.21.15.116:38560) 2024-03-04T07:50:35.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94cea00640 1 -- 172.21.15.116:0/1639183426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 msgr2=0x5644e91c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94cea00640 1 --2- 172.21.15.116:0/1639183426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94cea00640 1 -- 172.21.15.116:0/1639183426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91cd6b0 msgr2=0x5644e91cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94cea00640 1 --2- 172.21.15.116:0/1639183426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91cd6b0 0x5644e91cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94cea00640 1 -- 172.21.15.116:0/1639183426 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644e8f0b7d0 con 0x5644e91c8540 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94cea00640 1 --2- 172.21.15.116:0/1639183426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91cd040 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f94c00f72d0 tx=0x7f94c00f73c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce8df2d3584dd000 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94bfe00640 1 -- 172.21.15.116:0/1639183426 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c00f10d0 con 0x5644e91c8540 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94bfe00640 1 -- 172.21.15.116:0/1639183426 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94c00fc030 con 0x5644e91c8540 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.055+0000 7f94bfe00640 1 -- 172.21.15.116:0/1639183426 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c01006b0 con 0x5644e91c8540 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94c7400640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94b40a1d80 0x7f94b40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94c7400640 1 -- 172.21.15.116:0/3311599331 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f94c00ef2e0 con 0x7f94d0075f90 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94cf400640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94b40a1d80 0x7f94b40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 -- 172.21.15.116:0/1639183426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 msgr2=0x5644e91cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1639183426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91cd040 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f94c00f72d0 tx=0x7f94c00f73c0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 -- 172.21.15.116:0/1639183426 shutdown_connections 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1639183426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91cd6b0 0x5644e91cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1639183426 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91cd040 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1639183426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 -- 172.21.15.116:0/1639183426 >> 172.21.15.116:0/1639183426 conn(0x5644e91c30c0 msgr2=0x5644e91c34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 -- 172.21.15.116:0/1639183426 shutdown_connections 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.056+0000 7f94d89bd740 1 -- 172.21.15.116:0/1639183426 wait complete. 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.057+0000 7f94cf400640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94b40a1d80 0x7f94b40a4240 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x5644e91c8da0 tx=0x7f94c8010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.057+0000 7f94d89bd740 1 -- start start 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.057+0000 7f94d89bd740 1 Processor -- start 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91d1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91cd6b0 0x5644e91d9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94d89bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e90cb0c0 con 0x5644e91c7b50 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94d89bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e9128dd0 con 0x5644e91c8540 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94d89bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644e912cd30 con 0x5644e91cd6b0 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94d89bd740 10 client.0 _ll_register_callbacks cb 0x5644e90e8860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94cea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91d1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91d1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59092/0 (socket says 172.21.15.116:59092) 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 -- 172.21.15.116:0/490222772 learned_addr learned my addr 172.21.15.116:0/490222772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94cf400640 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91cd6b0 0x5644e91d9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 -- 172.21.15.116:0/490222772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91cd6b0 msgr2=0x5644e91d9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91cd6b0 0x5644e91d9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 -- 172.21.15.116:0/490222772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 msgr2=0x5644e91d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.058+0000 7f94c7e00640 1 -- 172.21.15.116:0/490222772 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644e8e413f0 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94c7e00640 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91d1fd0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f94b8012720 tx=0x7f94b8012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aecf5f3225aadaf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94bcc00640 1 -- 172.21.15.116:0/490222772 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b800f920 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94bcc00640 1 -- 172.21.15.116:0/490222772 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f94b800fac0 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94c00fc030 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94bcc00640 1 -- 172.21.15.116:0/490222772 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94b800fcc0 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94d89bd740 10 client.27878 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5644e8e010f0 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94bcc00640 1 -- 172.21.15.116:0/490222772 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94c00fc030 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.059+0000 7f94bcc00640 1 -- 172.21.15.116:0/490222772 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f94b801c440 con 0x5644e91c8540 2024-03-04T07:50:35.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94d89bd740 10 client.27878 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:35.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94d89bd740 10 client.27878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:35.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94d89bd740 20 client.27878 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:35.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94d89bd740 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e91dd4a0 0x5644e91df900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94d89bd740 4 client.27878 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:35.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644e913a270 con 0x5644e91dd4a0 2024-03-04T07:50:35.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94cea00640 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e91dd4a0 0x5644e91df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94cea00640 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e91dd4a0 0x5644e91df900 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff32f70ed96e5b server_cookie=ecd0e1195794e2ec in_seq=0 out_seq=0 2024-03-04T07:50:35.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.060+0000 7f94bcc00640 10 client.27878 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:35.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94bcc00640 1 -- 172.21.15.116:0/490222772 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5644e913a270 con 0x5644e91dd4a0 2024-03-04T07:50:35.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94bcc00640 10 client.27878 handle_command_reply: tid=0 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 client.27878 shutdown 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 msgr2=0x5644e91d1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91d1fd0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f94b8012720 tx=0x7f94b8012be0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 shutdown_connections 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e91dd4a0 0x5644e91df900 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91cd6b0 0x5644e91d9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91c8540 0x5644e91d1fd0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 --2- 172.21.15.116:0/490222772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94bcc00640 0 client.27878 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 >> 172.21.15.116:0/490222772 conn(0x5644e91c30c0 msgr2=0x5644e91c2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 shutdown_connections 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.061+0000 7f94d89bd740 1 -- 172.21.15.116:0/490222772 wait complete. 2024-03-04T07:50:35.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.062+0000 7f94d89bd740 20 client.27878 trim_cache size 0 max 16384 2024-03-04T07:50:35.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 Processor -- start 2024-03-04T07:50:35.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 -- start start 2024-03-04T07:50:35.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c8540 0x5644e91d8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01ad0a0 con 0x5644e91c8540 2024-03-04T07:50:35.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01af680 con 0x5644e91cd6b0 2024-03-04T07:50:35.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.079+0000 7f94d89bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01ad6a0 con 0x5644e91c7b50 2024-03-04T07:50:35.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c8540 0x5644e91d8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94cf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c8540 0x5644e91d8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38582/0 (socket says 172.21.15.116:38582) 2024-03-04T07:50:35.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 -- 172.21.15.116:0/2209764149 learned_addr learned my addr 172.21.15.116:0/2209764149 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94cea00640 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 -- 172.21.15.116:0/2209764149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 msgr2=0x5644e91d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 -- 172.21.15.116:0/2209764149 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 msgr2=0x5644e91d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 -- 172.21.15.116:0/2209764149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644e91dd240 con 0x5644e91c8540 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94cf400640 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.080+0000 7f94c7e00640 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c8540 0x5644e91d8fd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f94b8000c00 tx=0x7f94b800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a1177f919ad45e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94bf400640 1 -- 172.21.15.116:0/2209764149 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b801caa0 con 0x5644e91c8540 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94bf400640 1 -- 172.21.15.116:0/2209764149 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94b802b030 con 0x5644e91c8540 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94bf400640 1 -- 172.21.15.116:0/2209764149 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b80052d0 con 0x5644e91c8540 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 -- 172.21.15.116:0/2209764149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c8540 msgr2=0x5644e91d8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c8540 0x5644e91d8fd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f94b8000c00 tx=0x7f94b800f790 comp rx=0 tx=0).stop 2024-03-04T07:50:35.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 -- 172.21.15.116:0/2209764149 shutdown_connections 2024-03-04T07:50:35.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c8540 0x5644e91d8fd0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 --2- 172.21.15.116:0/2209764149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c7b50 0x5644e91d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 -- 172.21.15.116:0/2209764149 >> 172.21.15.116:0/2209764149 conn(0x5644e91c30c0 msgr2=0x5644e91d0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:35.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.081+0000 7f94d89bd740 1 -- 172.21.15.116:0/2209764149 shutdown_connections 2024-03-04T07:50:35.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 -- 172.21.15.116:0/2209764149 wait complete. 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 -- start start 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 Processor -- start 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c8540 0x5644e91d7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01add00 con 0x5644e91c7b50 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01af020 con 0x5644e91cd6b0 2024-03-04T07:50:35.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.082+0000 7f94d89bd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d01ae360 con 0x5644e91c8540 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94d89bd740 10 client.0 _ll_register_callbacks cb 0x7f94b8009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38596/0 (socket says 172.21.15.116:38596) 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 -- 172.21.15.116:0/1054288114 learned_addr learned my addr 172.21.15.116:0/1054288114 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cf400640 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94c7e00640 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c8540 0x5644e91d7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 -- 172.21.15.116:0/1054288114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c8540 msgr2=0x5644e91d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c8540 0x5644e91d7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 -- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 msgr2=0x5644e91d8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 -- 172.21.15.116:0/1054288114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644e9238dd0 con 0x5644e91c7b50 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94cea00640 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d7800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009830 tx=0x7f94c00f2a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3a8868fcbb84bc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:35.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94bcc00640 1 -- 172.21.15.116:0/1054288114 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c00b8270 con 0x5644e91c7b50 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94bcc00640 1 -- 172.21.15.116:0/1054288114 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f94c00f7e00 con 0x5644e91c7b50 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.083+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94b802b030 con 0x5644e91c7b50 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94bcc00640 1 -- 172.21.15.116:0/1054288114 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94c002c6c0 con 0x5644e91c7b50 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 10 client.17742 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5644e923a120 con 0x5644e91c7b50 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94bcc00640 1 -- 172.21.15.116:0/1054288114 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94b802b030 con 0x5644e91c7b50 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94bcc00640 1 -- 172.21.15.116:0/1054288114 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f94c004e060 con 0x5644e91c7b50 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 10 client.17742 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 10 client.17742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 20 client.17742 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e923a820 0x5644e923cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:35.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 4 client.17742 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5644e91345c0 con 0x5644e923a820 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.084+0000 7f94c7e00640 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e923a820 0x5644e923cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94c7e00640 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e923a820 0x5644e923cc40 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c25865e369b3f474 server_cookie=1469228ab31173c5 in_seq=0 out_seq=0 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94bcc00640 10 client.17742 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94bcc00640 1 -- 172.21.15.116:0/1054288114 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5644e91345c0 con 0x5644e923a820 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94bcc00640 10 client.17742 handle_command_reply: tid=0 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 client.17742 shutdown 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 msgr2=0x5644e91d7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d7800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009830 tx=0x7f94c00f2a10 comp rx=0 tx=0).stop 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 shutdown_connections 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5644e923a820 0x5644e923cc40 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5644e91cd6b0 0x5644e91d8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5644e91c8540 0x5644e91d7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 --2- 172.21.15.116:0/1054288114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644e91c7b50 0x5644e91d7800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94bcc00640 0 client.17742 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 >> 172.21.15.116:0/1054288114 conn(0x5644e91c30c0 msgr2=0x5644e91c2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 shutdown_connections 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.085+0000 7f94d89bd740 1 -- 172.21.15.116:0/1054288114 wait complete. 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d89bd740 20 client.17742 trim_cache size 0 max 16384 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:50:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "3293ac7f-cf44-42fb-8e58-94559bc5671a": { 2024-03-04T07:50:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:50:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "3293ac7f-cf44-42fb-8e58-94559bc5671a", 2024-03-04T07:50:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:50:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:50:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:50:35.093 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94b40a1d80 msgr2=0x7f94b40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94b40a1d80 0x7f94b40a4240 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x5644e91c8da0 tx=0x7f94c8010040 comp rx=0 tx=0).stop 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d0075f90 msgr2=0x7f94d019df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d0075f90 0x7f94d019df50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f94c00349e0 tx=0x7f94c0032450 comp rx=0 tx=0).stop 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 shutdown_connections 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f94d0077b70 0x7f94d01a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d00771d0 0x7f94d019e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f94b40a1d80 0x7f94b40a4240 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 --2- 172.21.15.116:0/3311599331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94d0075f90 0x7f94d019df50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 >> 172.21.15.116:0/3311599331 conn(0x7f94d00fed80 msgr2=0x7f94d0100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:35.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 shutdown_connections 2024-03-04T07:50:35.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:50:35.086+0000 7f94d5c00640 1 -- 172.21.15.116:0/3311599331 wait complete. 2024-03-04T07:50:35.103 INFO:tasks.cephfs.filesystem:scrub status for tag:3293ac7f-cf44-42fb-8e58-94559bc5671a - {'path': '/', 'tag': '3293ac7f-cf44-42fb-8e58-94559bc5671a', 'options': 'recursive,force'} 2024-03-04T07:50:35.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:34 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:50:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:34 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:34 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:50:35.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:34 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:34 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:50:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:34 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:34 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:50:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:34 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:35.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:34 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:50:35.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:34 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:35.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:34 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:50:35.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:34 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:50:36.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:35 smithi116 ceph-mon[32150]: pgmap v696: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:50:36.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:35 smithi152 ceph-mon[37154]: pgmap v696: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:50:36.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:35 smithi133 ceph-mon[36505]: pgmap v696: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:50:38.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:37 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:50:38.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:37 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:50:38.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:37 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:50:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 -- 172.21.15.116:0/1285658084 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c107c30 msgr2=0x7f852c108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 --2- 172.21.15.116:0/1285658084 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c107c30 0x7f852c108050 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f851c009900 tx=0x7f851c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 -- 172.21.15.116:0/1285658084 shutdown_connections 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 --2- 172.21.15.116:0/1285658084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c1097e0 0x7f852c110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 --2- 172.21.15.116:0/1285658084 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f852c108e20 0x7f852c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 --2- 172.21.15.116:0/1285658084 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c107c30 0x7f852c108050 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 -- 172.21.15.116:0/1285658084 >> 172.21.15.116:0/1285658084 conn(0x7f852c076be0 msgr2=0x7f852c079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 -- 172.21.15.116:0/1285658084 shutdown_connections 2024-03-04T07:50:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.775+0000 7f8531000640 1 -- 172.21.15.116:0/1285658084 wait complete. 2024-03-04T07:50:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 Processor -- start 2024-03-04T07:50:38.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 -- start start 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f852c107c30 0x7f852c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c108e20 0x7f852c19e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c1097e0 0x7f852c1a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c113880 con 0x7f852c1097e0 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c113700 con 0x7f852c108e20 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.777+0000 7f8531000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c113a00 con 0x7f852c107c30 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c108e20 0x7f852c19e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c108e20 0x7f852c19e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41208/0 (socket says 172.21.15.116:41208) 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 -- 172.21.15.116:0/2180646139 learned_addr learned my addr 172.21.15.116:0/2180646139 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f852a800640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c1097e0 0x7f852c1a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8529e00640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f852c107c30 0x7f852c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 -- 172.21.15.116:0/2180646139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f852c107c30 msgr2=0x7f852c19de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f852c107c30 0x7f852c19de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 -- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c1097e0 msgr2=0x7f852c1a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c1097e0 0x7f852c1a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 -- 172.21.15.116:0/2180646139 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c1a5a80 con 0x7f852c108e20 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.778+0000 7f8523e00640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c108e20 0x7f852c19e3a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f851400ea60 tx=0x7f851400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.779+0000 7f8522a00640 1 -- 172.21.15.116:0/2180646139 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851400cde0 con 0x7f852c108e20 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.779+0000 7f8522a00640 1 -- 172.21.15.116:0/2180646139 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85140035f0 con 0x7f852c108e20 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.779+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c1a5d70 con 0x7f852c108e20 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.779+0000 7f8522a00640 1 -- 172.21.15.116:0/2180646139 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514010670 con 0x7f852c108e20 2024-03-04T07:50:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.779+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c1a6280 con 0x7f852c108e20 2024-03-04T07:50:38.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 Processor -- start 2024-03-04T07:50:38.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 -- start start 2024-03-04T07:50:38.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6979c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc697ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8522a00640 1 -- 172.21.15.116:0/2180646139 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f85140037a0 con 0x7f852c108e20 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6981840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc68df370 con 0x55bbc697a240 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc6888550 con 0x55bbc697f3b0 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.782+0000 7f8533c2c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc68db410 con 0x55bbc6979850 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc697ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8523e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6981840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc697ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59508/0 (socket says 172.21.15.116:59508) 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 -- 172.21.15.116:0/956476078 learned_addr learned my addr 172.21.15.116:0/956476078 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f852a800640 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6979c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 -- 172.21.15.116:0/956476078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 msgr2=0x55bbc6979c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6979c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 -- 172.21.15.116:0/956476078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 msgr2=0x55bbc6981840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6981840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 -- 172.21.15.116:0/956476078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbc689c3c0 con 0x55bbc697a240 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f852a800640 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6979c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8523e00640 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6981840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.783+0000 7f8529e00640 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc697ed40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f851c009900 tx=0x7f851c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37bb49b4f30decf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f851b400640 1 -- 172.21.15.116:0/956476078 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c02cb80 con 0x55bbc697a240 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f851b400640 1 -- 172.21.15.116:0/956476078 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f851c005040 con 0x55bbc697a240 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f851b400640 1 -- 172.21.15.116:0/956476078 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c004920 con 0x55bbc697a240 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8522a00640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f85100a1ea0 0x7f85100a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8522a00640 1 -- 172.21.15.116:0/2180646139 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f85140cdfe0 con 0x7f852c108e20 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f852a800640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f85100a1ea0 0x7f85100a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8533c2c740 1 -- 172.21.15.116:0/956476078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 msgr2=0x55bbc697ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8533c2c740 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc697ed40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f851c009900 tx=0x7f851c009560 comp rx=0 tx=0).stop 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8533c2c740 1 -- 172.21.15.116:0/956476078 shutdown_connections 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8533c2c740 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6981840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8533c2c740 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc697ed40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8533c2c740 1 --2- 172.21.15.116:0/956476078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6979c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.784+0000 7f8533c2c740 1 -- 172.21.15.116:0/956476078 >> 172.21.15.116:0/956476078 conn(0x55bbc6974dc0 msgr2=0x55bbc69751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.785+0000 7f8533c2c740 1 -- 172.21.15.116:0/956476078 shutdown_connections 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.785+0000 7f8533c2c740 1 -- 172.21.15.116:0/956476078 wait complete. 2024-03-04T07:50:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.785+0000 7f852a800640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f85100a1ea0 0x7f85100a4360 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55bbc697aaa0 tx=0x7f8524009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.785+0000 7f8533c2c740 1 -- start start 2024-03-04T07:50:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.785+0000 7f8533c2c740 1 Processor -- start 2024-03-04T07:50:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6983b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc6984060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8533c2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc6982300 con 0x55bbc697a240 2024-03-04T07:50:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8533c2c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc689b9b0 con 0x55bbc697f3b0 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8533c2c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbc6982480 con 0x55bbc6979850 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8533c2c740 10 client.0 _ll_register_callbacks cb 0x55bbc68d7340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8523e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc6984060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f852a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8523e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc6984060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59524/0 (socket says 172.21.15.116:59524) 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8523e00640 1 -- 172.21.15.116:0/2423617360 learned_addr learned my addr 172.21.15.116:0/2423617360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.786+0000 7f8529e00640 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6983b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8523e00640 1 -- 172.21.15.116:0/2423617360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 msgr2=0x55bbc6983b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8523e00640 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6983b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8523e00640 1 -- 172.21.15.116:0/2423617360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 msgr2=0x55bbc698b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8523e00640 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8523e00640 1 -- 172.21.15.116:0/2423617360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbc65f4210 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8523e00640 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc6984060 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f85140976a0 tx=0x7f8514097030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb332bb1cc36157 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f850fe00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f850fe00640 1 -- 172.21.15.116:0/2423617360 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851409b3d0 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f850fe00640 1 -- 172.21.15.116:0/2423617360 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f8514097340 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f851c005040 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f850fe00640 1 -- 172.21.15.116:0/2423617360 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f851409acd0 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8533c2c740 10 client.17754 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bbc66c0620 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.787+0000 7f850fe00640 1 -- 172.21.15.116:0/2423617360 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f851c005040 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f850fe00640 1 -- 172.21.15.116:0/2423617360 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f851409a120 con 0x55bbc697a240 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f8533c2c740 10 client.17754 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f8533c2c740 10 client.17754 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f8533c2c740 20 client.17754 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f8533c2c740 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc698f8e0 0x55bbc6991d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f8533c2c740 4 client.17754 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbc68ec8b0 con 0x55bbc698f8e0 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.788+0000 7f8529e00640 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc698f8e0 0x55bbc6991d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.789+0000 7f850fe00640 10 client.17754 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.789+0000 7f8529e00640 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc698f8e0 0x55bbc6991d10 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f702e46861bb4f37 server_cookie=8178a60329d5d637 in_seq=0 out_seq=0 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.789+0000 7f850fe00640 1 -- 172.21.15.116:0/2423617360 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f851c0504b0 con 0x55bbc698f8e0 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.789+0000 7f850fe00640 10 client.17754 handle_command_reply: tid=0 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.789+0000 7f8533c2c740 1 client.17754 shutdown 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 msgr2=0x55bbc6984060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc6984060 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f85140976a0 tx=0x7f8514097030 comp rx=0 tx=0).stop 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 shutdown_connections 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc698f8e0 0x55bbc6991d10 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc6984060 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 --2- 172.21.15.116:0/2423617360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6983b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f850fe00640 0 client.17754 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 >> 172.21.15.116:0/2423617360 conn(0x55bbc6974dc0 msgr2=0x55bbc6973fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 shutdown_connections 2024-03-04T07:50:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 1 -- 172.21.15.116:0/2423617360 wait complete. 2024-03-04T07:50:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.790+0000 7f8533c2c740 20 client.17754 trim_cache size 0 max 16384 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 Processor -- start 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 -- start start 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc6979850 0x55bbc6973750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc697a240 0x55bbc698b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6973c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1acfd0 con 0x55bbc6979850 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1af5b0 con 0x55bbc697f3b0 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8533c2c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1ad5d0 con 0x55bbc697a240 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8529e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc6979850 0x55bbc6973750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8529e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc6979850 0x55bbc6973750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59540/0 (socket says 172.21.15.116:59540) 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f8529e00640 1 -- 172.21.15.116:0/1817192079 learned_addr learned my addr 172.21.15.116:0/1817192079 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.805+0000 7f852a800640 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6973c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f8523e00640 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc697a240 0x55bbc698b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f8529e00640 1 -- 172.21.15.116:0/1817192079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc697a240 msgr2=0x55bbc698b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f8529e00640 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc697a240 0x55bbc698b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f8529e00640 1 -- 172.21.15.116:0/1817192079 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 msgr2=0x55bbc6973c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f8529e00640 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6973c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f8529e00640 1 -- 172.21.15.116:0/1817192079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbc697aad0 con 0x55bbc6979850 2024-03-04T07:50:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f8529e00640 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc6979850 0x55bbc6973750 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f851c02c170 tx=0x7f851c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fc730f37dca1a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f851aa00640 1 -- 172.21.15.116:0/1817192079 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c005d50 con 0x55bbc6979850 2024-03-04T07:50:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f851aa00640 1 -- 172.21.15.116:0/1817192079 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f851c04b040 con 0x55bbc6979850 2024-03-04T07:50:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.806+0000 7f851aa00640 1 -- 172.21.15.116:0/1817192079 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c0046c0 con 0x55bbc6979850 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 -- 172.21.15.116:0/1817192079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc6979850 msgr2=0x55bbc6973750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc6979850 0x55bbc6973750 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f851c02c170 tx=0x7f851c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 -- 172.21.15.116:0/1817192079 shutdown_connections 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc6973c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc697a240 0x55bbc698b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1817192079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc6979850 0x55bbc6973750 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 -- 172.21.15.116:0/1817192079 >> 172.21.15.116:0/1817192079 conn(0x55bbc6974dc0 msgr2=0x55bbc6983b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 -- 172.21.15.116:0/1817192079 shutdown_connections 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 -- 172.21.15.116:0/1817192079 wait complete. 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.807+0000 7f8533c2c740 1 -- start start 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 1 Processor -- start 2024-03-04T07:50:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6988d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc69892d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1adc30 con 0x55bbc697a240 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1aef50 con 0x55bbc697f3b0 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1ae290 con 0x55bbc6979850 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8533c2c740 10 client.0 _ll_register_callbacks cb 0x7f851400ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f852a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8523e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc69892d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f852a800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41254/0 (socket says 172.21.15.116:41254) 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8523e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc69892d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59546/0 (socket says 172.21.15.116:59546) 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f852a800640 1 -- 172.21.15.116:0/1786377036 learned_addr learned my addr 172.21.15.116:0/1786377036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.808+0000 7f8529e00640 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6988d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8523e00640 1 -- 172.21.15.116:0/1786377036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 msgr2=0x55bbc6988d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8523e00640 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6988d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8523e00640 1 -- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 msgr2=0x55bbc698a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8523e00640 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8523e00640 1 -- 172.21.15.116:0/1786377036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbc698a860 con 0x55bbc697a240 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f852a800640 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8523e00640 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc69892d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f851409a9d0 tx=0x7f8514003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b991c49ebbc2fe80 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f850fe00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f850fe00640 1 -- 172.21.15.116:0/1786377036 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851409a810 con 0x55bbc697a240 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f850fe00640 1 -- 172.21.15.116:0/1786377036 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f85140d1d90 con 0x55bbc697a240 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f851c04b040 con 0x55bbc697a240 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f850fe00640 1 -- 172.21.15.116:0/1786377036 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f851409a330 con 0x55bbc697a240 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8533c2c740 10 client.17760 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.809+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55bbc69eb840 con 0x55bbc697a240 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f850fe00640 1 -- 172.21.15.116:0/1786377036 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f851c04b040 con 0x55bbc697a240 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f850fe00640 1 -- 172.21.15.116:0/1786377036 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8514004000 con 0x55bbc697a240 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f8533c2c740 10 client.17760 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f8533c2c740 10 client.17760 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f8533c2c740 20 client.17760 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc69ec140 0x55bbc69ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f8533c2c740 4 client.17760 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bbc697f120 con 0x55bbc69ec140 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.810+0000 7f8529e00640 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc69ec140 0x55bbc69ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.811+0000 7f8529e00640 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc69ec140 0x55bbc69ee560 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59ce855fa3f41078 server_cookie=951b7772a8128316 in_seq=0 out_seq=0 2024-03-04T07:50:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.811+0000 7f850fe00640 10 client.17760 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f850fe00640 1 -- 172.21.15.116:0/1786377036 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bbc697f120 con 0x55bbc69ec140 2024-03-04T07:50:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f850fe00640 10 client.17760 handle_command_reply: tid=0 2024-03-04T07:50:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 client.17760 shutdown 2024-03-04T07:50:38.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 msgr2=0x55bbc69892d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc69892d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f851409a9d0 tx=0x7f8514003d60 comp rx=0 tx=0).stop 2024-03-04T07:50:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 shutdown_connections 2024-03-04T07:50:38.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55bbc69ec140 0x55bbc69ee560 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55bbc697f3b0 0x55bbc698a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbc697a240 0x55bbc69892d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 --2- 172.21.15.116:0/1786377036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bbc6979850 0x55bbc6988d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 >> 172.21.15.116:0/1786377036 conn(0x55bbc6974dc0 msgr2=0x55bbc6978020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f850fe00640 0 client.17760 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 shutdown_connections 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 1 -- 172.21.15.116:0/1786377036 wait complete. 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.826+0000 7f8533c2c740 20 client.17760 trim_cache size 0 max 16384 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:50:38.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:50:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:50:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:50:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f85100a1ea0 msgr2=0x7f85100a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f85100a1ea0 0x7f85100a4360 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55bbc697aaa0 tx=0x7f8524009500 comp rx=0 tx=0).stop 2024-03-04T07:50:38.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c108e20 msgr2=0x7f852c19e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c108e20 0x7f852c19e3a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f851400ea60 tx=0x7f851400ef20 comp rx=0 tx=0).stop 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 shutdown_connections 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f852c1097e0 0x7f852c1a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f852c108e20 0x7f852c19e3a0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f85100a1ea0 0x7f85100a4360 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 --2- 172.21.15.116:0/2180646139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f852c107c30 0x7f852c19de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 >> 172.21.15.116:0/2180646139 conn(0x7f852c076be0 msgr2=0x7f852c10c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 shutdown_connections 2024-03-04T07:50:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:38.827+0000 7f8531000640 1 -- 172.21.15.116:0/2180646139 wait complete. 2024-03-04T07:50:39.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:38 smithi116 ceph-mon[32150]: pgmap v697: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:50:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:38 smithi152 ceph-mon[37154]: pgmap v697: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:50:39.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:38 smithi133 ceph-mon[36505]: pgmap v697: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-03-04T07:50:41.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:40 smithi116 ceph-mon[32150]: pgmap v698: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:50:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:40 smithi152 ceph-mon[37154]: pgmap v698: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:50:41.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:40 smithi133 ceph-mon[36505]: pgmap v698: 97 pgs: 97 active+clean; 3.4 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:50:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:42 smithi116 ceph-mon[32150]: pgmap v699: 97 pgs: 97 active+clean; 3.5 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-03-04T07:50:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:42 smithi152 ceph-mon[37154]: pgmap v699: 97 pgs: 97 active+clean; 3.5 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-03-04T07:50:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:42 smithi133 ceph-mon[36505]: pgmap v699: 97 pgs: 97 active+clean; 3.5 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 KiB/s wr, 15 op/s 2024-03-04T07:50:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:44 smithi116 ceph-mon[32150]: pgmap v700: 97 pgs: 97 active+clean; 3.5 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:50:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:44 smithi152 ceph-mon[37154]: pgmap v700: 97 pgs: 97 active+clean; 3.5 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:50:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:44 smithi133 ceph-mon[36505]: pgmap v700: 97 pgs: 97 active+clean; 3.5 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 12 op/s 2024-03-04T07:50:47.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:46 smithi116 ceph-mon[32150]: pgmap v701: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-04T07:50:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:46 smithi152 ceph-mon[37154]: pgmap v701: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-04T07:50:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:46 smithi133 ceph-mon[36505]: pgmap v701: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-04T07:50:48.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:47 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:50:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:47 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:47 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:47 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:47 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:50:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:47 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:47 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:47 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:47 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:50:48.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:47 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:47 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:48.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:47 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:48 smithi133 ceph-mon[36505]: pgmap v702: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:50:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:48 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:48 smithi116 ceph-mon[32150]: pgmap v702: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:50:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:48 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:48 smithi152 ceph-mon[37154]: pgmap v702: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:50:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:48 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:50.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:50.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:50 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:50.759 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:50 smithi116 ceph-mon[32150]: pgmap v703: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:50 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:50.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:50 smithi152 ceph-mon[37154]: pgmap v703: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:50 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:50.837 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:50 smithi133 ceph-mon[36505]: pgmap v703: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:50:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 -- 172.21.15.116:0/57646761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80ffe60 msgr2=0x7f1fb8100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 --2- 172.21.15.116:0/57646761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8100280 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009900 tx=0x7f1fa802bc50 comp rx=0 tx=0).stop 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 -- 172.21.15.116:0/57646761 shutdown_connections 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 --2- 172.21.15.116:0/57646761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fb8101160 0x7f1fb81015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 --2- 172.21.15.116:0/57646761 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb81007c0 0x7f1fb8100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 --2- 172.21.15.116:0/57646761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8100280 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 -- 172.21.15.116:0/57646761 >> 172.21.15.116:0/57646761 conn(0x7f1fb80fba60 msgr2=0x7f1fb80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 -- 172.21.15.116:0/57646761 shutdown_connections 2024-03-04T07:50:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.961+0000 7f1fbea00640 1 -- 172.21.15.116:0/57646761 wait complete. 2024-03-04T07:50:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.962+0000 7f1fbea00640 1 Processor -- start 2024-03-04T07:50:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbea00640 1 -- start start 2024-03-04T07:50:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81007c0 0x7f1fb819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbea00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fb8101160 0x7f1fb81a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbea00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb810f430 con 0x7f1fb81007c0 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbea00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb810f2b0 con 0x7f1fb80ffe60 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbea00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb810f5b0 con 0x7f1fb8101160 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38142/0 (socket says 172.21.15.116:38142) 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fb7e00640 1 -- 172.21.15.116:0/1424425381 learned_addr learned my addr 172.21.15.116:0/1424425381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fb7400640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81007c0 0x7f1fb819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.963+0000 7f1fbca00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fb8101160 0x7f1fb81a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb7e00640 1 -- 172.21.15.116:0/1424425381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fb8101160 msgr2=0x7f1fb81a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fb8101160 0x7f1fb81a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb7e00640 1 -- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81007c0 msgr2=0x7f1fb819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81007c0 0x7f1fb819a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb7e00640 1 -- 172.21.15.116:0/1424425381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb81a1700 con 0x7f1fb80ffe60 2024-03-04T07:50:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8199ae0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009230 tx=0x7f1fa8005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb4c00640 1 -- 172.21.15.116:0/1424425381 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8042020 con 0x7f1fb80ffe60 2024-03-04T07:50:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb4c00640 1 -- 172.21.15.116:0/1424425381 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fa80044d0 con 0x7f1fb80ffe60 2024-03-04T07:50:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fb4c00640 1 -- 172.21.15.116:0/1424425381 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa803e460 con 0x7f1fb80ffe60 2024-03-04T07:50:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb81a1990 con 0x7f1fb80ffe60 2024-03-04T07:50:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.964+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb81a1e90 con 0x7f1fb80ffe60 2024-03-04T07:50:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.967+0000 7f1fb4c00640 1 -- 172.21.15.116:0/1424425381 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1fa8045020 con 0x7f1fb80ffe60 2024-03-04T07:50:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.968+0000 7f1fb4c00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1f9c0a1ea0 0x7f1f9c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fb4c00640 1 -- 172.21.15.116:0/1424425381 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1fa80efda0 con 0x7f1fb80ffe60 2024-03-04T07:50:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fb7400640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1f9c0a1ea0 0x7f1f9c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 Processor -- start 2024-03-04T07:50:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 -- start start 2024-03-04T07:50:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fb7400640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1f9c0a1ea0 0x7f1f9c0a4360 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40095f0 tx=0x7f1fa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081c850 0x555d4081cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d40821d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d40824840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d40782370 con 0x555d4081c850 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4072b550 con 0x555d4081d240 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.969+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4077e410 con 0x555d408223b0 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081c850 0x555d4081cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d40821d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081c850 0x555d4081cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51912/0 (socket says 172.21.15.116:51912) 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7e00640 1 -- 172.21.15.116:0/682171051 learned_addr learned my addr 172.21.15.116:0/682171051 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fbca00640 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d40824840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7400640 1 -- 172.21.15.116:0/682171051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 msgr2=0x555d40824840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7400640 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d40824840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7400640 1 -- 172.21.15.116:0/682171051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081c850 msgr2=0x555d4081cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7400640 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081c850 0x555d4081cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7400640 1 -- 172.21.15.116:0/682171051 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d4073f3c0 con 0x555d4081d240 2024-03-04T07:50:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.970+0000 7f1fb7400640 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d40821d40 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1fb819aff0 tx=0x7f1fa4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52e9ad718b4e92f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.971+0000 7f1fad200640 1 -- 172.21.15.116:0/682171051 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4026430 con 0x555d4081d240 2024-03-04T07:50:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.971+0000 7f1fad200640 1 -- 172.21.15.116:0/682171051 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fa401c8f0 con 0x555d4081d240 2024-03-04T07:50:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.971+0000 7f1fad200640 1 -- 172.21.15.116:0/682171051 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa401da20 con 0x555d4081d240 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.971+0000 7f1fc17b9740 1 -- 172.21.15.116:0/682171051 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 msgr2=0x555d40821d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.971+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d40821d40 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1fb819aff0 tx=0x7f1fa4014000 comp rx=0 tx=0).stop 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 -- 172.21.15.116:0/682171051 shutdown_connections 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d40824840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d40821d40 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/682171051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081c850 0x555d4081cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 -- 172.21.15.116:0/682171051 >> 172.21.15.116:0/682171051 conn(0x555d40817dc0 msgr2=0x555d408181c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 -- 172.21.15.116:0/682171051 shutdown_connections 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 -- 172.21.15.116:0/682171051 wait complete. 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.972+0000 7f1fc17b9740 1 -- start start 2024-03-04T07:50:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 1 Processor -- start 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d40826be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081d240 0x555d40827120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d4082e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d40825360 con 0x555d4081d240 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4073e9b0 con 0x555d4081c850 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d408254e0 con 0x555d408223b0 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fc17b9740 10 client.0 _ll_register_callbacks cb 0x555d4077a340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.973+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d40826be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d40826be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38160/0 (socket says 172.21.15.116:38160) 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 -- 172.21.15.116:0/2586654661 learned_addr learned my addr 172.21.15.116:0/2586654661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7400640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081d240 0x555d40827120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fbca00640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d4082e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 -- 172.21.15.116:0/2586654661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 msgr2=0x555d4082e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d4082e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 -- 172.21.15.116:0/2586654661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081d240 msgr2=0x555d40827120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081d240 0x555d40827120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 -- 172.21.15.116:0/2586654661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d40497210 con 0x555d4081c850 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fbca00640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d4082e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d40826be0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80bc790 tx=0x7f1fa802cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e855184db40c6f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fa1600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fa1600640 1 -- 172.21.15.116:0/2586654661 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80d4020 con 0x555d4081c850 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fa1600640 1 -- 172.21.15.116:0/2586654661 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1fa80bcef0 con 0x555d4081c850 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.974+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1fa401c8f0 con 0x555d4081c850 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fa1600640 1 -- 172.21.15.116:0/2586654661 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fa80c5230 con 0x555d4081c850 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 10 client.27914 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555d40563620 con 0x555d4081c850 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fa1600640 1 -- 172.21.15.116:0/2586654661 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fa401c8f0 con 0x555d4081c850 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fa1600640 1 -- 172.21.15.116:0/2586654661 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1fa80b87e0 con 0x555d4081c850 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 10 client.27914 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 10 client.27914 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 20 client.27914 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d40832950 0x555d40834df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 4 client.27914 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d4078f8b0 con 0x555d40832950 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.975+0000 7f1fb7400640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d40832950 0x555d40834df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.976+0000 7f1fb7400640 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d40832950 0x555d40834df0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14d45d0a30ff9b8e server_cookie=ee901b36b3988818 in_seq=0 out_seq=0 2024-03-04T07:50:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.976+0000 7f1fa1600640 10 client.27914 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.976+0000 7f1fa1600640 1 -- 172.21.15.116:0/2586654661 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555d4078f8b0 con 0x555d40832950 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.976+0000 7f1fa1600640 10 client.27914 handle_command_reply: tid=0 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 client.27914 shutdown 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 msgr2=0x555d40826be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d40826be0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80bc790 tx=0x7f1fa802cea0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 shutdown_connections 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d40832950 0x555d40834df0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d408223b0 0x555d4082e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d4081d240 0x555d40827120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fa1600640 0 client.27914 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/2586654661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d40826be0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 >> 172.21.15.116:0/2586654661 conn(0x555d40817dc0 msgr2=0x555d40816fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 shutdown_connections 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 1 -- 172.21.15.116:0/2586654661 wait complete. 2024-03-04T07:50:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.977+0000 7f1fc17b9740 20 client.27914 trim_cache size 0 max 16384 2024-03-04T07:50:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 Processor -- start 2024-03-04T07:50:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 -- start start 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d4081b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081d240 0x555d4082e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4081b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81a8ca0 con 0x555d408223b0 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81ab280 con 0x555d4081c850 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.992+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81a92a0 con 0x555d4081d240 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d4081b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d4081b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38170/0 (socket says 172.21.15.116:38170) 2024-03-04T07:50:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 -- 172.21.15.116:0/425437170 learned_addr learned my addr 172.21.15.116:0/425437170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7400640 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081d240 0x555d4082e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fbca00640 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4081b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 -- 172.21.15.116:0/425437170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081d240 msgr2=0x555d4082e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081d240 0x555d4082e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 -- 172.21.15.116:0/425437170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 msgr2=0x555d4081b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4081b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 -- 172.21.15.116:0/425437170 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d40818b30 con 0x555d4081c850 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d4081b020 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80b9410 tx=0x7f1fa80bafa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c440f40cf7ac8c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fa3e00640 1 -- 172.21.15.116:0/425437170 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80d4070 con 0x555d4081c850 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fa3e00640 1 -- 172.21.15.116:0/425437170 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fa80bbee0 con 0x555d4081c850 2024-03-04T07:50:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.993+0000 7f1fa3e00640 1 -- 172.21.15.116:0/425437170 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80d5220 con 0x555d4081c850 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 -- 172.21.15.116:0/425437170 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 msgr2=0x555d4081b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d4081b020 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80b9410 tx=0x7f1fa80bafa0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 -- 172.21.15.116:0/425437170 shutdown_connections 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4081b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081d240 0x555d4082e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/425437170 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081c850 0x555d4081b020 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 -- 172.21.15.116:0/425437170 >> 172.21.15.116:0/425437170 conn(0x555d40817dc0 msgr2=0x555d408181a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 -- 172.21.15.116:0/425437170 shutdown_connections 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.994+0000 7f1fc17b9740 1 -- 172.21.15.116:0/425437170 wait complete. 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 -- start start 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 Processor -- start 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081c850 0x555d4082bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d4082c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4082d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81a9900 con 0x555d408223b0 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81aac20 con 0x555d4081d240 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb81a9f60 con 0x555d4081c850 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.995+0000 7f1fc17b9740 10 client.0 _ll_register_callbacks cb 0x7f1fa80ed7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d4082c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d4082c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:38174/0 (socket says 172.21.15.116:38174) 2024-03-04T07:50:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fbca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4082d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 -- 172.21.15.116:0/4064997120 learned_addr learned my addr 172.21.15.116:0/4064997120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081c850 0x555d4082bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 -- 172.21.15.116:0/4064997120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081c850 msgr2=0x555d4082bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081c850 0x555d4082bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 -- 172.21.15.116:0/4064997120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 msgr2=0x555d4082d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4082d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 -- 172.21.15.116:0/4064997120 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d4082d720 con 0x555d4081d240 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fb7400640 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d4082c120 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40094a0 tx=0x7f1fa40091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8240a039b8ae0e45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fa1600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.996+0000 7f1fa1600640 1 -- 172.21.15.116:0/4064997120 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa402b040 con 0x555d4081d240 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fa1600640 1 -- 172.21.15.116:0/4064997120 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1fa40050d0 con 0x555d4081d240 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1fa80bbee0 con 0x555d4081d240 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fa1600640 1 -- 172.21.15.116:0/4064997120 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fa40052d0 con 0x555d4081d240 2024-03-04T07:50:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 10 client.27920 fetch_fsmap learned FSMap version 19 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555d4088e6d0 con 0x555d4081d240 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fa1600640 1 -- 172.21.15.116:0/4064997120 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fa80bbee0 con 0x555d4081d240 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fa1600640 1 -- 172.21.15.116:0/4064997120 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1fa403d050 con 0x555d4081d240 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 10 client.27920 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 10 client.27920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 20 client.27920 populate_metadata read hostname 'smithi116' 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d4088ec90 0x555d408910b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 4 client.27920 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.997+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555d40789c00 con 0x555d4088ec90 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.998+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d4088ec90 0x555d408910b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.998+0000 7f1fb7e00640 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d4088ec90 0x555d408910b0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=656d3b3e36e58a34 server_cookie=b699ce71fd1ccb72 in_seq=0 out_seq=0 2024-03-04T07:50:52.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:51.998+0000 7f1fa1600640 10 client.27920 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.013+0000 7f1fa1600640 1 -- 172.21.15.116:0/4064997120 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555d40789c00 con 0x555d4088ec90 2024-03-04T07:50:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.013+0000 7f1fa1600640 10 client.27920 handle_command_reply: tid=0 2024-03-04T07:50:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.013+0000 7f1fc17b9740 1 client.27920 shutdown 2024-03-04T07:50:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 msgr2=0x555d4082c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d4082c120 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40094a0 tx=0x7f1fa40091e0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 shutdown_connections 2024-03-04T07:50:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555d4088ec90 0x555d408910b0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d408223b0 0x555d4082d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555d4081d240 0x555d4082c120 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 --2- 172.21.15.116:0/4064997120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d4081c850 0x555d4082bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fa1600640 0 client.27920 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 >> 172.21.15.116:0/4064997120 conn(0x555d40817dc0 msgr2=0x555d4081b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 shutdown_connections 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 1 -- 172.21.15.116:0/4064997120 wait complete. 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.014+0000 7f1fc17b9740 20 client.27920 trim_cache size 0 max 16384 2024-03-04T07:50:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:50:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:50:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:50:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:50:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1f9c0a1ea0 msgr2=0x7f1f9c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1f9c0a1ea0 0x7f1f9c0a4360 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40095f0 tx=0x7f1fa4005f90 comp rx=0 tx=0).stop 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb80ffe60 msgr2=0x7f1fb8199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8199ae0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009230 tx=0x7f1fa8005d50 comp rx=0 tx=0).stop 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 shutdown_connections 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1f9c0a1ea0 0x7f1f9c0a4360 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fb8101160 0x7f1fb81a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81007c0 0x7f1fb819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 --2- 172.21.15.116:0/1424425381 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1fb80ffe60 0x7f1fb8199ae0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 >> 172.21.15.116:0/1424425381 conn(0x7f1fb80fba60 msgr2=0x7f1fb80fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 shutdown_connections 2024-03-04T07:50:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:50:52.015+0000 7f1fbea00640 1 -- 172.21.15.116:0/1424425381 wait complete. 2024-03-04T07:50:52.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:52.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:51 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:52.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:51 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:52.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:52.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:51 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:52 smithi116 ceph-mon[32150]: pgmap v704: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:50:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:52 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:50:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:52 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:50:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:52 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:52 smithi152 ceph-mon[37154]: pgmap v704: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:50:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:52 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:50:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:52 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:50:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:52 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:52 smithi133 ceph-mon[36505]: pgmap v704: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:50:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:52 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:50:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:52 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:50:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:52 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:50:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:54 smithi133 ceph-mon[36505]: pgmap v705: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:50:54.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:54 smithi116 ceph-mon[32150]: pgmap v705: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:50:54.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:54 smithi152 ceph-mon[37154]: pgmap v705: 97 pgs: 97 active+clean; 3.6 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:50:57.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:56 smithi116 ceph-mon[32150]: pgmap v706: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-03-04T07:50:57.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:56 smithi152 ceph-mon[37154]: pgmap v706: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-03-04T07:50:57.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:56 smithi133 ceph-mon[36505]: pgmap v706: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-03-04T07:50:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:50:58 smithi116 ceph-mon[32150]: pgmap v707: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:50:58 smithi152 ceph-mon[37154]: pgmap v707: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:50:59.286 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:50:58 smithi133 ceph-mon[36505]: pgmap v707: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:00 smithi116 ceph-mon[32150]: pgmap v708: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:00 smithi152 ceph-mon[37154]: pgmap v708: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:00 smithi133 ceph-mon[36505]: pgmap v708: 97 pgs: 97 active+clean; 3.7 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:02 smithi116 ceph-mon[32150]: pgmap v709: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:51:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:02 smithi152 ceph-mon[37154]: pgmap v709: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:51:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:02 smithi133 ceph-mon[36505]: pgmap v709: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:51:05.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:51:05.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:51:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 -- 172.21.15.116:0/1177848995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102630 msgr2=0x7ff970102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 --2- 172.21.15.116:0/1177848995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102630 0x7ff970102ab0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009900 tx=0x7ff95c02bc50 comp rx=0 tx=0).stop 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 -- 172.21.15.116:0/1177848995 shutdown_connections 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 --2- 172.21.15.116:0/1177848995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970102ff0 0x7ff970109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 --2- 172.21.15.116:0/1177848995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102630 0x7ff970102ab0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 --2- 172.21.15.116:0/1177848995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff970101440 0x7ff970101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 -- 172.21.15.116:0/1177848995 >> 172.21.15.116:0/1177848995 conn(0x7ff9700fc9b0 msgr2=0x7ff9700fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 -- 172.21.15.116:0/1177848995 shutdown_connections 2024-03-04T07:51:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.151+0000 7ff975800640 1 -- 172.21.15.116:0/1177848995 wait complete. 2024-03-04T07:51:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.152+0000 7ff975800640 1 Processor -- start 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff975800640 1 -- start start 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff975800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970101440 0x7ff970199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff975800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff970102630 0x7ff970199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff975800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102ff0 0x7ff9701a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff975800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010d120 con 0x7ff970101440 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff975800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010cfa0 con 0x7ff970102ff0 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff975800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010d2a0 con 0x7ff970102630 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff96fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970101440 0x7ff970199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff96f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff970102630 0x7ff970199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff96fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970101440 0x7ff970199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35822/0 (socket says 172.21.15.116:35822) 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff974e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102ff0 0x7ff9701a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.153+0000 7ff96fe00640 1 -- 172.21.15.116:0/1776350262 learned_addr learned my addr 172.21.15.116:0/1776350262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff974e00640 1 -- 172.21.15.116:0/1776350262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff970102630 msgr2=0x7ff970199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff974e00640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff970102630 0x7ff970199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff974e00640 1 -- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970101440 msgr2=0x7ff970199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff974e00640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970101440 0x7ff970199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff974e00640 1 -- 172.21.15.116:0/1776350262 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9701a1540 con 0x7ff970102ff0 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff96f400640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff970102630 0x7ff970199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff96fe00640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970101440 0x7ff970199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff974e00640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102ff0 0x7ff9701a0ed0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff968009fd0 tx=0x7ff96800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff96cc00640 1 -- 172.21.15.116:0/1776350262 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96800ce20 con 0x7ff970102ff0 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff96cc00640 1 -- 172.21.15.116:0/1776350262 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff968005020 con 0x7ff970102ff0 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff96cc00640 1 -- 172.21.15.116:0/1776350262 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968010720 con 0x7ff970102ff0 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9701a1830 con 0x7ff970102ff0 2024-03-04T07:51:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.154+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9701a1d50 con 0x7ff970102ff0 2024-03-04T07:51:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.157+0000 7ff979ba7740 1 Processor -- start 2024-03-04T07:51:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff96cc00640 1 -- 172.21.15.116:0/1776350262 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ff968005540 con 0x7ff970102ff0 2024-03-04T07:51:05.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff979ba7740 1 -- start start 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb54a10 0x56505bb54e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb59f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb5a570 0x56505bb5ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff979ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56505baba370 con 0x56505bb55400 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff979ba7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56505ba63550 con 0x56505bb54a10 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff979ba7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56505bab6410 con 0x56505bb5a570 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.158+0000 7ff96f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb54a10 0x56505bb54e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb54a10 0x56505bb54e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60262/0 (socket says 172.21.15.116:60262) 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb59f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 -- 172.21.15.116:0/621395382 learned_addr learned my addr 172.21.15.116:0/621395382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb5a570 0x56505bb5ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 -- 172.21.15.116:0/621395382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb5a570 msgr2=0x56505bb5ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb5a570 0x56505bb5ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 -- 172.21.15.116:0/621395382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 msgr2=0x56505bb59f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb59f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 -- 172.21.15.116:0/621395382 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56505ba773c0 con 0x56505bb54a10 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96fe00640 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb59f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96cc00640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9540a1df0 0x7ff9540a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96cc00640 1 -- 172.21.15.116:0/1776350262 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff9680ce450 con 0x7ff970102ff0 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.159+0000 7ff96f400640 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb54a10 0x56505bb54e30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff95c006010 tx=0x7ff95c005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec7b60547df8daa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.160+0000 7ff96fe00640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9540a1df0 0x7ff9540a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.160+0000 7ff965600640 1 -- 172.21.15.116:0/621395382 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c004490 con 0x56505bb54a10 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.160+0000 7ff965600640 1 -- 172.21.15.116:0/621395382 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff95c049070 con 0x56505bb54a10 2024-03-04T07:51:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.160+0000 7ff965600640 1 -- 172.21.15.116:0/621395382 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c005630 con 0x56505bb54a10 2024-03-04T07:51:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.160+0000 7ff96fe00640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9540a1df0 0x7ff9540a42b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x56505bb55800 tx=0x7ff96000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.160+0000 7ff979ba7740 1 -- 172.21.15.116:0/621395382 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb54a10 msgr2=0x56505bb54e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.160+0000 7ff979ba7740 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb54a10 0x56505bb54e30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff95c006010 tx=0x7ff95c005b50 comp rx=0 tx=0).stop 2024-03-04T07:51:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 -- 172.21.15.116:0/621395382 shutdown_connections 2024-03-04T07:51:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb5a570 0x56505bb5ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb59f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 --2- 172.21.15.116:0/621395382 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb54a10 0x56505bb54e30 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 -- 172.21.15.116:0/621395382 >> 172.21.15.116:0/621395382 conn(0x56505bb4ff80 msgr2=0x56505bb50380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 -- 172.21.15.116:0/621395382 shutdown_connections 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 -- 172.21.15.116:0/621395382 wait complete. 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 -- start start 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.161+0000 7ff979ba7740 1 Processor -- start 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb5ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb5f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb662f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff979ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56505bb5d4c0 con 0x56505bb55400 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff979ba7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56505ba769b0 con 0x56505bb5a570 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff979ba7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56505bb5d640 con 0x56505bb54a10 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff979ba7740 10 client.0 _ll_register_callbacks cb 0x56505bab2340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff974e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff974e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb662f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60278/0 (socket says 172.21.15.116:60278) 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff96f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb5f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff974e00640 1 -- 172.21.15.116:0/2728512267 learned_addr learned my addr 172.21.15.116:0/2728512267 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff96fe00640 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb5ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff974e00640 1 -- 172.21.15.116:0/2728512267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 msgr2=0x56505bb5ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff974e00640 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb5ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff974e00640 1 -- 172.21.15.116:0/2728512267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 msgr2=0x56505bb5f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.162+0000 7ff974e00640 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb5f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff974e00640 1 -- 172.21.15.116:0/2728512267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56505b7cf210 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff96f400640 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb5f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff974e00640 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb662f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff96809ae40 tx=0x7ff96809b200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3064e81d9ff2e5f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff95a000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff95a000640 1 -- 172.21.15.116:0/2728512267 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9680a3450 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff95a000640 1 -- 172.21.15.116:0/2728512267 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff9680a35f0 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff95c049070 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff95a000640 1 -- 172.21.15.116:0/2728512267 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff96809a590 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff979ba7740 10 client.27932 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.163+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56505b89b620 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff95a000640 1 -- 172.21.15.116:0/2728512267 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff95c049070 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff95a000640 1 -- 172.21.15.116:0/2728512267 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff9680a01b0 con 0x56505bb5a570 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff979ba7740 10 client.27932 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff979ba7740 10 client.27932 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:05.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff979ba7740 20 client.27932 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff979ba7740 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bb6aa40 0x56505bb6ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff979ba7740 4 client.27932 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56505bac78b0 con 0x56505bb6aa40 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.164+0000 7ff96fe00640 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bb6aa40 0x56505bb6ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.165+0000 7ff95a000640 10 client.27932 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.165+0000 7ff96fe00640 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bb6aa40 0x56505bb6ce30 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b10646a061f74eab server_cookie=7f6aab556a841228 in_seq=0 out_seq=0 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.165+0000 7ff95a000640 1 -- 172.21.15.116:0/2728512267 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56505bac78b0 con 0x56505bb6aa40 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.165+0000 7ff95a000640 10 client.27932 handle_command_reply: tid=0 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.165+0000 7ff979ba7740 1 client.27932 shutdown 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.165+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 msgr2=0x56505bb662f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.165+0000 7ff979ba7740 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb662f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff96809ae40 tx=0x7ff96809b200 comp rx=0 tx=0).stop 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 shutdown_connections 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bb6aa40 0x56505bb6ce30 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb662f0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb5f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff95a000640 0 client.27932 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 --2- 172.21.15.116:0/2728512267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb5ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 >> 172.21.15.116:0/2728512267 conn(0x56505bb4ff80 msgr2=0x56505bb4f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 shutdown_connections 2024-03-04T07:51:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 1 -- 172.21.15.116:0/2728512267 wait complete. 2024-03-04T07:51:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.166+0000 7ff979ba7740 20 client.27932 trim_cache size 0 max 16384 2024-03-04T07:51:05.180 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:04 smithi116 ceph-mon[32150]: pgmap v710: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 Processor -- start 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 -- start start 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb567c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb662b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb56d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701a8990 con 0x56505bb55400 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701aaf70 con 0x56505bb5a570 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.181+0000 7ff979ba7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701a8f90 con 0x56505bb54a10 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb56d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff96f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb662b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb56d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60286/0 (socket says 172.21.15.116:60286) 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 -- 172.21.15.116:0/3820175569 learned_addr learned my addr 172.21.15.116:0/3820175569 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff96fe00640 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 -- 172.21.15.116:0/3820175569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 msgr2=0x56505bb567c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb567c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 -- 172.21.15.116:0/3820175569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 msgr2=0x56505bb662b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb662b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 -- 172.21.15.116:0/3820175569 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56505bb55c90 con 0x56505bb5a570 2024-03-04T07:51:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff974e00640 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb56d00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff9680975a0 tx=0x7ff968099650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcefb0a684c46f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff964c00640 1 -- 172.21.15.116:0/3820175569 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96809aaa0 con 0x56505bb5a570 2024-03-04T07:51:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff964c00640 1 -- 172.21.15.116:0/3820175569 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9680051d0 con 0x56505bb5a570 2024-03-04T07:51:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.182+0000 7ff964c00640 1 -- 172.21.15.116:0/3820175569 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96809af10 con 0x56505bb5a570 2024-03-04T07:51:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 -- 172.21.15.116:0/3820175569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 msgr2=0x56505bb56d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb56d00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff9680975a0 tx=0x7ff968099650 comp rx=0 tx=0).stop 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 -- 172.21.15.116:0/3820175569 shutdown_connections 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb56d00 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb662b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 --2- 172.21.15.116:0/3820175569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 -- 172.21.15.116:0/3820175569 >> 172.21.15.116:0/3820175569 conn(0x56505bb4ff80 msgr2=0x56505bb5ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 -- 172.21.15.116:0/3820175569 shutdown_connections 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.183+0000 7ff979ba7740 1 -- 172.21.15.116:0/3820175569 wait complete. 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 -- start start 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 Processor -- start 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb63c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb64170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb65180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701a95f0 con 0x56505bb55400 2024-03-04T07:51:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701aa910 con 0x56505bb5a570 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9701a9c50 con 0x56505bb54a10 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff979ba7740 10 client.0 _ll_register_callbacks cb 0x7ff9680973e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.184+0000 7ff974e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb65180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff96f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb64170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb65180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60300/0 (socket says 172.21.15.116:60300) 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 -- 172.21.15.116:0/787272090 learned_addr learned my addr 172.21.15.116:0/787272090 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff96fe00640 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb63c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 -- 172.21.15.116:0/787272090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 msgr2=0x56505bb63c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb63c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 -- 172.21.15.116:0/787272090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 msgr2=0x56505bb64170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb64170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 -- 172.21.15.116:0/787272090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56505bb65770 con 0x56505bb5a570 2024-03-04T07:51:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff974e00640 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb65180 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff9680a33b0 tx=0x7ff9680054b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cd1385fe453caf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff95a000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff95a000640 1 -- 172.21.15.116:0/787272090 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9680a7070 con 0x56505bb5a570 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff95a000640 1 -- 172.21.15.116:0/787272090 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff9680a3450 con 0x56505bb5a570 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.185+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9680051d0 con 0x56505bb5a570 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff95a000640 1 -- 172.21.15.116:0/787272090 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff96809aaa0 con 0x56505bb5a570 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 10 client.27938 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56505bbc64e0 con 0x56505bb5a570 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff95a000640 1 -- 172.21.15.116:0/787272090 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff968085e50 con 0x56505bb5a570 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff95a000640 1 -- 172.21.15.116:0/787272090 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff9680aa020 con 0x56505bb5a570 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 10 client.27938 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 10 client.27938 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 20 client.27938 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bbc6da0 0x56505bbc91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 4 client.27938 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:51:05.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56505bac1c00 con 0x56505bbc6da0 2024-03-04T07:51:05.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.186+0000 7ff96fe00640 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bbc6da0 0x56505bbc91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.187+0000 7ff96fe00640 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bbc6da0 0x56505bbc91c0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89095a35e5e00db8 server_cookie=c8b934a4cb7cc36e in_seq=0 out_seq=0 2024-03-04T07:51:05.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.187+0000 7ff95a000640 10 client.27938 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff95a000640 1 -- 172.21.15.116:0/787272090 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56505bac1c00 con 0x56505bbc6da0 2024-03-04T07:51:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff95a000640 10 client.27938 handle_command_reply: tid=0 2024-03-04T07:51:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 client.27938 shutdown 2024-03-04T07:51:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 msgr2=0x56505bb65180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb65180 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff9680a33b0 tx=0x7ff9680054b0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 shutdown_connections 2024-03-04T07:51:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56505bbc6da0 0x56505bbc91c0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56505bb5a570 0x56505bb65180 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff95a000640 0 client.27938 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56505bb55400 0x56505bb64170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 --2- 172.21.15.116:0/787272090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56505bb54a10 0x56505bb63c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 >> 172.21.15.116:0/787272090 conn(0x56505bb4ff80 msgr2=0x56505bb5ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 shutdown_connections 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 1 -- 172.21.15.116:0/787272090 wait complete. 2024-03-04T07:51:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.201+0000 7ff979ba7740 20 client.27938 trim_cache size 0 max 16384 2024-03-04T07:51:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:51:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:51:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:51:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9540a1df0 msgr2=0x7ff9540a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9540a1df0 0x7ff9540a42b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x56505bb55800 tx=0x7ff96000b410 comp rx=0 tx=0).stop 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102ff0 msgr2=0x7ff9701a0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102ff0 0x7ff9701a0ed0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff968009fd0 tx=0x7ff96800ef10 comp rx=0 tx=0).stop 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 shutdown_connections 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff970102ff0 0x7ff9701a0ed0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ff9540a1df0 0x7ff9540a42b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff970102630 0x7ff970199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 --2- 172.21.15.116:0/1776350262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff970101440 0x7ff970199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 >> 172.21.15.116:0/1776350262 conn(0x7ff9700fc9b0 msgr2=0x7ff970073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 shutdown_connections 2024-03-04T07:51:05.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:05.202+0000 7ff975800640 1 -- 172.21.15.116:0/1776350262 wait complete. 2024-03-04T07:51:05.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:04 smithi152 ceph-mon[37154]: pgmap v710: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 -- 172.21.15.116:0/4149656003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a741097a0 msgr2=0x7f1a74110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4149656003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a741097a0 0x7f1a74110030 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009be0 tx=0x7f1a6c02bef0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 -- 172.21.15.116:0/4149656003 shutdown_connections 2024-03-04T07:51:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4149656003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a741097a0 0x7f1a74110030 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4149656003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a74108de0 0x7f1a74109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4149656003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a74107bf0 0x7f1a74108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 -- 172.21.15.116:0/4149656003 >> 172.21.15.116:0/4149656003 conn(0x7f1a74076b90 msgr2=0x7f1a74078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 -- 172.21.15.116:0/4149656003 shutdown_connections 2024-03-04T07:51:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.305+0000 7f1a78c00640 1 -- 172.21.15.116:0/4149656003 wait complete. 2024-03-04T07:51:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.306+0000 7f1a78c00640 1 Processor -- start 2024-03-04T07:51:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a78c00640 1 -- start start 2024-03-04T07:51:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a78c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a74107bf0 0x7f1a7419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a78c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a74108de0 0x7f1a7419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a78c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a741097a0 0x7f1a741a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a78c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a74113870 con 0x7f1a74108de0 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a78c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741136f0 con 0x7f1a74107bf0 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a78c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741139f0 con 0x7f1a741097a0 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a71e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a74107bf0 0x7f1a7419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a71400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a74108de0 0x7f1a7419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a71e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a74107bf0 0x7f1a7419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60318/0 (socket says 172.21.15.116:60318) 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a71e00640 1 -- 172.21.15.116:0/4034161429 learned_addr learned my addr 172.21.15.116:0/4034161429 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.307+0000 7f1a72800640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a741097a0 0x7f1a741a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a71e00640 1 -- 172.21.15.116:0/4034161429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a741097a0 msgr2=0x7f1a741a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a71e00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a741097a0 0x7f1a741a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a71e00640 1 -- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a74108de0 msgr2=0x7f1a7419e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a71e00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a74108de0 0x7f1a7419e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a71e00640 1 -- 172.21.15.116:0/4034161429 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a741a5aa0 con 0x7f1a74107bf0 2024-03-04T07:51:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a71e00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a74107bf0 0x7f1a7419de80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400e9a0 tx=0x7f1a6400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a6aa00640 1 -- 172.21.15.116:0/4034161429 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6400cd80 con 0x7f1a74107bf0 2024-03-04T07:51:05.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a6aa00640 1 -- 172.21.15.116:0/4034161429 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a64010910 con 0x7f1a74107bf0 2024-03-04T07:51:05.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a6aa00640 1 -- 172.21.15.116:0/4034161429 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a64018a30 con 0x7f1a74107bf0 2024-03-04T07:51:05.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a741a5d90 con 0x7f1a74107bf0 2024-03-04T07:51:05.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.308+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a741a6250 con 0x7f1a74107bf0 2024-03-04T07:51:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.311+0000 7f1a7b980740 1 Processor -- start 2024-03-04T07:51:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.311+0000 7f1a6aa00640 1 -- 172.21.15.116:0/4034161429 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f1a64010ac0 con 0x7f1a74107bf0 2024-03-04T07:51:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.311+0000 7f1a7b980740 1 -- start start 2024-03-04T07:51:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869178f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f86917e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f869180b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a7b980740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8690a6090 con 0x55f86917e6b0 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a7b980740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8690ddd30 con 0x55f869178b50 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a7b980740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f869086a30 con 0x55f869179540 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869178f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869178f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60324/0 (socket says 172.21.15.116:60324) 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a71400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f869180b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 -- 172.21.15.116:0/902786866 learned_addr learned my addr 172.21.15.116:0/902786866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a71e00640 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f86917e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 -- 172.21.15.116:0/902786866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 msgr2=0x55f86917e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f86917e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 -- 172.21.15.116:0/902786866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 msgr2=0x55f869180b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f869180b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.312+0000 7f1a72800640 1 -- 172.21.15.116:0/902786866 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f868ebc7d0 con 0x55f869178b50 2024-03-04T07:51:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.313+0000 7f1a72800640 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869178f70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009be0 tx=0x7f1a6c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6be8f83b6887ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.313+0000 7f1a5f000640 1 -- 172.21.15.116:0/902786866 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c043030 con 0x55f869178b50 2024-03-04T07:51:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.313+0000 7f1a5f000640 1 -- 172.21.15.116:0/902786866 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a6c033b90 con 0x55f869178b50 2024-03-04T07:51:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.313+0000 7f1a5f000640 1 -- 172.21.15.116:0/902786866 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c02c890 con 0x55f869178b50 2024-03-04T07:51:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.313+0000 7f1a6aa00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1a580a1e00 0x7f1a580a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.313+0000 7f1a6aa00640 1 -- 172.21.15.116:0/4034161429 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1a640ce050 con 0x7f1a74107bf0 2024-03-04T07:51:05.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.313+0000 7f1a71e00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1a580a1e00 0x7f1a580a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 -- 172.21.15.116:0/902786866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 msgr2=0x55f869178f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869178f70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009be0 tx=0x7f1a6c009310 comp rx=0 tx=0).stop 2024-03-04T07:51:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a71e00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1a580a1e00 0x7f1a580a42c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1a64097750 tx=0x7f1a64000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 -- 172.21.15.116:0/902786866 shutdown_connections 2024-03-04T07:51:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f869180b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f86917e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 --2- 172.21.15.116:0/902786866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869178f70 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 -- 172.21.15.116:0/902786866 >> 172.21.15.116:0/902786866 conn(0x55f8691740c0 msgr2=0x55f8691744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 -- 172.21.15.116:0/902786866 shutdown_connections 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 -- 172.21.15.116:0/902786866 wait complete. 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.314+0000 7f1a7b980740 1 -- start start 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 1 Processor -- start 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869182f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f869183450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f86918a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f86907c0c0 con 0x55f86917e6b0 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8690d9dd0 con 0x55f869178b50 2024-03-04T07:51:05.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f869181600 con 0x55f869179540 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a7b980740 10 client.0 _ll_register_callbacks cb 0x55f869099860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a72800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869182f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a71400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f869183450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a71400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f869183450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43508/0 (socket says 172.21.15.116:43508) 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a71400640 1 -- 172.21.15.116:0/3532258661 learned_addr learned my addr 172.21.15.116:0/3532258661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a72800640 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869182f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60328/0 (socket says 172.21.15.116:60328) 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.315+0000 7f1a72800640 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f86918a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a71400640 1 -- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 msgr2=0x55f869182f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a71400640 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869182f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a71400640 1 -- 172.21.15.116:0/3532258661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 msgr2=0x55f86918a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a71400640 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f86918a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a71400640 1 -- 172.21.15.116:0/3532258661 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f868df23f0 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a71400640 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f869183450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a60012720 tx=0x7f1a60012be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9580e0482a08535b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a53400640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a53400640 1 -- 172.21.15.116:0/3532258661 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000f8c0 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a6c033b90 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.316+0000 7f1a53400640 1 -- 172.21.15.116:0/3532258661 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1a6000fa60 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a53400640 1 -- 172.21.15.116:0/3532258661 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a6000fc60 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 10 client.27814 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f868db20f0 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a53400640 1 -- 172.21.15.116:0/3532258661 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a6c033b90 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a53400640 1 -- 172.21.15.116:0/3532258661 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1a6001c4b0 con 0x55f869179540 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 10 client.27814 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 10 client.27814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:05.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 20 client.27814 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f86918ea60 0x55f869190e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 4 client.27814 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.317+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8690eb270 con 0x55f86918ea60 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.318+0000 7f1a72800640 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f86918ea60 0x55f869190e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.318+0000 7f1a72800640 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f86918ea60 0x55f869190e80 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb750ee7f67acab8 server_cookie=a134f4ff32b00142 in_seq=0 out_seq=0 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.318+0000 7f1a53400640 10 client.27814 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a53400640 1 -- 172.21.15.116:0/3532258661 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f8690eb270 con 0x55f86918ea60 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a53400640 10 client.27814 handle_command_reply: tid=0 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 client.27814 shutdown 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 msgr2=0x55f869183450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f869183450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a60012720 tx=0x7f1a60012be0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 shutdown_connections 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f86918ea60 0x55f869190e80 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f86917e6b0 0x55f86918a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869179540 0x55f869183450 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 --2- 172.21.15.116:0/3532258661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869178b50 0x55f869182f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a53400640 0 client.27814 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 >> 172.21.15.116:0/3532258661 conn(0x55f8691740c0 msgr2=0x55f869173360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 shutdown_connections 2024-03-04T07:51:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 1 -- 172.21.15.116:0/3532258661 wait complete. 2024-03-04T07:51:05.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.319+0000 7f1a7b980740 20 client.27814 trim_cache size 0 max 16384 2024-03-04T07:51:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.336+0000 7f1a7b980740 1 Processor -- start 2024-03-04T07:51:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.336+0000 7f1a7b980740 1 -- start start 2024-03-04T07:51:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.336+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869178b50 0x55f869182990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869179540 0x55f86918a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f86917e6b0 0x55f8691731e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a7b980740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741acf70 con 0x55f869179540 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a7b980740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741af550 con 0x55f86917e6b0 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a7b980740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741ad570 con 0x55f869178b50 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869179540 0x55f86918a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869179540 0x55f86918a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35914/0 (socket says 172.21.15.116:35914) 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 -- 172.21.15.116:0/2908306364 learned_addr learned my addr 172.21.15.116:0/2908306364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a72800640 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f86917e6b0 0x55f8691731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a72800640 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869178b50 0x55f869182990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 -- 172.21.15.116:0/2908306364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869178b50 msgr2=0x55f869182990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869178b50 0x55f869182990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 -- 172.21.15.116:0/2908306364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f86917e6b0 msgr2=0x55f8691731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f86917e6b0 0x55f8691731e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 -- 172.21.15.116:0/2908306364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f869181780 con 0x55f869179540 2024-03-04T07:51:05.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.337+0000 7f1a71400640 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869179540 0x55f86918a3e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1a600059d0 tx=0x7f1a6000fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3b52558249478e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.339 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:04 smithi133 ceph-mon[36505]: pgmap v710: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a5e600640 1 -- 172.21.15.116:0/2908306364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6002b030 con 0x55f869179540 2024-03-04T07:51:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a5e600640 1 -- 172.21.15.116:0/2908306364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a6001cb10 con 0x55f869179540 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a5e600640 1 -- 172.21.15.116:0/2908306364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000f790 con 0x55f869179540 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 -- 172.21.15.116:0/2908306364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869179540 msgr2=0x55f86918a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869179540 0x55f86918a3e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1a600059d0 tx=0x7f1a6000fe00 comp rx=0 tx=0).stop 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 -- 172.21.15.116:0/2908306364 shutdown_connections 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f86917e6b0 0x55f8691731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869179540 0x55f86918a3e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 --2- 172.21.15.116:0/2908306364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f869178b50 0x55f869182990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 -- 172.21.15.116:0/2908306364 >> 172.21.15.116:0/2908306364 conn(0x55f8691740c0 msgr2=0x55f869172ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 -- 172.21.15.116:0/2908306364 shutdown_connections 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.338+0000 7f1a7b980740 1 -- 172.21.15.116:0/2908306364 wait complete. 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 -- start start 2024-03-04T07:51:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 Processor -- start 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869178b50 0x55f8691881d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869179540 0x55f869188710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f86917e6b0 0x55f869189720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741adbd0 con 0x55f869178b50 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741aeef0 con 0x55f869179540 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741ae230 con 0x55f86917e6b0 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.339+0000 7f1a7b980740 10 client.0 _ll_register_callbacks cb 0x7f1a60009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869179540 0x55f869188710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869179540 0x55f869188710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60346/0 (socket says 172.21.15.116:60346) 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 -- 172.21.15.116:0/1408818783 learned_addr learned my addr 172.21.15.116:0/1408818783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a72800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869178b50 0x55f8691881d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 -- 172.21.15.116:0/1408818783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f86917e6b0 msgr2=0x55f869189720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f86917e6b0 0x55f869189720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 -- 172.21.15.116:0/1408818783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869178b50 msgr2=0x55f8691881d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869178b50 0x55f8691881d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 -- 172.21.15.116:0/1408818783 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f869189d10 con 0x55f869179540 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a71400640 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869179540 0x55f869188710 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1a6001cae0 tx=0x7f1a6001df00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10e233a9a1d2fe77 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a53400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a53400640 1 -- 172.21.15.116:0/1408818783 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6002b070 con 0x55f869179540 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a6001cb10 con 0x55f869179540 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.340+0000 7f1a53400640 1 -- 172.21.15.116:0/1408818783 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1a6000f9a0 con 0x55f869179540 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a53400640 1 -- 172.21.15.116:0/1408818783 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a60005050 con 0x55f869179540 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 10 client.27956 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f8691eaa00 con 0x55f869179540 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a53400640 1 -- 172.21.15.116:0/1408818783 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a6001cb10 con 0x55f869179540 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a53400640 1 -- 172.21.15.116:0/1408818783 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1a60003b20 con 0x55f869179540 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 10 client.27956 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 10 client.27956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 20 client.27956 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f8691eb2a0 0x55f8691ed6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 4 client.27956 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f8690e55c0 con 0x55f8691eb2a0 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.341+0000 7f1a72800640 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f8691eb2a0 0x55f8691ed6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a53400640 10 client.27956 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a72800640 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f8691eb2a0 0x55f8691ed6c0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=460082687ee27a1c server_cookie=7975beb15c9fb8a4 in_seq=0 out_seq=0 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a53400640 1 -- 172.21.15.116:0/1408818783 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f8690e55c0 con 0x55f8691eb2a0 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a53400640 10 client.27956 handle_command_reply: tid=0 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 client.27956 shutdown 2024-03-04T07:51:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869179540 msgr2=0x55f869188710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869179540 0x55f869188710 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1a6001cae0 tx=0x7f1a6001df00 comp rx=0 tx=0).stop 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 shutdown_connections 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f8691eb2a0 0x55f8691ed6c0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f86917e6b0 0x55f869189720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f869179540 0x55f869188710 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 --2- 172.21.15.116:0/1408818783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f869178b50 0x55f8691881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a53400640 0 client.27956 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 >> 172.21.15.116:0/1408818783 conn(0x55f8691740c0 msgr2=0x55f869177bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 shutdown_connections 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 1 -- 172.21.15.116:0/1408818783 wait complete. 2024-03-04T07:51:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.342+0000 7f1a7b980740 20 client.27956 trim_cache size 0 max 16384 2024-03-04T07:51:05.349 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:51:05.349 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:51:05.349 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:51:05.349 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1a580a1e00 msgr2=0x7f1a580a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1a580a1e00 0x7f1a580a42c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1a64097750 tx=0x7f1a64000bc0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a74107bf0 msgr2=0x7f1a7419de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a74107bf0 0x7f1a7419de80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400e9a0 tx=0x7f1a6400ee60 comp rx=0 tx=0).stop 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 shutdown_connections 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f1a580a1e00 0x7f1a580a42c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1a741097a0 0x7f1a741a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a74108de0 0x7f1a7419e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 --2- 172.21.15.116:0/4034161429 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a74107bf0 0x7f1a7419de80 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 >> 172.21.15.116:0/4034161429 conn(0x7f1a74076b90 msgr2=0x7f1a7410c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 shutdown_connections 2024-03-04T07:51:05.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.343+0000 7f1a78c00640 1 -- 172.21.15.116:0/4034161429 wait complete. 2024-03-04T07:51:05.359 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:51:05.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:51:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 -- 172.21.15.116:0/2452254922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53001027d0 msgr2=0x7f5300104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 --2- 172.21.15.116:0/2452254922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53001027d0 0x7f5300104c40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009ca0 tx=0x7f52e802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 -- 172.21.15.116:0/2452254922 shutdown_connections 2024-03-04T07:51:05.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 --2- 172.21.15.116:0/2452254922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5300105180 0x7f5300105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 --2- 172.21.15.116:0/2452254922 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53001027d0 0x7f5300104c40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 --2- 172.21.15.116:0/2452254922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53000ffe60 0x7f5300102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 -- 172.21.15.116:0/2452254922 >> 172.21.15.116:0/2452254922 conn(0x7f53000fba80 msgr2=0x7f53000fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 -- 172.21.15.116:0/2452254922 shutdown_connections 2024-03-04T07:51:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.612+0000 7f5305a00640 1 -- 172.21.15.116:0/2452254922 wait complete. 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 Processor -- start 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 -- start start 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53000ffe60 0x7f530019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53001027d0 0x7f530019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300105180 0x7f53001a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5300113840 con 0x7f5300105180 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53001136c0 con 0x7f53000ffe60 2024-03-04T07:51:05.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.613+0000 7f5305a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53001139c0 con 0x7f53001027d0 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53000ffe60 0x7f530019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53000ffe60 0x7f530019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:60364/0 (socket says 172.21.15.116:60364) 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52ff400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300105180 0x7f53001a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 -- 172.21.15.116:0/4146695226 learned_addr learned my addr 172.21.15.116:0/4146695226 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fe000640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53001027d0 0x7f530019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 -- 172.21.15.116:0/4146695226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53001027d0 msgr2=0x7f530019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53001027d0 0x7f530019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 -- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300105180 msgr2=0x7f53001a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300105180 0x7f53001a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 -- 172.21.15.116:0/4146695226 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53001a5b00 con 0x7f53000ffe60 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.614+0000 7f52fea00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53000ffe60 0x7f530019dee0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f52f000e9f0 tx=0x7f52f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.615+0000 7f52f7400640 1 -- 172.21.15.116:0/4146695226 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f000cd80 con 0x7f53000ffe60 2024-03-04T07:51:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.615+0000 7f52f7400640 1 -- 172.21.15.116:0/4146695226 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52f0010910 con 0x7f53000ffe60 2024-03-04T07:51:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.615+0000 7f52f7400640 1 -- 172.21.15.116:0/4146695226 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0018990 con 0x7f53000ffe60 2024-03-04T07:51:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.615+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53001a5df0 con 0x7f53000ffe60 2024-03-04T07:51:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.615+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53001a6200 con 0x7f53000ffe60 2024-03-04T07:51:05.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.616+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53000fd570 con 0x7f53000ffe60 2024-03-04T07:51:05.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.618+0000 7f52f7400640 1 -- 172.21.15.116:0/4146695226 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f52f0010ac0 con 0x7f53000ffe60 2024-03-04T07:51:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.619+0000 7f52f7400640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52e00a1e00 0x7f52e00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.619+0000 7f52f7400640 1 -- 172.21.15.116:0/4146695226 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f52f00ce050 con 0x7f53000ffe60 2024-03-04T07:51:05.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.619+0000 7f52fe000640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52e00a1e00 0x7f52e00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:05.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.620+0000 7f52fe000640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52e00a1e00 0x7f52e00a42c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f52e802c700 tx=0x7f52e80323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:05.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.623+0000 7f52f7400640 1 -- 172.21.15.116:0/4146695226 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f52f009ae30 con 0x7f53000ffe60 2024-03-04T07:51:05.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.843+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f530002d2a0 con 0x7f53000ffe60 2024-03-04T07:51:05.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.844+0000 7f52f7400640 1 -- 172.21.15.116:0/4146695226 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f52f009eed0 con 0x7f53000ffe60 2024-03-04T07:51:05.845 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:51:05.845 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:51:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:51:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52e00a1e00 msgr2=0x7f52e00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52e00a1e00 0x7f52e00a42c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f52e802c700 tx=0x7f52e80323c0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53000ffe60 msgr2=0x7f530019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53000ffe60 0x7f530019dee0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f52f000e9f0 tx=0x7f52f000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 shutdown_connections 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5300105180 0x7f53001a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f52e00a1e00 0x7f52e00a42c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53001027d0 0x7f530019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 --2- 172.21.15.116:0/4146695226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f53000ffe60 0x7f530019dee0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 >> 172.21.15.116:0/4146695226 conn(0x7f53000fba80 msgr2=0x7f53001007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.847+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 shutdown_connections 2024-03-04T07:51:05.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:05.848+0000 7f5305a00640 1 -- 172.21.15.116:0/4146695226 wait complete. 2024-03-04T07:51:05.867 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:51:05.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 -- 172.21.15.116:0/2698288087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb1380771d0 msgr2=0x7fb138077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 --2- 172.21.15.116:0/2698288087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb1380771d0 0x7fb138077630 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb120009ca0 tx=0x7fb12002c100 comp rx=0 tx=0).stop 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 -- 172.21.15.116:0/2698288087 shutdown_connections 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 --2- 172.21.15.116:0/2698288087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb138077b70 0x7fb138078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 --2- 172.21.15.116:0/2698288087 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb1380771d0 0x7fb138077630 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 --2- 172.21.15.116:0/2698288087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb138075f90 0x7fb1380763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 -- 172.21.15.116:0/2698288087 >> 172.21.15.116:0/2698288087 conn(0x7fb1380fed80 msgr2=0x7fb1381011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:06.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 -- 172.21.15.116:0/2698288087 shutdown_connections 2024-03-04T07:51:06.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.085+0000 7fb13de00640 1 -- 172.21.15.116:0/2698288087 wait complete. 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 Processor -- start 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 -- start start 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb138075f90 0x7fb13819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1380771d0 0x7fb13819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb138077b70 0x7fb1381a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381138a0 con 0x7fb1380771d0 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138113720 con 0x7fb138075f90 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.086+0000 7fb13de00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138113a20 con 0x7fb138077b70 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb138075f90 0x7fb13819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb138075f90 0x7fb13819ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59360/0 (socket says 172.21.15.116:59360) 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 -- 172.21.15.116:0/3954471329 learned_addr learned my addr 172.21.15.116:0/3954471329 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:06.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136000640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1380771d0 0x7fb13819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb137400640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb138077b70 0x7fb1381a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 -- 172.21.15.116:0/3954471329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb138077b70 msgr2=0x7fb1381a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb138077b70 0x7fb1381a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 -- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1380771d0 msgr2=0x7fb13819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1380771d0 0x7fb13819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 -- 172.21.15.116:0/3954471329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1381a5af0 con 0x7fb138075f90 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb137400640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb138077b70 0x7fb1381a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.087+0000 7fb136a00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb138075f90 0x7fb13819ded0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb13000d950 tx=0x7fb13000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.088+0000 7fb13ca00640 1 -- 172.21.15.116:0/3954471329 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130017030 con 0x7fb138075f90 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.088+0000 7fb13ca00640 1 -- 172.21.15.116:0/3954471329 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb130004020 con 0x7fb138075f90 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.088+0000 7fb13ca00640 1 -- 172.21.15.116:0/3954471329 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1300047a0 con 0x7fb138075f90 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.088+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1381a5de0 con 0x7fb138075f90 2024-03-04T07:51:06.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.088+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1381a61f0 con 0x7fb138075f90 2024-03-04T07:51:06.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.092+0000 7fb140a84740 1 Processor -- start 2024-03-04T07:51:06.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.092+0000 7fb140a84740 1 -- start start 2024-03-04T07:51:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb13ca00640 1 -- 172.21.15.116:0/3954471329 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb1300041d0 con 0x7fb138075f90 2024-03-04T07:51:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb140a84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x5582024810c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb140a84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x558202486190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb140a84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x558202488c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb140a84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582023e5d60 con 0x558202481690 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb140a84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55820238ecf0 con 0x558202486800 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb140a84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582023e1e00 con 0x558202480ca0 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb136a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x558202486190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb137400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x5582024810c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb136000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x558202488c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb136000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x558202488c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59372/0 (socket says 172.21.15.116:59372) 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb137400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x5582024810c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:33094/0 (socket says 172.21.15.116:33094) 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb136000640 1 -- 172.21.15.116:0/3943567563 learned_addr learned my addr 172.21.15.116:0/3943567563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb136000640 1 -- 172.21.15.116:0/3943567563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 msgr2=0x5582024810c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.093+0000 7fb136000640 1 --2- 172.21.15.116:0/3943567563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x5582024810c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb136000640 1 -- 172.21.15.116:0/3943567563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 msgr2=0x558202486190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb136000640 1 --2- 172.21.15.116:0/3943567563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x558202486190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb136000640 1 -- 172.21.15.116:0/3943567563 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582020fa3f0 con 0x558202486800 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb136000640 1 --2- 172.21.15.116:0/3943567563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x558202488c90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb120005e30 tx=0x7fb120009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52f23badee90a5fa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb127e00640 1 -- 172.21.15.116:0/3943567563 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120043070 con 0x558202486800 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb13ca00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb11c0a1e00 0x7fb11c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb127e00640 1 -- 172.21.15.116:0/3943567563 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb120003970 con 0x558202486800 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb127e00640 1 -- 172.21.15.116:0/3943567563 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120039040 con 0x558202486800 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.094+0000 7fb13ca00640 1 -- 172.21.15.116:0/3954471329 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb1300cd7f0 con 0x7fb138075f90 2024-03-04T07:51:06.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb137400640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb11c0a1e00 0x7fb11c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 -- 172.21.15.116:0/3943567563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 msgr2=0x558202488c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 --2- 172.21.15.116:0/3943567563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x558202488c90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb120005e30 tx=0x7fb120009800 comp rx=0 tx=0).stop 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 -- 172.21.15.116:0/3943567563 shutdown_connections 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 --2- 172.21.15.116:0/3943567563 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x558202488c90 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 --2- 172.21.15.116:0/3943567563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x558202486190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 --2- 172.21.15.116:0/3943567563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x5582024810c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 -- 172.21.15.116:0/3943567563 >> 172.21.15.116:0/3943567563 conn(0x55820247c210 msgr2=0x55820247c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 -- 172.21.15.116:0/3943567563 shutdown_connections 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb137400640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb11c0a1e00 0x7fb11c0a42c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x558202481ef0 tx=0x7fb1280093d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.095+0000 7fb140a84740 1 -- 172.21.15.116:0/3943567563 wait complete. 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 -- start start 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 Processor -- start 2024-03-04T07:51:06.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202480ca0 0x55820248b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202481690 0x55820248b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x5582024926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558202489750 con 0x558202486800 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x558202384220 con 0x558202480ca0 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582024898d0 con 0x558202481690 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.096+0000 7fb140a84740 10 client.0 _ll_register_callbacks cb 0x5582020f9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202480ca0 0x55820248b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202480ca0 0x55820248b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59380/0 (socket says 172.21.15.116:59380) 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 -- 172.21.15.116:0/4233190919 learned_addr learned my addr 172.21.15.116:0/4233190919 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb137400640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x5582024926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136000640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202481690 0x55820248b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 -- 172.21.15.116:0/4233190919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202481690 msgr2=0x55820248b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202481690 0x55820248b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 -- 172.21.15.116:0/4233190919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 msgr2=0x5582024926f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x5582024926f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 -- 172.21.15.116:0/4233190919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582021c47d0 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb137400640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x5582024926f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb136a00640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202480ca0 0x55820248b170 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb130009610 tx=0x7fb130096180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4d8b8eb2d4f268 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb124c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb124c00640 1 -- 172.21.15.116:0/4233190919 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13009a020 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb124c00640 1 -- 172.21.15.116:0/4233190919 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb130096b50 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.097+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb120003970 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb124c00640 1 -- 172.21.15.116:0/4233190919 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb130099e40 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 10 client.27974 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x558202495ad0 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb124c00640 1 -- 172.21.15.116:0/4233190919 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb120003970 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb124c00640 1 -- 172.21.15.116:0/4233190919 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb1300cdbe0 con 0x558202480ca0 2024-03-04T07:51:06.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 10 client.27974 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 10 client.27974 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 20 client.27974 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024970c0 0x558202499580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 4 client.27974 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582023f32a0 con 0x5582024970c0 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.098+0000 7fb136000640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024970c0 0x558202499580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.099+0000 7fb124c00640 10 client.27974 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.099+0000 7fb136000640 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024970c0 0x558202499580 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8eb8b407c5c139c server_cookie=2a473debad48173d in_seq=0 out_seq=0 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb124c00640 1 -- 172.21.15.116:0/4233190919 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5582023f32a0 con 0x5582024970c0 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb124c00640 10 client.27974 handle_command_reply: tid=0 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 client.27974 shutdown 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202480ca0 msgr2=0x55820248b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202480ca0 0x55820248b170 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb130009610 tx=0x7fb130096180 comp rx=0 tx=0).stop 2024-03-04T07:51:06.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 shutdown_connections 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024970c0 0x558202499580 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x5582024926f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202481690 0x55820248b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 --2- 172.21.15.116:0/4233190919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202480ca0 0x55820248b170 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb124c00640 0 client.27974 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 >> 172.21.15.116:0/4233190919 conn(0x55820247c210 msgr2=0x55820247b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 shutdown_connections 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 1 -- 172.21.15.116:0/4233190919 wait complete. 2024-03-04T07:51:06.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.100+0000 7fb140a84740 20 client.27974 trim_cache size 0 max 16384 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 Processor -- start 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 -- start start 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x55820248ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x5582024926b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x55820248b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381acff0 con 0x558202481690 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381af5d0 con 0x558202486800 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb140a84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381ad5f0 con 0x558202480ca0 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb137400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x55820248b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb136000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x5582024926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb137400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x55820248b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59396/0 (socket says 172.21.15.116:59396) 2024-03-04T07:51:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb137400640 1 -- 172.21.15.116:0/422667601 learned_addr learned my addr 172.21.15.116:0/422667601 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.114+0000 7fb136a00640 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x55820248ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb137400640 1 -- 172.21.15.116:0/422667601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 msgr2=0x55820248ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb137400640 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x55820248ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb137400640 1 -- 172.21.15.116:0/422667601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 msgr2=0x5582024926b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb137400640 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x5582024926b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb137400640 1 -- 172.21.15.116:0/422667601 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558202481a70 con 0x558202486800 2024-03-04T07:51:06.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb137400640 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x55820248b2c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fb128012480 tx=0x7fb128014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f1cdcfd836f6468 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb127400640 1 -- 172.21.15.116:0/422667601 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128029670 con 0x558202486800 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb127400640 1 -- 172.21.15.116:0/422667601 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb128024070 con 0x558202486800 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.115+0000 7fb127400640 1 -- 172.21.15.116:0/422667601 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128020c60 con 0x558202486800 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 -- 172.21.15.116:0/422667601 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 msgr2=0x55820248b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x55820248b2c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fb128012480 tx=0x7fb128014000 comp rx=0 tx=0).stop 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 -- 172.21.15.116:0/422667601 shutdown_connections 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202486800 0x55820248b2c0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202481690 0x5582024926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 --2- 172.21.15.116:0/422667601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x55820248ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 -- 172.21.15.116:0/422667601 >> 172.21.15.116:0/422667601 conn(0x55820247c210 msgr2=0x558202482c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 -- 172.21.15.116:0/422667601 shutdown_connections 2024-03-04T07:51:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 -- 172.21.15.116:0/422667601 wait complete. 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.116+0000 7fb140a84740 1 -- start start 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 1 Processor -- start 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x558202490530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202481690 0x558202490a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x558202491a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381adc50 con 0x558202486800 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381aef70 con 0x558202481690 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381ae2b0 con 0x558202480ca0 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb140a84740 10 client.0 _ll_register_callbacks cb 0x7fb130004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb136000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202481690 0x558202490a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb137400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x558202491a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb136000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202481690 0x558202490a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59400/0 (socket says 172.21.15.116:59400) 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb136000640 1 -- 172.21.15.116:0/1220180413 learned_addr learned my addr 172.21.15.116:0/1220180413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.117+0000 7fb136a00640 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x558202490530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb136000640 1 -- 172.21.15.116:0/1220180413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 msgr2=0x558202490530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb136000640 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x558202490530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb136000640 1 -- 172.21.15.116:0/1220180413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 msgr2=0x558202491a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb136000640 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x558202491a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb136000640 1 -- 172.21.15.116:0/1220180413 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558202492070 con 0x558202481690 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb136000640 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202481690 0x558202490a70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fb120006010 tx=0x7fb12002c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c079f95e0df380 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb124c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb124c00640 1 -- 172.21.15.116:0/1220180413 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120056060 con 0x558202481690 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb128024070 con 0x558202481690 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.118+0000 7fb124c00640 1 -- 172.21.15.116:0/1220180413 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb120032890 con 0x558202481690 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb124c00640 1 -- 172.21.15.116:0/1220180413 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb120039040 con 0x558202481690 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 10 client.27980 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5582024f3190 con 0x558202481690 2024-03-04T07:51:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb124c00640 1 -- 172.21.15.116:0/1220180413 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb128024070 con 0x558202481690 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb124c00640 1 -- 172.21.15.116:0/1220180413 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb12005b070 con 0x558202481690 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 10 client.27980 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 10 client.27980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 20 client.27980 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024f36f0 0x5582024f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 4 client.27980 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5582023ed5f0 con 0x5582024f36f0 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.119+0000 7fb136a00640 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024f36f0 0x5582024f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb136a00640 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024f36f0 0x5582024f5ae0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9b99bdd0a9150e server_cookie=7fffcdbce63b3ce in_seq=0 out_seq=0 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb124c00640 10 client.27980 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb124c00640 1 -- 172.21.15.116:0/1220180413 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5582023ed5f0 con 0x5582024f36f0 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb124c00640 10 client.27980 handle_command_reply: tid=0 2024-03-04T07:51:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 client.27980 shutdown 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202481690 msgr2=0x558202490a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202481690 0x558202490a70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fb120006010 tx=0x7fb12002c100 comp rx=0 tx=0).stop 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 shutdown_connections 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5582024f36f0 0x5582024f5ae0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558202486800 0x558202491a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x558202481690 0x558202490a70 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 --2- 172.21.15.116:0/1220180413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558202480ca0 0x558202490530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb124c00640 0 client.27980 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 >> 172.21.15.116:0/1220180413 conn(0x55820247c210 msgr2=0x55820248ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 shutdown_connections 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 1 -- 172.21.15.116:0/1220180413 wait complete. 2024-03-04T07:51:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.120+0000 7fb140a84740 20 client.27980 trim_cache size 0 max 16384 2024-03-04T07:51:06.126 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:51:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb11c0a1e00 msgr2=0x7fb11c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb11c0a1e00 0x7fb11c0a42c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x558202481ef0 tx=0x7fb1280093d0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb138075f90 msgr2=0x7fb13819ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb138075f90 0x7fb13819ded0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb13000d950 tx=0x7fb13000de10 comp rx=0 tx=0).stop 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 shutdown_connections 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb11c0a1e00 0x7fb11c0a42c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb138077b70 0x7fb1381a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1380771d0 0x7fb13819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 --2- 172.21.15.116:0/3954471329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb138075f90 0x7fb13819ded0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 >> 172.21.15.116:0/3954471329 conn(0x7fb1380fed80 msgr2=0x7fb138100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 shutdown_connections 2024-03-04T07:51:06.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:06.121+0000 7fb13de00640 1 -- 172.21.15.116:0/3954471329 wait complete. 2024-03-04T07:51:06.136 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:05 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4146695226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:51:06.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:05 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4146695226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:51:06.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:05 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4146695226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:51:07.141 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:51:07.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:51:07.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:51:07.222 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:06 smithi116 ceph-mon[32150]: pgmap v711: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:51:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:06 smithi152 ceph-mon[37154]: pgmap v711: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:51:07.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:06 smithi133 ceph-mon[36505]: pgmap v711: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-03-04T07:51:07.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 -- 172.21.15.116:0/2462389271 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210075610 msgr2=0x7f5210073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 --2- 172.21.15.116:0/2462389271 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210075610 0x7f5210073a60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5200009be0 tx=0x7f520002bef0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 -- 172.21.15.116:0/2462389271 shutdown_connections 2024-03-04T07:51:07.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 --2- 172.21.15.116:0/2462389271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101094a0 0x7f521010b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 --2- 172.21.15.116:0/2462389271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5210074030 0x7f52100744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 --2- 172.21.15.116:0/2462389271 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210075610 0x7f5210073a60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 -- 172.21.15.116:0/2462389271 >> 172.21.15.116:0/2462389271 conn(0x7f52100fcb70 msgr2=0x7f52100fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 -- 172.21.15.116:0/2462389271 shutdown_connections 2024-03-04T07:51:07.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.362+0000 7f5216e00640 1 -- 172.21.15.116:0/2462389271 wait complete. 2024-03-04T07:51:07.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 Processor -- start 2024-03-04T07:51:07.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 -- start start 2024-03-04T07:51:07.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210074030 0x7f5210199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5210075610 0x7f521019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101094a0 0x7f52101a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521010f440 con 0x7f52101094a0 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521010f2c0 con 0x7f5210074030 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.363+0000 7f5216e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521010f5c0 con 0x7f5210075610 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210074030 0x7f5210199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210074030 0x7f5210199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59430/0 (socket says 172.21.15.116:59430) 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 -- 172.21.15.116:0/320980428 learned_addr learned my addr 172.21.15.116:0/320980428 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f5214e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101094a0 0x7f52101a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520f400640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5210075610 0x7f521019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 -- 172.21.15.116:0/320980428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5210075610 msgr2=0x7f521019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5210075610 0x7f521019a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 -- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101094a0 msgr2=0x7f52101a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101094a0 0x7f52101a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 -- 172.21.15.116:0/320980428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52101a1780 con 0x7f5210074030 2024-03-04T07:51:07.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520fe00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210074030 0x7f5210199b60 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5200006010 tx=0x7f5200033830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.364+0000 7f520cc00640 1 -- 172.21.15.116:0/320980428 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200043020 con 0x7f5210074030 2024-03-04T07:51:07.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.365+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52101a1a10 con 0x7f5210074030 2024-03-04T07:51:07.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.365+0000 7f520cc00640 1 -- 172.21.15.116:0/320980428 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f520002ca60 con 0x7f5210074030 2024-03-04T07:51:07.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.365+0000 7f520cc00640 1 -- 172.21.15.116:0/320980428 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52000347b0 con 0x7f5210074030 2024-03-04T07:51:07.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.365+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52101a1fb0 con 0x7f5210074030 2024-03-04T07:51:07.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 Processor -- start 2024-03-04T07:51:07.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f520cc00640 1 -- 172.21.15.116:0/320980428 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5200046020 con 0x7f5210074030 2024-03-04T07:51:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 -- start start 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2bf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193f1d2e20 con 0x56193f2bf0d0 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193f224a20 con 0x56193f2bfac0 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5219b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193f1cdb00 con 0x56193f2c4c30 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f520fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5214e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2bf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f520fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2c45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59438/0 (socket says 172.21.15.116:59438) 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f5214e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2bf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35512/0 (socket says 172.21.15.116:35512) 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.368+0000 7f520fe00640 1 -- 172.21.15.116:0/10876792 learned_addr learned my addr 172.21.15.116:0/10876792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520f400640 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520fe00640 1 -- 172.21.15.116:0/10876792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 msgr2=0x56193f2c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520fe00640 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520fe00640 1 -- 172.21.15.116:0/10876792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 msgr2=0x56193f2bf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520fe00640 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2bf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520fe00640 1 -- 172.21.15.116:0/10876792 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56193f1e2e10 con 0x56193f2bfac0 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f5214e00640 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2bf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520cc00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f51f40a1d80 0x7f51f40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520fe00640 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2c45c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f52000f8430 tx=0x7f52000f88f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=628a00ff1a687408 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f520cc00640 1 -- 172.21.15.116:0/320980428 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f52000f04c0 con 0x7f5210074030 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f5205600640 1 -- 172.21.15.116:0/10876792 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52000b9450 con 0x56193f2bfac0 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.369+0000 7f5214e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f51f40a1d80 0x7f51f40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5205600640 1 -- 172.21.15.116:0/10876792 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52000bd8b0 con 0x56193f2bfac0 2024-03-04T07:51:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5205600640 1 -- 172.21.15.116:0/10876792 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52000bcb10 con 0x56193f2bfac0 2024-03-04T07:51:07.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5219b6f740 1 -- 172.21.15.116:0/10876792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 msgr2=0x56193f2c45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5219b6f740 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2c45c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f52000f8430 tx=0x7f52000f88f0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5214e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f51f40a1d80 0x7f51f40a4240 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x56193f2c0320 tx=0x7f5208009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5219b6f740 1 -- 172.21.15.116:0/10876792 shutdown_connections 2024-03-04T07:51:07.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5219b6f740 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5219b6f740 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2c45c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5219b6f740 1 --2- 172.21.15.116:0/10876792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2bf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.370+0000 7f5219b6f740 1 -- 172.21.15.116:0/10876792 >> 172.21.15.116:0/10876792 conn(0x56193f2ba640 msgr2=0x56193f2baa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.371+0000 7f5219b6f740 1 -- 172.21.15.116:0/10876792 shutdown_connections 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.371+0000 7f5219b6f740 1 -- 172.21.15.116:0/10876792 wait complete. 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.371+0000 7f5219b6f740 1 -- start start 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.371+0000 7f5219b6f740 1 Processor -- start 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bf0d0 0x56193f2c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bfac0 0x56193f2c9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5219b6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193f1e0f40 con 0x56193f2bfac0 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5219b6f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193f220ac0 con 0x56193f2bf0d0 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5219b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56193f224a20 con 0x56193f2c4c30 2024-03-04T07:51:07.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5219b6f740 10 client.0 _ll_register_callbacks cb 0x56193f21faa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bf0d0 0x56193f2c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bf0d0 0x56193f2c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59452/0 (socket says 172.21.15.116:59452) 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 -- 172.21.15.116:0/741169311 learned_addr learned my addr 172.21.15.116:0/741169311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520f400640 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bfac0 0x56193f2c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f5214e00640 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 -- 172.21.15.116:0/741169311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 msgr2=0x56193f2d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 -- 172.21.15.116:0/741169311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bfac0 msgr2=0x56193f2c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bfac0 0x56193f2c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.372+0000 7f520fe00640 1 -- 172.21.15.116:0/741169311 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56193f0017e0 con 0x56193f2bf0d0 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f520fe00640 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bf0d0 0x56193f2c9100 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f52000c5420 tx=0x7f5200033a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93298410c8990e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f51fa000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f51fa000640 1 -- 172.21.15.116:0/741169311 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52000bccf0 con 0x56193f2bf0d0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f51fa000640 1 -- 172.21.15.116:0/741169311 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f52000f00d0 con 0x56193f2bf0d0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52000bd8b0 con 0x56193f2bf0d0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f51fa000640 1 -- 172.21.15.116:0/741169311 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52000bcf80 con 0x56193f2bf0d0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f5219b6f740 10 client.27992 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56193f061f70 con 0x56193f2bf0d0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f51fa000640 1 -- 172.21.15.116:0/741169311 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52000bd8b0 con 0x56193f2bf0d0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f51fa000640 1 -- 172.21.15.116:0/741169311 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f52000bd410 con 0x56193f2bf0d0 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.373+0000 7f5219b6f740 10 client.27992 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:07.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f5219b6f740 10 client.27992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f5219b6f740 20 client.27992 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f5219b6f740 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f2d4c50 0x56193f2d7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f5219b6f740 4 client.27992 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56193f231f60 con 0x56193f2d4c50 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f520f400640 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f2d4c50 0x56193f2d7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f51fa000640 10 client.27992 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.374+0000 7f520f400640 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f2d4c50 0x56193f2d7040 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3dd784fb2993159 server_cookie=cd309ee3b15fbb39 in_seq=0 out_seq=0 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f51fa000640 1 -- 172.21.15.116:0/741169311 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56193f231f60 con 0x56193f2d4c50 2024-03-04T07:51:07.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f51fa000640 10 client.27992 handle_command_reply: tid=0 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 client.27992 shutdown 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bf0d0 msgr2=0x56193f2c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bf0d0 0x56193f2c9100 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f52000c5420 tx=0x7f5200033a20 comp rx=0 tx=0).stop 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 shutdown_connections 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f2d4c50 0x56193f2d7040 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2c4c30 0x56193f2d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bfac0 0x56193f2c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 --2- 172.21.15.116:0/741169311 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bf0d0 0x56193f2c9100 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f51fa000640 0 client.27992 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.375+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 >> 172.21.15.116:0/741169311 conn(0x56193f2ba640 msgr2=0x56193f2bb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.376+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 shutdown_connections 2024-03-04T07:51:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.376+0000 7f5219b6f740 1 -- 172.21.15.116:0/741169311 wait complete. 2024-03-04T07:51:07.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.376+0000 7f5219b6f740 20 client.27992 trim_cache size 0 max 16384 2024-03-04T07:51:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 Processor -- start 2024-03-04T07:51:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 -- start start 2024-03-04T07:51:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2c0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bfac0 0x56193f2d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2c4c30 0x56193f2b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52101a8d40 con 0x56193f2bf0d0 2024-03-04T07:51:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52101ab320 con 0x56193f2c4c30 2024-03-04T07:51:07.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.393+0000 7f5219b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52101a9340 con 0x56193f2bfac0 2024-03-04T07:51:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2c4c30 0x56193f2b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2c4c30 0x56193f2b98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59460/0 (socket says 172.21.15.116:59460) 2024-03-04T07:51:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 -- 172.21.15.116:0/1755373667 learned_addr learned my addr 172.21.15.116:0/1755373667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f520fe00640 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f520f400640 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bfac0 0x56193f2d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 -- 172.21.15.116:0/1755373667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bfac0 msgr2=0x56193f2d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bfac0 0x56193f2d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 -- 172.21.15.116:0/1755373667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 msgr2=0x56193f2c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2c0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 -- 172.21.15.116:0/1755373667 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56193eef9170 con 0x56193f2c4c30 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f520fe00640 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.394+0000 7f5214e00640 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2c4c30 0x56193f2b98e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5208004360 tx=0x7f520801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b612fa9d57bdd1fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5204c00640 1 -- 172.21.15.116:0/1755373667 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208004450 con 0x56193f2c4c30 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5204c00640 1 -- 172.21.15.116:0/1755373667 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f520802a960 con 0x56193f2c4c30 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5204c00640 1 -- 172.21.15.116:0/1755373667 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520801fa60 con 0x56193f2c4c30 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 -- 172.21.15.116:0/1755373667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2c4c30 msgr2=0x56193f2b98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2c4c30 0x56193f2b98e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5208004360 tx=0x7f520801b000 comp rx=0 tx=0).stop 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 -- 172.21.15.116:0/1755373667 shutdown_connections 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2c4c30 0x56193f2b98e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bfac0 0x56193f2d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 --2- 172.21.15.116:0/1755373667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2bf0d0 0x56193f2c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 -- 172.21.15.116:0/1755373667 >> 172.21.15.116:0/1755373667 conn(0x56193f2ba640 msgr2=0x56193f2b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 -- 172.21.15.116:0/1755373667 shutdown_connections 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.395+0000 7f5219b6f740 1 -- 172.21.15.116:0/1755373667 wait complete. 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 -- start start 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 Processor -- start 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bf0d0 0x56193f2ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2c4c30 0x56193f2cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52101a99a0 con 0x56193f2c4c30 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52101aacc0 con 0x56193f2bfac0 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52101aa000 con 0x56193f2bf0d0 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.396+0000 7f5219b6f740 10 client.0 _ll_register_callbacks cb 0x7f5208009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59474/0 (socket says 172.21.15.116:59474) 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 -- 172.21.15.116:0/2060287232 learned_addr learned my addr 172.21.15.116:0/2060287232 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520fe00640 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bf0d0 0x56193f2ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f5214e00640 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2c4c30 0x56193f2cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 -- 172.21.15.116:0/2060287232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bf0d0 msgr2=0x56193f2ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bf0d0 0x56193f2ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 -- 172.21.15.116:0/2060287232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2c4c30 msgr2=0x56193f2cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2c4c30 0x56193f2cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 -- 172.21.15.116:0/2060287232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56193f2d02a0 con 0x56193f2bfac0 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f520f400640 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2cede0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x56193f2bb5b0 tx=0x7f51fc01eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7a7359306c26ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.397+0000 7f51fa000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f51fa000640 1 -- 172.21.15.116:0/2060287232 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc0142d0 con 0x56193f2bfac0 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f51fa000640 1 -- 172.21.15.116:0/2060287232 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f51fc0149a0 con 0x56193f2bfac0 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f520802a960 con 0x56193f2bfac0 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f51fa000640 1 -- 172.21.15.116:0/2060287232 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51fc014ba0 con 0x56193f2bfac0 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 10 client.27998 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:07.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56193f330f80 con 0x56193f2bfac0 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f51fa000640 1 -- 172.21.15.116:0/2060287232 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f520802a960 con 0x56193f2bfac0 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f51fa000640 1 -- 172.21.15.116:0/2060287232 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f51fc026410 con 0x56193f2bfac0 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 10 client.27998 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 10 client.27998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 20 client.27998 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f331800 0x56193f333ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 4 client.27998 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.398+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56193f22c2b0 con 0x56193f331800 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.399+0000 7f520fe00640 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f331800 0x56193f333ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.399+0000 7f51fa000640 10 client.27998 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.399+0000 7f520fe00640 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f331800 0x56193f333ca0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6163520279ae66b7 server_cookie=fc3950d4bcf2316d in_seq=0 out_seq=0 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f51fa000640 1 -- 172.21.15.116:0/2060287232 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56193f22c2b0 con 0x56193f331800 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f51fa000640 10 client.27998 handle_command_reply: tid=0 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 client.27998 shutdown 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 msgr2=0x56193f2cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2cede0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x56193f2bb5b0 tx=0x7f51fc01eb10 comp rx=0 tx=0).stop 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 shutdown_connections 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56193f331800 0x56193f333ca0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56193f2c4c30 0x56193f2cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f51fa000640 0 client.27998 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56193f2bfac0 0x56193f2cede0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 --2- 172.21.15.116:0/2060287232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56193f2bf0d0 0x56193f2ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 >> 172.21.15.116:0/2060287232 conn(0x56193f2ba640 msgr2=0x56193f2bd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 shutdown_connections 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.400+0000 7f5219b6f740 1 -- 172.21.15.116:0/2060287232 wait complete. 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5219b6f740 20 client.27998 trim_cache size 0 max 16384 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "65bdfa53-8afe-43d6-8781-f2da7ab34ced", 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:51:07.406 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:51:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f51f40a1d80 msgr2=0x7f51f40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f51f40a1d80 0x7f51f40a4240 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x56193f2c0320 tx=0x7f5208009260 comp rx=0 tx=0).stop 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210074030 msgr2=0x7f5210199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210074030 0x7f5210199b60 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5200006010 tx=0x7f5200033830 comp rx=0 tx=0).stop 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 shutdown_connections 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52101094a0 0x7f52101a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f51f40a1d80 0x7f51f40a4240 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5210075610 0x7f521019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 --2- 172.21.15.116:0/320980428 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5210074030 0x7f5210199b60 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 >> 172.21.15.116:0/320980428 conn(0x7f52100fcb70 msgr2=0x7f52101053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 shutdown_connections 2024-03-04T07:51:07.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.401+0000 7f5216e00640 1 -- 172.21.15.116:0/320980428 wait complete. 2024-03-04T07:51:07.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:51:07.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:51:07.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 -- 172.21.15.116:0/964051824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac108de0 msgr2=0x7f2dac109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 --2- 172.21.15.116:0/964051824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac108de0 0x7f2dac109260 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009900 tx=0x7f2d9c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:51:07.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 -- 172.21.15.116:0/964051824 shutdown_connections 2024-03-04T07:51:07.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 --2- 172.21.15.116:0/964051824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dac1097a0 0x7f2dac110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 --2- 172.21.15.116:0/964051824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac108de0 0x7f2dac109260 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 --2- 172.21.15.116:0/964051824 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac107bf0 0x7f2dac108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 -- 172.21.15.116:0/964051824 >> 172.21.15.116:0/964051824 conn(0x7f2dac076b90 msgr2=0x7f2dac078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 -- 172.21.15.116:0/964051824 shutdown_connections 2024-03-04T07:51:07.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.626+0000 7f2db4800640 1 -- 172.21.15.116:0/964051824 wait complete. 2024-03-04T07:51:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 Processor -- start 2024-03-04T07:51:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 -- start start 2024-03-04T07:51:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dac107bf0 0x7f2dac19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac108de0 0x7f2dac19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac1097a0 0x7f2dac1a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1138c0 con 0x7f2dac1097a0 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac113740 con 0x7f2dac108de0 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.627+0000 7f2db4800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac113a40 con 0x7f2dac107bf0 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac108de0 0x7f2dac19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac108de0 0x7f2dac19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59492/0 (socket says 172.21.15.116:59492) 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac1097a0 0x7f2dac1a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 -- 172.21.15.116:0/613711256 learned_addr learned my addr 172.21.15.116:0/613711256 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1e00640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dac107bf0 0x7f2dac19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 -- 172.21.15.116:0/613711256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dac107bf0 msgr2=0x7f2dac19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dac107bf0 0x7f2dac19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 -- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac1097a0 msgr2=0x7f2dac1a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac1097a0 0x7f2dac1a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 -- 172.21.15.116:0/613711256 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac1a5ad0 con 0x7f2dac108de0 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2db1400640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac108de0 0x7f2dac19e3f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000c00 tx=0x7f2d9c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.628+0000 7f2da6a00640 1 -- 172.21.15.116:0/613711256 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c043020 con 0x7f2dac108de0 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.629+0000 7f2da6a00640 1 -- 172.21.15.116:0/613711256 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d9c0054e0 con 0x7f2dac108de0 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.629+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac1a5d60 con 0x7f2dac108de0 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.629+0000 7f2da6a00640 1 -- 172.21.15.116:0/613711256 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c003c40 con 0x7f2dac108de0 2024-03-04T07:51:07.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.629+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac1a6250 con 0x7f2dac108de0 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 Processor -- start 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2da6a00640 1 -- 172.21.15.116:0/613711256 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2d9c046020 con 0x7f2dac108de0 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 -- start start 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45a9b50 0x55a7b45a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b44d7090 con 0x55a7b45af6b0 2024-03-04T07:51:07.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b450ed30 con 0x55a7b45a9b50 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.633+0000 7f2db756d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b44b7a30 con 0x55a7b45aa540 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45a9b50 0x55a7b45a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45a9b50 0x55a7b45a9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59508/0 (socket says 172.21.15.116:59508) 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 -- 172.21.15.116:0/2851038799 learned_addr learned my addr 172.21.15.116:0/2851038799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db1400640 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db1e00640 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 -- 172.21.15.116:0/2851038799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 msgr2=0x55a7b45af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 -- 172.21.15.116:0/2851038799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 msgr2=0x55a7b45b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 -- 172.21.15.116:0/2851038799 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7b42ed7d0 con 0x55a7b45a9b50 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db1e00640 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2db2800640 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45a9b50 0x55a7b45a9f70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009fd0 tx=0x7f2da80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff78f6383ef9ee05 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.634+0000 7f2da6a00640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2d940a1d80 0x7f2d940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2d9b000640 1 -- 172.21.15.116:0/2851038799 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8004d20 con 0x55a7b45a9b50 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2d9b000640 1 -- 172.21.15.116:0/2851038799 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2da8004050 con 0x55a7b45a9b50 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2d9b000640 1 -- 172.21.15.116:0/2851038799 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8005260 con 0x55a7b45a9b50 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2da6a00640 1 -- 172.21.15.116:0/613711256 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2d9c0efde0 con 0x7f2dac108de0 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db1e00640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2d940a1d80 0x7f2d940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db756d740 1 -- 172.21.15.116:0/2851038799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45a9b50 msgr2=0x55a7b45a9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db756d740 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45a9b50 0x55a7b45a9f70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009fd0 tx=0x7f2da80118d0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db756d740 1 -- 172.21.15.116:0/2851038799 shutdown_connections 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db756d740 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db756d740 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db756d740 1 --2- 172.21.15.116:0/2851038799 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45a9b50 0x55a7b45a9f70 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.635+0000 7f2db756d740 1 -- 172.21.15.116:0/2851038799 >> 172.21.15.116:0/2851038799 conn(0x55a7b45a50c0 msgr2=0x55a7b45a54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.636+0000 7f2db756d740 1 -- 172.21.15.116:0/2851038799 shutdown_connections 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.636+0000 7f2db1e00640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2d940a1d80 0x7f2d940a4240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2da00095f0 tx=0x7f2da0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.636+0000 7f2db756d740 1 -- 172.21.15.116:0/2851038799 wait complete. 2024-03-04T07:51:07.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.636+0000 7f2db756d740 1 -- start start 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.636+0000 7f2db756d740 1 Processor -- start 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db756d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45a9b50 0x55a7b45b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db756d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45b4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db756d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45bb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db756d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b44ad0c0 con 0x55a7b45af6b0 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db756d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b450add0 con 0x55a7b45aa540 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db756d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7b45b2660 con 0x55a7b45a9b50 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db756d740 10 client.0 _ll_register_callbacks cb 0x55a7b44ca860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db2800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45bb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45b4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59524/0 (socket says 172.21.15.116:59524) 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 -- 172.21.15.116:0/3860584900 learned_addr learned my addr 172.21.15.116:0/3860584900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1e00640 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45a9b50 0x55a7b45b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 -- 172.21.15.116:0/3860584900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45a9b50 msgr2=0x55a7b45b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45a9b50 0x55a7b45b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 -- 172.21.15.116:0/3860584900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 msgr2=0x55a7b45bb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45bb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 -- 172.21.15.116:0/3860584900 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7b42233f0 con 0x55a7b45aa540 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.637+0000 7f2db1400640 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45b4490 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0b94e0 tx=0x7f2d9c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=186cf96fd324c515 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2d8f400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2d8f400640 1 -- 172.21.15.116:0/3860584900 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c0bc650 con 0x55a7b45aa540 2024-03-04T07:51:07.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2d8f400640 1 -- 172.21.15.116:0/3860584900 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2d9c0ed9f0 con 0x55a7b45aa540 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2da8004050 con 0x55a7b45aa540 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2d8f400640 1 -- 172.21.15.116:0/3860584900 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d9c0bc470 con 0x55a7b45aa540 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 10 client.28010 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a7b41e30f0 con 0x55a7b45aa540 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2d8f400640 1 -- 172.21.15.116:0/3860584900 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2da8004050 con 0x55a7b45aa540 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2d8f400640 1 -- 172.21.15.116:0/3860584900 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2d9c0c4e20 con 0x55a7b45aa540 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 10 client.28010 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 10 client.28010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 20 client.28010 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b45bf9a0 0x55a7b45c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 4 client.28010 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.638+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a7b451c270 con 0x55a7b45bf9a0 2024-03-04T07:51:07.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.639+0000 7f2db1e00640 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b45bf9a0 0x55a7b45c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.639+0000 7f2db1e00640 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b45bf9a0 0x55a7b45c1dc0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65d466ab9ff120c5 server_cookie=49aec263b9fe6d70 in_seq=0 out_seq=0 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.639+0000 7f2d8f400640 10 client.28010 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2d8f400640 1 -- 172.21.15.116:0/3860584900 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a7b451c270 con 0x55a7b45bf9a0 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2d8f400640 10 client.28010 handle_command_reply: tid=0 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 client.28010 shutdown 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 msgr2=0x55a7b45b4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45b4490 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0b94e0 tx=0x7f2d9c009560 comp rx=0 tx=0).stop 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 shutdown_connections 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b45bf9a0 0x55a7b45c1dc0 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45af6b0 0x55a7b45bb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45b4490 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 --2- 172.21.15.116:0/3860584900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45a9b50 0x55a7b45b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2d8f400640 0 client.28010 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 >> 172.21.15.116:0/3860584900 conn(0x55a7b45a50c0 msgr2=0x55a7b45a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.640+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 shutdown_connections 2024-03-04T07:51:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.641+0000 7f2db756d740 1 -- 172.21.15.116:0/3860584900 wait complete. 2024-03-04T07:51:07.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.641+0000 7f2db756d740 20 client.28010 trim_cache size 0 max 16384 2024-03-04T07:51:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 Processor -- start 2024-03-04T07:51:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 -- start start 2024-03-04T07:51:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45abad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45bb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45af6b0 0x55a7b45ac010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1acff0 con 0x55a7b45a9b50 2024-03-04T07:51:07.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1af5d0 con 0x55a7b45af6b0 2024-03-04T07:51:07.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db756d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1ad5f0 con 0x55a7b45aa540 2024-03-04T07:51:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45af6b0 0x55a7b45ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.658+0000 7f2db2800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45af6b0 0x55a7b45ac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59538/0 (socket says 172.21.15.116:59538) 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db2800640 1 -- 172.21.15.116:0/2536044089 learned_addr learned my addr 172.21.15.116:0/2536044089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db1e00640 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db1400640 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db2800640 1 -- 172.21.15.116:0/2536044089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 msgr2=0x55a7b45bb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db2800640 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45bb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db2800640 1 -- 172.21.15.116:0/2536044089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 msgr2=0x55a7b45abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db2800640 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45abad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db2800640 1 -- 172.21.15.116:0/2536044089 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7b45c2420 con 0x55a7b45af6b0 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db1400640 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45bb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2db2800640 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45af6b0 0x55a7b45ac010 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009fd0 tx=0x7f2da8003bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=104277a4366fd3bc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2d9a600640 1 -- 172.21.15.116:0/2536044089 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8003da0 con 0x55a7b45af6b0 2024-03-04T07:51:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2d9a600640 1 -- 172.21.15.116:0/2536044089 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2da8004e50 con 0x55a7b45af6b0 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.659+0000 7f2d9a600640 1 -- 172.21.15.116:0/2536044089 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8005030 con 0x55a7b45af6b0 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 -- 172.21.15.116:0/2536044089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45af6b0 msgr2=0x55a7b45ac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45af6b0 0x55a7b45ac010 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009fd0 tx=0x7f2da8003bb0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 -- 172.21.15.116:0/2536044089 shutdown_connections 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45af6b0 0x55a7b45ac010 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45aa540 0x55a7b45bb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 --2- 172.21.15.116:0/2536044089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 -- 172.21.15.116:0/2536044089 >> 172.21.15.116:0/2536044089 conn(0x55a7b45a50c0 msgr2=0x55a7b45b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 -- 172.21.15.116:0/2536044089 shutdown_connections 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.660+0000 7f2db756d740 1 -- 172.21.15.116:0/2536044089 wait complete. 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 -- start start 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 Processor -- start 2024-03-04T07:51:07.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45bad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45bb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45af6b0 0x55a7b45b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1adc50 con 0x55a7b45a9b50 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1aef70 con 0x55a7b45aa540 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac1ae2b0 con 0x55a7b45af6b0 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db756d740 10 client.0 _ll_register_callbacks cb 0x7f2d9c0f3550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45bad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45bb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db1400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45bb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:59552/0 (socket says 172.21.15.116:59552) 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.661+0000 7f2db1400640 1 -- 172.21.15.116:0/1565273454 learned_addr learned my addr 172.21.15.116:0/1565273454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db2800640 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45af6b0 0x55a7b45b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db1400640 1 -- 172.21.15.116:0/1565273454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45af6b0 msgr2=0x55a7b45b9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db1400640 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45af6b0 0x55a7b45b9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db1400640 1 -- 172.21.15.116:0/1565273454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 msgr2=0x55a7b45bad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db1400640 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45bad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db1400640 1 -- 172.21.15.116:0/1565273454 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7b45b9b30 con 0x55a7b45aa540 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db2800640 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45af6b0 0x55a7b45b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db1400640 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45bb240 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0b9490 tx=0x7f2d9c0bc6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e1bde9256f4a491 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2d8f400640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2d8f400640 1 -- 172.21.15.116:0/1565273454 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c0db060 con 0x55a7b45aa540 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2d8f400640 1 -- 172.21.15.116:0/1565273454 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2d9c0d0b60 con 0x55a7b45aa540 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2da8004e50 con 0x55a7b45aa540 2024-03-04T07:51:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2d8f400640 1 -- 172.21.15.116:0/1565273454 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d9c0a2050 con 0x55a7b45aa540 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db756d740 10 client.28016 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.662+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a7b461bc40 con 0x55a7b45aa540 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2d8f400640 1 -- 172.21.15.116:0/1565273454 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2da8004e50 con 0x55a7b45aa540 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2d8f400640 1 -- 172.21.15.116:0/1565273454 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2d9c0bbf60 con 0x55a7b45aa540 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2db756d740 10 client.28016 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2db756d740 10 client.28016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2db756d740 20 client.28016 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2db756d740 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b461c460 0x55a7b461e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2db756d740 4 client.28016 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a7b45165c0 con 0x55a7b461c460 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.663+0000 7f2db1e00640 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b461c460 0x55a7b461e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db1e00640 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b461c460 0x55a7b461e880 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5320e9915f0a546d server_cookie=5dfa472cbcc60101 in_seq=0 out_seq=0 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2d8f400640 10 client.28016 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2d8f400640 1 -- 172.21.15.116:0/1565273454 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a7b45165c0 con 0x55a7b461c460 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2d8f400640 10 client.28016 handle_command_reply: tid=0 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 client.28016 shutdown 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 msgr2=0x55a7b45bb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45bb240 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0b9490 tx=0x7f2d9c0bc6c0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 shutdown_connections 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a7b461c460 0x55a7b461e880 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a7b45af6b0 0x55a7b45b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a7b45aa540 0x55a7b45bb240 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 --2- 172.21.15.116:0/1565273454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7b45a9b50 0x55a7b45bad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2d8f400640 0 client.28016 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:07.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 >> 172.21.15.116:0/1565273454 conn(0x55a7b45a50c0 msgr2=0x55a7b45b3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 shutdown_connections 2024-03-04T07:51:07.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 1 -- 172.21.15.116:0/1565273454 wait complete. 2024-03-04T07:51:07.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.664+0000 7f2db756d740 20 client.28016 trim_cache size 0 max 16384 2024-03-04T07:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:51:07.672 INFO:teuthology.orchestra.run.smithi116.stdout: "65bdfa53-8afe-43d6-8781-f2da7ab34ced": { 2024-03-04T07:51:07.672 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:51:07.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "65bdfa53-8afe-43d6-8781-f2da7ab34ced", 2024-03-04T07:51:07.672 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:51:07.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:51:07.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:51:07.672 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:51:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2d940a1d80 msgr2=0x7f2d940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2d940a1d80 0x7f2d940a4240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2da00095f0 tx=0x7f2da0005f90 comp rx=0 tx=0).stop 2024-03-04T07:51:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac108de0 msgr2=0x7f2dac19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac108de0 0x7f2dac19e3f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000c00 tx=0x7f2d9c02c650 comp rx=0 tx=0).stop 2024-03-04T07:51:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 shutdown_connections 2024-03-04T07:51:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac1097a0 0x7f2dac1a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2dac108de0 0x7f2dac19e3f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f2d940a1d80 0x7f2d940a4240 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 --2- 172.21.15.116:0/613711256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dac107bf0 0x7f2dac19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 >> 172.21.15.116:0/613711256 conn(0x7f2dac076b90 msgr2=0x7f2dac10c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 shutdown_connections 2024-03-04T07:51:07.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:07.665+0000 7f2db4800640 1 -- 172.21.15.116:0/613711256 wait complete. 2024-03-04T07:51:07.682 INFO:tasks.cephfs.filesystem:scrub status for tag:65bdfa53-8afe-43d6-8781-f2da7ab34ced - {'path': '/', 'tag': '65bdfa53-8afe-43d6-8781-f2da7ab34ced', 'options': 'recursive,force'} 2024-03-04T07:51:08.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:07 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:51:08.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:07 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:08.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:07 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:51:08.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:07 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:08.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:07 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:51:08.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:07 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:51:08.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:07 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:08.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:07 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:51:08.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:07 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:08.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:07 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:51:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:07 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:51:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:07 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:07 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:51:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:07 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:08.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:07 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:51:09.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:08 smithi133 ceph-mon[36505]: pgmap v712: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:08 smithi116 ceph-mon[32150]: pgmap v712: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:08 smithi152 ceph-mon[37154]: pgmap v712: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:10 smithi116 ceph-mon[32150]: pgmap v713: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:51:11.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:10 smithi152 ceph-mon[37154]: pgmap v713: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:51:11.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:10 smithi133 ceph-mon[36505]: pgmap v713: 97 pgs: 97 active+clean; 3.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:51:13.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:12 smithi116 ceph-mon[32150]: pgmap v714: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:12 smithi152 ceph-mon[37154]: pgmap v714: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:12 smithi133 ceph-mon[36505]: pgmap v714: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:15.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:14 smithi116 ceph-mon[32150]: pgmap v715: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:51:15.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:14 smithi152 ceph-mon[37154]: pgmap v715: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:51:15.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:14 smithi133 ceph-mon[36505]: pgmap v715: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-04T07:51:17.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:16 smithi116 ceph-mon[32150]: pgmap v716: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:16 smithi152 ceph-mon[37154]: pgmap v716: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:16 smithi133 ceph-mon[36505]: pgmap v716: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.340+0000 7feac9c00640 1 -- 172.21.15.116:0/529941744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4109600 msgr2=0x7feac410fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.340+0000 7feac9c00640 1 --2- 172.21.15.116:0/529941744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4109600 0x7feac410fe90 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7feabc009ca0 tx=0x7feabc02c100 comp rx=0 tx=0).stop 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.340+0000 7feac9c00640 1 -- 172.21.15.116:0/529941744 shutdown_connections 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.340+0000 7feac9c00640 1 --2- 172.21.15.116:0/529941744 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4109600 0x7feac410fe90 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.340+0000 7feac9c00640 1 --2- 172.21.15.116:0/529941744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac4108c40 0x7feac41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.340+0000 7feac9c00640 1 --2- 172.21.15.116:0/529941744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4107a50 0x7feac4107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.341+0000 7feac9c00640 1 -- 172.21.15.116:0/529941744 >> 172.21.15.116:0/529941744 conn(0x7feac4076a10 msgr2=0x7feac4078e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.341+0000 7feac9c00640 1 -- 172.21.15.116:0/529941744 shutdown_connections 2024-03-04T07:51:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.341+0000 7feac9c00640 1 -- 172.21.15.116:0/529941744 wait complete. 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.341+0000 7feac9c00640 1 Processor -- start 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9c00640 1 -- start start 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac4107a50 0x7feac419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4108c40 0x7feac419e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4109600 0x7feac41a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41136b0 con 0x7feac4109600 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac4113530 con 0x7feac4108c40 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac4113830 con 0x7feac4107a50 2024-03-04T07:51:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4109600 0x7feac41a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4109600 0x7feac41a5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47912/0 (socket says 172.21.15.116:47912) 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac9200640 1 -- 172.21.15.116:0/3744489037 learned_addr learned my addr 172.21.15.116:0/3744489037 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac3400640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4108c40 0x7feac419e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.342+0000 7feac3e00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac4107a50 0x7feac419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac9200640 1 -- 172.21.15.116:0/3744489037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac4107a50 msgr2=0x7feac419dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac9200640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac4107a50 0x7feac419dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac9200640 1 -- 172.21.15.116:0/3744489037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4108c40 msgr2=0x7feac419e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac9200640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4108c40 0x7feac419e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac9200640 1 -- 172.21.15.116:0/3744489037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac41a5900 con 0x7feac4109600 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac3400640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4108c40 0x7feac419e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac3e00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac4107a50 0x7feac419dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac9200640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4109600 0x7feac41a5260 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feabc02c640 tx=0x7feabc035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac0c00640 1 -- 172.21.15.116:0/3744489037 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc039040 con 0x7feac4109600 2024-03-04T07:51:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac41a5b90 con 0x7feac4109600 2024-03-04T07:51:18.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac0c00640 1 -- 172.21.15.116:0/3744489037 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feabc044070 con 0x7feac4109600 2024-03-04T07:51:18.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.343+0000 7feac0c00640 1 -- 172.21.15.116:0/3744489037 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc004460 con 0x7feac4109600 2024-03-04T07:51:18.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.344+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac41a6080 con 0x7feac4109600 2024-03-04T07:51:18.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.347+0000 7feac0c00640 1 -- 172.21.15.116:0/3744489037 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7feabc004600 con 0x7feac4109600 2024-03-04T07:51:18.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.347+0000 7feacdff3740 1 Processor -- start 2024-03-04T07:51:18.347 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.347+0000 7feacdff3740 1 -- start start 2024-03-04T07:51:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feacdff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feacdff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feacdff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6aa570 0x557dbf6aca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feacdff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dbf60a370 con 0x557dbf6a5400 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feacdff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dbf5b3550 con 0x557dbf6aa570 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feacdff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dbf606410 con 0x557dbf6a4a10 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6aa570 0x557dbf6aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6a9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47914/0 (socket says 172.21.15.116:47914) 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 -- 172.21.15.116:0/3494833700 learned_addr learned my addr 172.21.15.116:0/3494833700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3400640 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 -- 172.21.15.116:0/3494833700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 msgr2=0x557dbf6a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 -- 172.21.15.116:0/3494833700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6aa570 msgr2=0x557dbf6aca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6aa570 0x557dbf6aca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3e00640 1 -- 172.21.15.116:0/3494833700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557dbf5c73c0 con 0x557dbf6a5400 2024-03-04T07:51:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.348+0000 7feac3400640 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.349+0000 7feac0c00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7feaa80a1d70 0x7feaa80a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.349+0000 7feac3e00640 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6a9f00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feab4009c90 tx=0x7feab40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1268061f0d33f67e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.349+0000 7feac0c00640 1 -- 172.21.15.116:0/3744489037 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7feabc0f0580 con 0x7feac4109600 2024-03-04T07:51:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.349+0000 7feac3400640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7feaa80a1d70 0x7feaa80a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.349+0000 7feab9600640 1 -- 172.21.15.116:0/3494833700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4012450 con 0x557dbf6a5400 2024-03-04T07:51:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.349+0000 7feab9600640 1 -- 172.21.15.116:0/3494833700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab4012ad0 con 0x557dbf6a5400 2024-03-04T07:51:18.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.349+0000 7feab9600640 1 -- 172.21.15.116:0/3494833700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab400fbf0 con 0x557dbf6a5400 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 -- 172.21.15.116:0/3494833700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 msgr2=0x557dbf6a9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feac3400640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7feaa80a1d70 0x7feaa80a4230 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x557dbf6a5c60 tx=0x7feab0005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6a9f00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feab4009c90 tx=0x7feab40117b0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 -- 172.21.15.116:0/3494833700 shutdown_connections 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6aa570 0x557dbf6aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6a9f00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 --2- 172.21.15.116:0/3494833700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 -- 172.21.15.116:0/3494833700 >> 172.21.15.116:0/3494833700 conn(0x557dbf69ff80 msgr2=0x557dbf6a0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 -- 172.21.15.116:0/3494833700 shutdown_connections 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.350+0000 7feacdff3740 1 -- 172.21.15.116:0/3494833700 wait complete. 2024-03-04T07:51:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 -- start start 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 Processor -- start 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6aed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6aa570 0x557dbf6b6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dbf6ad520 con 0x557dbf6aa570 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dbf5c69b0 con 0x557dbf6a5400 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557dbf6ad6a0 con 0x557dbf6a4a10 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.351+0000 7feacdff3740 10 client.0 _ll_register_callbacks cb 0x557dbf602340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:53986/0 (socket says 172.21.15.116:53986) 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 -- 172.21.15.116:0/2948340791 learned_addr learned my addr 172.21.15.116:0/2948340791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac9200640 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6aa570 0x557dbf6b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3e00640 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 -- 172.21.15.116:0/2948340791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 msgr2=0x557dbf6aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6aed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 -- 172.21.15.116:0/2948340791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6aa570 msgr2=0x557dbf6b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6aa570 0x557dbf6b6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 -- 172.21.15.116:0/2948340791 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557dbf31f210 con 0x557dbf6a5400 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feac3400640 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6af2c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7feab0010a60 tx=0x7feab00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3df8e78ac86fd20c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feaae000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.352+0000 7feaae000640 1 -- 172.21.15.116:0/2948340791 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0004220 con 0x557dbf6a5400 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feaae000640 1 -- 172.21.15.116:0/2948340791 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7feab00043c0 con 0x557dbf6a5400 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feab4012ad0 con 0x557dbf6a5400 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feaae000640 1 -- 172.21.15.116:0/2948340791 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7feab00217b0 con 0x557dbf6a5400 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 10 client.28028 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557dbf3eb620 con 0x557dbf6a5400 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feaae000640 1 -- 172.21.15.116:0/2948340791 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab4012ad0 con 0x557dbf6a5400 2024-03-04T07:51:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feaae000640 1 -- 172.21.15.116:0/2948340791 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7feab0003c50 con 0x557dbf6a5400 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 10 client.28028 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 10 client.28028 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 20 client.28028 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf6babe0 0x557dbf6bd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 4 client.28028 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.353+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557dbf6178b0 con 0x557dbf6babe0 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.354+0000 7feac3e00640 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf6babe0 0x557dbf6bd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.354+0000 7feac3e00640 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf6babe0 0x557dbf6bd010 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=471982669b1b74a8 server_cookie=da957aed2d2081db in_seq=0 out_seq=0 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.354+0000 7feaae000640 10 client.28028 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feaae000640 1 -- 172.21.15.116:0/2948340791 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557dbf6178b0 con 0x557dbf6babe0 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feaae000640 10 client.28028 handle_command_reply: tid=0 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 client.28028 shutdown 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 msgr2=0x557dbf6af2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6af2c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7feab0010a60 tx=0x7feab00046c0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 shutdown_connections 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf6babe0 0x557dbf6bd010 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6aa570 0x557dbf6b6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feaae000640 0 client.28028 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6af2c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 --2- 172.21.15.116:0/2948340791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6a4a10 0x557dbf6aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 >> 172.21.15.116:0/2948340791 conn(0x557dbf69ff80 msgr2=0x557dbf69f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 shutdown_connections 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 1 -- 172.21.15.116:0/2948340791 wait complete. 2024-03-04T07:51:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.355+0000 7feacdff3740 20 client.28028 trim_cache size 0 max 16384 2024-03-04T07:51:18.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.370+0000 7feacdff3740 1 Processor -- start 2024-03-04T07:51:18.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feacdff3740 1 -- start start 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feacdff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a4a10 0x557dbf6a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feacdff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feacdff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6c1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feacdff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41acdd0 con 0x557dbf6a4a10 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feacdff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41af3b0 con 0x557dbf6a5400 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feacdff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41ad3d0 con 0x557dbf6aa570 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feac3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a4a10 0x557dbf6a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feac3e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a4a10 0x557dbf6a38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47920/0 (socket says 172.21.15.116:47920) 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feac3400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feac3e00640 1 -- 172.21.15.116:0/2126511956 learned_addr learned my addr 172.21.15.116:0/2126511956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.371+0000 7feac9200640 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6c1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac3400640 1 -- 172.21.15.116:0/2126511956 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 msgr2=0x557dbf6c1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac3400640 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6c1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac3400640 1 -- 172.21.15.116:0/2126511956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a4a10 msgr2=0x557dbf6a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac3400640 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a4a10 0x557dbf6a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac3400640 1 -- 172.21.15.116:0/2126511956 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557dbf6a5c90 con 0x557dbf6a5400 2024-03-04T07:51:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac9200640 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6c1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac3e00640 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a4a10 0x557dbf6a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feac3400640 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6b6250 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7feab001f410 tx=0x7feab000bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7be19d2c433d42 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feab8c00640 1 -- 172.21.15.116:0/2126511956 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0010e60 con 0x557dbf6a5400 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feab8c00640 1 -- 172.21.15.116:0/2126511956 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab0010870 con 0x557dbf6a5400 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.372+0000 7feab8c00640 1 -- 172.21.15.116:0/2126511956 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0004450 con 0x557dbf6a5400 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 -- 172.21.15.116:0/2126511956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 msgr2=0x557dbf6b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6b6250 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7feab001f410 tx=0x7feab000bfa0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 -- 172.21.15.116:0/2126511956 shutdown_connections 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6c1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a5400 0x557dbf6b6250 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 --2- 172.21.15.116:0/2126511956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a4a10 0x557dbf6a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 -- 172.21.15.116:0/2126511956 >> 172.21.15.116:0/2126511956 conn(0x557dbf69ff80 msgr2=0x557dbf6a31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 -- 172.21.15.116:0/2126511956 shutdown_connections 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 -- 172.21.15.116:0/2126511956 wait complete. 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 -- start start 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.373+0000 7feacdff3740 1 Processor -- start 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feacdff3740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a4a10 0x557dbf6b6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feacdff3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6b4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feacdff3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feacdff3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41ada30 con 0x557dbf6a5400 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feacdff3740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41aed50 con 0x557dbf6a4a10 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feacdff3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac41ae090 con 0x557dbf6aa570 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feacdff3740 10 client.0 _ll_register_callbacks cb 0x7feab00126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feac3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a4a10 0x557dbf6b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feac3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feac3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a4a10 0x557dbf6b6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:54006/0 (socket says 172.21.15.116:54006) 2024-03-04T07:51:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feac3e00640 1 -- 172.21.15.116:0/2858065575 learned_addr learned my addr 172.21.15.116:0/2858065575 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feac9200640 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feac3e00640 1 -- 172.21.15.116:0/2858065575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 msgr2=0x557dbf6b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.374+0000 7feac3e00640 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feac3e00640 1 -- 172.21.15.116:0/2858065575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 msgr2=0x557dbf6b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feac3e00640 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feac3e00640 1 -- 172.21.15.116:0/2858065575 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557dbf6b54f0 con 0x557dbf6a4a10 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feac3e00640 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a4a10 0x557dbf6b6170 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x557dbf6a5e70 tx=0x7feab4012450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed52cb15f7debbee server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feaae000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feaae000640 1 -- 172.21.15.116:0/2858065575 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4004020 con 0x557dbf6a4a10 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feaae000640 1 -- 172.21.15.116:0/2858065575 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7feab4020040 con 0x557dbf6a4a10 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feab0010870 con 0x557dbf6a4a10 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feaae000640 1 -- 172.21.15.116:0/2858065575 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7feab40041c0 con 0x557dbf6a4a10 2024-03-04T07:51:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feacdff3740 10 client.28034 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.375+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x557dbf717530 con 0x557dbf6a4a10 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feaae000640 1 -- 172.21.15.116:0/2858065575 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab0010870 con 0x557dbf6a4a10 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feaae000640 1 -- 172.21.15.116:0/2858065575 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7feab403a050 con 0x557dbf6a4a10 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feacdff3740 10 client.28034 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feacdff3740 10 client.28034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feacdff3740 20 client.28034 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feacdff3740 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf717bd0 0x557dbf719ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feacdff3740 4 client.28034 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557dbf611c00 con 0x557dbf717bd0 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.376+0000 7feac3400640 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf717bd0 0x557dbf719ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.377+0000 7feac3400640 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf717bd0 0x557dbf719ff0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89ea2f3233da9376 server_cookie=efd36a9f82b038be in_seq=0 out_seq=0 2024-03-04T07:51:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.377+0000 7feaae000640 10 client.28034 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feaae000640 1 -- 172.21.15.116:0/2858065575 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557dbf611c00 con 0x557dbf717bd0 2024-03-04T07:51:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feaae000640 10 client.28034 handle_command_reply: tid=0 2024-03-04T07:51:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 client.28034 shutdown 2024-03-04T07:51:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a4a10 msgr2=0x557dbf6b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a4a10 0x557dbf6b6170 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x557dbf6a5e70 tx=0x7feab4012450 comp rx=0 tx=0).stop 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 shutdown_connections 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x557dbf717bd0 0x557dbf719ff0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557dbf6aa570 0x557dbf6b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feaae000640 0 client.28034 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dbf6a5400 0x557dbf6b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 --2- 172.21.15.116:0/2858065575 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x557dbf6a4a10 0x557dbf6b6170 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 >> 172.21.15.116:0/2858065575 conn(0x557dbf69ff80 msgr2=0x557dbf6a3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 shutdown_connections 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.392+0000 7feacdff3740 1 -- 172.21.15.116:0/2858065575 wait complete. 2024-03-04T07:51:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feacdff3740 20 client.28034 trim_cache size 0 max 16384 2024-03-04T07:51:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:51:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:51:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:51:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7feaa80a1d70 msgr2=0x7feaa80a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7feaa80a1d70 0x7feaa80a4230 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x557dbf6a5c60 tx=0x7feab0005c90 comp rx=0 tx=0).stop 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4109600 msgr2=0x7feac41a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4109600 0x7feac41a5260 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feabc02c640 tx=0x7feabc035d10 comp rx=0 tx=0).stop 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 shutdown_connections 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feac4109600 0x7feac41a5260 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feac4108c40 0x7feac419e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7feaa80a1d70 0x7feaa80a4230 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 --2- 172.21.15.116:0/3744489037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac4107a50 0x7feac419dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.393+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 >> 172.21.15.116:0/3744489037 conn(0x7feac4076a10 msgr2=0x7feac410bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.394+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 shutdown_connections 2024-03-04T07:51:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:18.394+0000 7feac9c00640 1 -- 172.21.15.116:0/3744489037 wait complete. 2024-03-04T07:51:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:18 smithi116 ceph-mon[32150]: pgmap v717: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:51:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:18 smithi152 ceph-mon[37154]: pgmap v717: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:51:19.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:18 smithi133 ceph-mon[36505]: pgmap v717: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 9 op/s 2024-03-04T07:51:20.045 INFO:tasks.workunit.client.0.smithi116.stdout:499/600: open fd = 3 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:500/600: open fd = 3 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:501/600: open fd = 3 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:502/600: open fd = 3 2024-03-04T07:51:20.046 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:503/600: open fd = 3 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:504/600: open fd = 3 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:505/600: open fd = 3 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.047 INFO:tasks.workunit.client.0.smithi116.stdout:506/600: open fd = 3 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:507/600: open fd = 3 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:508/600: open fd = 3 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:509/600: open fd = 3 2024-03-04T07:51:20.048 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:510/600: open fd = 3 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:511/600: open fd = 3 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:512/600: open fd = 3 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.049 INFO:tasks.workunit.client.0.smithi116.stdout:513/600: open fd = 3 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:514/600: open fd = 3 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:515/600: open fd = 3 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:516/600: open fd = 3 2024-03-04T07:51:20.050 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:517/600: open fd = 3 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:518/600: open fd = 3 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:519/600: open fd = 3 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:520/600: open fd = 3 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:521/600: open fd = 3 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:522/600: open fd = 3 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.052 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:523/600: open fd = 3 2024-03-04T07:51:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:524/600: open fd = 3 2024-03-04T07:51:20.053 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.054 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.054 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.054 INFO:tasks.workunit.client.0.smithi116.stdout:525/600: open fd = 3 2024-03-04T07:51:20.054 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.054 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.054 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:526/600: open fd = 3 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:527/600: open fd = 3 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:528/600: open fd = 3 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:529/600: open fd = 3 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.055 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:530/600: open fd = 3 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:531/600: open fd = 3 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:532/600: open fd = 3 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.056 INFO:tasks.workunit.client.0.smithi116.stdout:533/600: open fd = 3 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:534/600: open fd = 3 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:535/600: open fd = 3 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.057 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:536/600: open fd = 3 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:537/600: open fd = 3 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.058 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:538/600: open fd = 3 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:539/600: open fd = 3 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.059 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:540/600: open fd = 3 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:541/600: open fd = 3 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.060 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:542/600: open fd = 3 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:543/600: open fd = 3 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:544/600: open fd = 3 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:545/600: open fd = 3 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:546/600: open fd = 3 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.062 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:547/600: open fd = 3 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:548/600: open fd = 3 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.063 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:549/600: open fd = 3 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:550/600: open fd = 3 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.064 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:551/600: open fd = 3 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:552/600: open fd = 3 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:553/600: open fd = 3 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:554/600: open fd = 3 2024-03-04T07:51:20.065 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:555/600: open fd = 3 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:556/600: open fd = 3 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:557/600: open fd = 3 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.066 INFO:tasks.workunit.client.0.smithi116.stdout:558/600: open fd = 3 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:559/600: open fd = 3 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:560/600: open fd = 3 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:51:20.067 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:51:21.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:20 smithi116 ceph-mon[32150]: pgmap v718: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-03-04T07:51:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:20 smithi152 ceph-mon[37154]: pgmap v718: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-03-04T07:51:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:20 smithi133 ceph-mon[36505]: pgmap v718: 97 pgs: 97 active+clean; 3.9 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-03-04T07:51:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:22 smithi116 ceph-mon[32150]: pgmap v719: 97 pgs: 97 active+clean; 4.0 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:22 smithi152 ceph-mon[37154]: pgmap v719: 97 pgs: 97 active+clean; 4.0 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:23.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:22 smithi133 ceph-mon[36505]: pgmap v719: 97 pgs: 97 active+clean; 4.0 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T07:51:25.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:24 smithi116 ceph-mon[32150]: pgmap v720: 97 pgs: 97 active+clean; 4.0 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:25.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:24 smithi152 ceph-mon[37154]: pgmap v720: 97 pgs: 97 active+clean; 4.0 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:25.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:24 smithi133 ceph-mon[36505]: pgmap v720: 97 pgs: 97 active+clean; 4.0 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-03-04T07:51:27.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:26 smithi116 ceph-mon[32150]: pgmap v721: 97 pgs: 97 active+clean; 4.1 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:51:27.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:26 smithi152 ceph-mon[37154]: pgmap v721: 97 pgs: 97 active+clean; 4.1 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:51:27.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:26 smithi133 ceph-mon[36505]: pgmap v721: 97 pgs: 97 active+clean; 4.1 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:51:29.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:28 smithi116 ceph-mon[32150]: pgmap v722: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:29.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:28 smithi152 ceph-mon[37154]: pgmap v722: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:29.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:28 smithi133 ceph-mon[36505]: pgmap v722: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:31.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:30 smithi116 ceph-mon[32150]: pgmap v723: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:31.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:30 smithi152 ceph-mon[37154]: pgmap v723: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:31.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:30 smithi133 ceph-mon[36505]: pgmap v723: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.536+0000 7f4f5ae00640 1 -- 172.21.15.116:0/3609541422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 msgr2=0x7f4f541092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.536+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/3609541422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f541092a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009900 tx=0x7f4f3c02bc40 comp rx=0 tx=0).stop 2024-03-04T07:51:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.537+0000 7f4f5ae00640 1 -- 172.21.15.116:0/3609541422 shutdown_connections 2024-03-04T07:51:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.537+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/3609541422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f541097e0 0x7f4f54110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.537+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/3609541422 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f541092a0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.537+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/3609541422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54107c30 0x7f4f54108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.537+0000 7f4f5ae00640 1 -- 172.21.15.116:0/3609541422 >> 172.21.15.116:0/3609541422 conn(0x7f4f54076be0 msgr2=0x7f4f54079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.537+0000 7f4f5ae00640 1 -- 172.21.15.116:0/3609541422 shutdown_connections 2024-03-04T07:51:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.537+0000 7f4f5ae00640 1 -- 172.21.15.116:0/3609541422 wait complete. 2024-03-04T07:51:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.538+0000 7f4f5ae00640 1 Processor -- start 2024-03-04T07:51:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.538+0000 7f4f5ae00640 1 -- start start 2024-03-04T07:51:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f5ae00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f54107c30 0x7f4f5419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f5ae00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f5419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f5ae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f541097e0 0x7f4f541a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f5ae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f54113900 con 0x7f4f541097e0 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f5ae00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f54113780 con 0x7f4f54108e20 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f5ae00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f54113a80 con 0x7f4f54107c30 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f58e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f541097e0 0x7f4f541a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f5419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f53e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f54107c30 0x7f4f5419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f5419e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55802/0 (socket says 172.21.15.116:55802) 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f53e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f54107c30 0x7f4f5419de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43300/0 (socket says 172.21.15.116:43300) 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f4b400640 1 -- 172.21.15.116:0/1220101185 learned_addr learned my addr 172.21.15.116:0/1220101185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f58e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f541097e0 0x7f4f541a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52456/0 (socket says 172.21.15.116:52456) 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f58e00640 1 -- 172.21.15.116:0/1220101185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f54107c30 msgr2=0x7f4f5419de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f58e00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f54107c30 0x7f4f5419de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f58e00640 1 -- 172.21.15.116:0/1220101185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 msgr2=0x7f4f5419e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f58e00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f5419e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f58e00640 1 -- 172.21.15.116:0/1220101185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f541a5a80 con 0x7f4f541097e0 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.539+0000 7f4f4b400640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f5419e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.540+0000 7f4f58e00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f541097e0 0x7f4f541a5410 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009fd0 tx=0x7f4f4c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.540+0000 7f4f51600640 1 -- 172.21.15.116:0/1220101185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00ce20 con 0x7f4f541097e0 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.540+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f541a5d70 con 0x7f4f541097e0 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.540+0000 7f4f51600640 1 -- 172.21.15.116:0/1220101185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f4c005020 con 0x7f4f541097e0 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.540+0000 7f4f51600640 1 -- 172.21.15.116:0/1220101185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c010720 con 0x7f4f541097e0 2024-03-04T07:51:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.540+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f541a6230 con 0x7f4f541097e0 2024-03-04T07:51:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.543+0000 7f4f5db6d740 1 Processor -- start 2024-03-04T07:51:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f51600640 1 -- 172.21.15.116:0/1220101185 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4f4c005540 con 0x7f4f541097e0 2024-03-04T07:51:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f5db6d740 1 -- start start 2024-03-04T07:51:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f13c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 0x555a69f18d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f193b0 0x555a69f1b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a69e79370 con 0x555a69f193b0 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a69e22550 con 0x555a69f13850 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a69e75410 con 0x555a69f14240 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.544+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f13c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f13c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55814/0 (socket says 172.21.15.116:55814) 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 -- 172.21.15.116:0/3563347270 learned_addr learned my addr 172.21.15.116:0/3563347270 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f193b0 0x555a69f1b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f53e00640 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 0x555a69f18d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 -- 172.21.15.116:0/3563347270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 msgr2=0x555a69f18d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 0x555a69f18d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 -- 172.21.15.116:0/3563347270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f193b0 msgr2=0x555a69f1b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f193b0 0x555a69f1b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 -- 172.21.15.116:0/3563347270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a69e363c0 con 0x555a69f13850 2024-03-04T07:51:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f13c70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c02c250 tx=0x7f4f3c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28f9ad9f9c057bef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f51600640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f380a1e20 0x7f4f380a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.545+0000 7f4f49600640 1 -- 172.21.15.116:0/3563347270 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c02cd40 con 0x555a69f13850 2024-03-04T07:51:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f51600640 1 -- 172.21.15.116:0/1220101185 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4f4c0ce440 con 0x7f4f541097e0 2024-03-04T07:51:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f49600640 1 -- 172.21.15.116:0/3563347270 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f3c004ba0 con 0x555a69f13850 2024-03-04T07:51:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f49600640 1 -- 172.21.15.116:0/3563347270 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c0035f0 con 0x555a69f13850 2024-03-04T07:51:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f53e00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f380a1e20 0x7f4f380a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f53e00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f380a1e20 0x7f4f380a42e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x555a69f14640 tx=0x7f4f44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3563347270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 msgr2=0x555a69f13c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f13c70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c02c250 tx=0x7f4f3c009260 comp rx=0 tx=0).stop 2024-03-04T07:51:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3563347270 shutdown_connections 2024-03-04T07:51:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f193b0 0x555a69f1b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 0x555a69f18d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3563347270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f13c70 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.546+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3563347270 >> 172.21.15.116:0/3563347270 conn(0x555a69f0edc0 msgr2=0x555a69f0f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.547+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3563347270 shutdown_connections 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.547+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3563347270 wait complete. 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.547+0000 7f4f5db6d740 1 -- start start 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.547+0000 7f4f5db6d740 1 Processor -- start 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f1db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f1e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f193b0 0x555a69f250d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a69f1c300 con 0x555a69f13850 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a69e359b0 con 0x555a69f14240 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a69f1c480 con 0x555a69f193b0 2024-03-04T07:51:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f5db6d740 10 client.0 _ll_register_callbacks cb 0x555a69e71340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f1e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f1e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55830/0 (socket says 172.21.15.116:55830) 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 -- 172.21.15.116:0/1566553929 learned_addr learned my addr 172.21.15.116:0/1566553929 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f53e00640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f1db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f58e00640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f193b0 0x555a69f250d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 -- 172.21.15.116:0/1566553929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f193b0 msgr2=0x555a69f250d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f193b0 0x555a69f250d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 -- 172.21.15.116:0/1566553929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 msgr2=0x555a69f1db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f1db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f4b400640 1 -- 172.21.15.116:0/1566553929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a69b8e210 con 0x555a69f14240 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.548+0000 7f4f58e00640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f193b0 0x555a69f250d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f4b400640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f1e090 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c006010 tx=0x7f4f3c004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f66e5bf134f6f739 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f42000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f42000640 1 -- 172.21.15.116:0/1566553929 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c050030 con 0x555a69f14240 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f3c004ba0 con 0x555a69f14240 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f42000640 1 -- 172.21.15.116:0/1566553929 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4f3c004f10 con 0x555a69f14240 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f42000640 1 -- 172.21.15.116:0/1566553929 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f3c05c050 con 0x555a69f14240 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f5db6d740 10 client.28046 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555a69c5a620 con 0x555a69f14240 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.549+0000 7f4f42000640 1 -- 172.21.15.116:0/1566553929 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f3c004ba0 con 0x555a69f14240 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f42000640 1 -- 172.21.15.116:0/1566553929 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4f3c003b80 con 0x555a69f14240 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f5db6d740 10 client.28046 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f5db6d740 10 client.28046 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f5db6d740 20 client.28046 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f295c0 0x555a69f2b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f5db6d740 4 client.28046 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a69e868b0 con 0x555a69f295c0 2024-03-04T07:51:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f53e00640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f295c0 0x555a69f2b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f42000640 10 client.28046 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:31.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.550+0000 7f4f53e00640 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f295c0 0x555a69f2b9b0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c729fa2e8d2b79b9 server_cookie=2705f5f65496a66f in_seq=0 out_seq=0 2024-03-04T07:51:31.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f42000640 1 -- 172.21.15.116:0/1566553929 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555a69e868b0 con 0x555a69f295c0 2024-03-04T07:51:31.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f42000640 10 client.28046 handle_command_reply: tid=0 2024-03-04T07:51:31.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f5db6d740 1 client.28046 shutdown 2024-03-04T07:51:31.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 msgr2=0x555a69f1e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f1e090 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c006010 tx=0x7f4f3c004530 comp rx=0 tx=0).stop 2024-03-04T07:51:31.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 shutdown_connections 2024-03-04T07:51:31.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f295c0 0x555a69f2b9b0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.551+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f193b0 0x555a69f250d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.552+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f1e090 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.552+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/1566553929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f1db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.552+0000 7f4f42000640 0 client.28046 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:31.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.552+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 >> 172.21.15.116:0/1566553929 conn(0x555a69f0edc0 msgr2=0x555a69f0dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:31.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.552+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 shutdown_connections 2024-03-04T07:51:31.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.552+0000 7f4f5db6d740 1 -- 172.21.15.116:0/1566553929 wait complete. 2024-03-04T07:51:31.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.552+0000 7f4f5db6d740 20 client.28046 trim_cache size 0 max 16384 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 Processor -- start 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 -- start start 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f0edc0 0x555a69f12730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f25090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f30190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541acf30 con 0x555a69f13850 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541af510 con 0x555a69f14240 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.567+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541ad530 con 0x555a69f0edc0 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f0edc0 0x555a69f12730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f0edc0 0x555a69f12730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:43330/0 (socket says 172.21.15.116:43330) 2024-03-04T07:51:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 -- 172.21.15.116:0/3081791546 learned_addr learned my addr 172.21.15.116:0/3081791546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f58e00640 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f30190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f25090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 -- 172.21.15.116:0/3081791546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 msgr2=0x555a69f30190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f30190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 -- 172.21.15.116:0/3081791546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 msgr2=0x555a69f25090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f25090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 -- 172.21.15.116:0/3081791546 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a69f13c30 con 0x555a69f0edc0 2024-03-04T07:51:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f25090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f53e00640 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f0edc0 0x555a69f12730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x555a69f0fd30 tx=0x7f4f4401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c32f4a8e1ed58fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.568+0000 7f4f48c00640 1 -- 172.21.15.116:0/3081791546 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44003a40 con 0x555a69f0edc0 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f48c00640 1 -- 172.21.15.116:0/3081791546 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f44030070 con 0x555a69f0edc0 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f48c00640 1 -- 172.21.15.116:0/3081791546 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4401f640 con 0x555a69f0edc0 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3081791546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f0edc0 msgr2=0x555a69f12730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f0edc0 0x555a69f12730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x555a69f0fd30 tx=0x7f4f4401a000 comp rx=0 tx=0).stop 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3081791546 shutdown_connections 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f14240 0x555a69f30190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f13850 0x555a69f25090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3081791546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f0edc0 0x555a69f12730 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3081791546 >> 172.21.15.116:0/3081791546 conn(0x555a69f18ef0 msgr2=0x555a69f12020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3081791546 shutdown_connections 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.569+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3081791546 wait complete. 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 -- start start 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 Processor -- start 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f0edc0 0x555a69f22d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 0x555a69f242c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541adb90 con 0x555a69f0edc0 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541aeeb0 con 0x555a69f13850 2024-03-04T07:51:31.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541ae1f0 con 0x555a69f14240 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.570+0000 7f4f5db6d740 10 client.0 _ll_register_callbacks cb 0x7f4f3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f53e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f0edc0 0x555a69f22d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f232b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:55850/0 (socket says 172.21.15.116:55850) 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 -- 172.21.15.116:0/3531099450 learned_addr learned my addr 172.21.15.116:0/3531099450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 -- 172.21.15.116:0/3531099450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 msgr2=0x555a69f242c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 0x555a69f242c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 -- 172.21.15.116:0/3531099450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f0edc0 msgr2=0x555a69f22d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f0edc0 0x555a69f22d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 -- 172.21.15.116:0/3531099450 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a69f248b0 con 0x555a69f13850 2024-03-04T07:51:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f4b400640 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f232b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009850 tx=0x7f4f3c005140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2247857681f8de37 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f42000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f42000640 1 -- 172.21.15.116:0/3531099450 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c065060 con 0x555a69f13850 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f44030070 con 0x555a69f13850 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.571+0000 7f4f42000640 1 -- 172.21.15.116:0/3531099450 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4f3c0348a0 con 0x555a69f13850 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f42000640 1 -- 172.21.15.116:0/3531099450 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f3c050040 con 0x555a69f13850 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 10 client.28052 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x555a69f85360 con 0x555a69f13850 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f42000640 1 -- 172.21.15.116:0/3531099450 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f44030070 con 0x555a69f13850 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f42000640 1 -- 172.21.15.116:0/3531099450 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f4f3c06a070 con 0x555a69f13850 2024-03-04T07:51:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 10 client.28052 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 10 client.28052 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 20 client.28052 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f193b0 0x555a69f87c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 4 client.28052 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555a69f1d940 con 0x555a69f193b0 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.572+0000 7f4f53e00640 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f193b0 0x555a69f87c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.573+0000 7f4f53e00640 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f193b0 0x555a69f87c60 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0a23cf7341b5a7b server_cookie=e65cea0922a20008 in_seq=0 out_seq=0 2024-03-04T07:51:31.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.573+0000 7f4f42000640 10 client.28052 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f42000640 1 -- 172.21.15.116:0/3531099450 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555a69f1d940 con 0x555a69f193b0 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f42000640 10 client.28052 handle_command_reply: tid=0 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 client.28052 shutdown 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 msgr2=0x555a69f232b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f232b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009850 tx=0x7f4f3c005140 comp rx=0 tx=0).stop 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 shutdown_connections 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x555a69f193b0 0x555a69f87c60 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555a69f14240 0x555a69f242c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x555a69f13850 0x555a69f232b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 --2- 172.21.15.116:0/3531099450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555a69f0edc0 0x555a69f22d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f42000640 0 client.28052 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 >> 172.21.15.116:0/3531099450 conn(0x555a69f18ef0 msgr2=0x555a69f128b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.589+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 shutdown_connections 2024-03-04T07:51:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.590+0000 7f4f5db6d740 1 -- 172.21.15.116:0/3531099450 wait complete. 2024-03-04T07:51:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.590+0000 7f4f5db6d740 20 client.28052 trim_cache size 0 max 16384 2024-03-04T07:51:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:51:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:51:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:51:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f380a1e20 msgr2=0x7f4f380a42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f380a1e20 0x7f4f380a42e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x555a69f14640 tx=0x7f4f44005f90 comp rx=0 tx=0).stop 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f541097e0 msgr2=0x7f4f541a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f541097e0 0x7f4f541a5410 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009fd0 tx=0x7f4f4c00ef10 comp rx=0 tx=0).stop 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 shutdown_connections 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f541097e0 0x7f4f541a5410 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4f54108e20 0x7f4f5419e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f4f380a1e20 0x7f4f380a42e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 --2- 172.21.15.116:0/1220101185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f54107c30 0x7f4f5419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:31.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 >> 172.21.15.116:0/1220101185 conn(0x7f4f54076be0 msgr2=0x7f4f5410c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:31.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 shutdown_connections 2024-03-04T07:51:31.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:31.591+0000 7f4f5ae00640 1 -- 172.21.15.116:0/1220101185 wait complete. 2024-03-04T07:51:32.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:32 smithi133 ceph-mon[36505]: pgmap v724: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:51:32.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:32 smithi116 ceph-mon[32150]: pgmap v724: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:51:32.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:32 smithi152 ceph-mon[37154]: pgmap v724: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:51:35.212 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:34 smithi116 ceph-mon[32150]: pgmap v725: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:51:35.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:34 smithi152 ceph-mon[37154]: pgmap v725: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:51:35.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:34 smithi133 ceph-mon[36505]: pgmap v725: 97 pgs: 97 active+clean; 4.1 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-03-04T07:51:37.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:36 smithi116 ceph-mon[32150]: pgmap v726: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:51:37.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:36 smithi152 ceph-mon[37154]: pgmap v726: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:51:37.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:36 smithi133 ceph-mon[36505]: pgmap v726: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 21 KiB/s wr, 12 op/s 2024-03-04T07:51:37.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:51:37.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.907+0000 7f3a4da00640 1 -- 172.21.15.116:0/1761748797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481015e0 msgr2=0x7f3a48101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.907+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1761748797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481015e0 0x7f3a48101a00 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009900 tx=0x7f3a3802bc30 comp rx=0 tx=0).stop 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.907+0000 7f3a4da00640 1 -- 172.21.15.116:0/1761748797 shutdown_connections 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.907+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1761748797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a48103190 0x7f3a48109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.907+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1761748797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481027d0 0x7f3a48102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.907+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1761748797 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481015e0 0x7f3a48101a00 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.907+0000 7f3a4da00640 1 -- 172.21.15.116:0/1761748797 >> 172.21.15.116:0/1761748797 conn(0x7f3a480fcb70 msgr2=0x7f3a480fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.908+0000 7f3a4da00640 1 -- 172.21.15.116:0/1761748797 shutdown_connections 2024-03-04T07:51:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.908+0000 7f3a4da00640 1 -- 172.21.15.116:0/1761748797 wait complete. 2024-03-04T07:51:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 Processor -- start 2024-03-04T07:51:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 -- start start 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481015e0 0x7f3a48199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481027d0 0x7f3a4819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a48103190 0x7f3a481a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4810d2b0 con 0x7f3a48103190 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4810d130 con 0x7f3a481027d0 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a4da00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4810d430 con 0x7f3a481015e0 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.909+0000 7f3a47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a48103190 0x7f3a481a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481015e0 0x7f3a48199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481015e0 0x7f3a48199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:38492/0 (socket says 172.21.15.116:38492) 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a48103190 0x7f3a481a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50140/0 (socket says 172.21.15.116:50140) 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 -- 172.21.15.116:0/1252947818 learned_addr learned my addr 172.21.15.116:0/1252947818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:37.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481027d0 0x7f3a4819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 -- 172.21.15.116:0/1252947818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481027d0 msgr2=0x7f3a4819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481027d0 0x7f3a4819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 -- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a48103190 msgr2=0x7f3a481a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a48103190 0x7f3a481a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 -- 172.21.15.116:0/1252947818 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a481a17e0 con 0x7f3a481015e0 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.910+0000 7f3a46a00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481015e0 0x7f3a48199bc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3a38000c00 tx=0x7f3a3802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.911+0000 7f3a3f400640 1 -- 172.21.15.116:0/1252947818 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38043020 con 0x7f3a481015e0 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.911+0000 7f3a3f400640 1 -- 172.21.15.116:0/1252947818 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a380054e0 con 0x7f3a481015e0 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.911+0000 7f3a3f400640 1 -- 172.21.15.116:0/1252947818 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38003ce0 con 0x7f3a481015e0 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.911+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a481a1a70 con 0x7f3a481015e0 2024-03-04T07:51:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.911+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a481a1f40 con 0x7f3a481015e0 2024-03-04T07:51:37.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.914+0000 7f3a50793740 1 Processor -- start 2024-03-04T07:51:37.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.914+0000 7f3a50793740 1 -- start start 2024-03-04T07:51:37.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.914+0000 7f3a3f400640 1 -- 172.21.15.116:0/1252947818 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3a38046020 con 0x7f3a481015e0 2024-03-04T07:51:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a50793740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2bb50 0x55f010e2bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a50793740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2c540 0x55f010e31040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a50793740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e33b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a50793740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f010d59090 con 0x55f010e2bb50 2024-03-04T07:51:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a50793740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f010d90d30 con 0x55f010e2c540 2024-03-04T07:51:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a50793740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f010d39a30 con 0x55f010e316b0 2024-03-04T07:51:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2bb50 0x55f010e2bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a46a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2c540 0x55f010e31040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2bb50 0x55f010e2bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50154/0 (socket says 172.21.15.116:50154) 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 -- 172.21.15.116:0/4214865164 learned_addr learned my addr 172.21.15.116:0/4214865164 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e33b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 -- 172.21.15.116:0/4214865164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 msgr2=0x55f010e33b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e33b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 -- 172.21.15.116:0/4214865164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2c540 msgr2=0x55f010e31040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2c540 0x55f010e31040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a47400640 1 -- 172.21.15.116:0/4214865164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f010b6f7d0 con 0x55f010e2bb50 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.915+0000 7f3a46a00640 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2c540 0x55f010e31040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.916+0000 7f3a47400640 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2bb50 0x55f010e2bf70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3a40010370 tx=0x7f3a40010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987076b61d43da84 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.916+0000 7f3a3f400640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a2c0a1d80 0x7f3a2c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.916+0000 7f3a37e00640 1 -- 172.21.15.116:0/4214865164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40016070 con 0x55f010e2bb50 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.916+0000 7f3a37e00640 1 -- 172.21.15.116:0/4214865164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a40004d20 con 0x55f010e2bb50 2024-03-04T07:51:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.916+0000 7f3a37e00640 1 -- 172.21.15.116:0/4214865164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a400053e0 con 0x55f010e2bb50 2024-03-04T07:51:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.916+0000 7f3a3f400640 1 -- 172.21.15.116:0/1252947818 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3a380efe30 con 0x7f3a481015e0 2024-03-04T07:51:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.916+0000 7f3a46a00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a2c0a1d80 0x7f3a2c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 -- 172.21.15.116:0/4214865164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2bb50 msgr2=0x55f010e2bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a46a00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a2c0a1d80 0x7f3a2c0a4240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3a380bcaf0 tx=0x7f3a380f38b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:37.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2bb50 0x55f010e2bf70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3a40010370 tx=0x7f3a40010830 comp rx=0 tx=0).stop 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 -- 172.21.15.116:0/4214865164 shutdown_connections 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e33b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2c540 0x55f010e31040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 --2- 172.21.15.116:0/4214865164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2bb50 0x55f010e2bf70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 -- 172.21.15.116:0/4214865164 >> 172.21.15.116:0/4214865164 conn(0x55f010e270c0 msgr2=0x55f010e274c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 -- 172.21.15.116:0/4214865164 shutdown_connections 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.917+0000 7f3a50793740 1 -- 172.21.15.116:0/4214865164 wait complete. 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 -- start start 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 Processor -- start 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2bb50 0x55f010e35f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e36480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e316b0 0x55f010e3d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f010d2f0c0 con 0x55f010e2c540 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f010d8cdd0 con 0x55f010e316b0 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f010e34660 con 0x55f010e2bb50 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a50793740 10 client.0 _ll_register_callbacks cb 0x55f010d4c860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a3fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e36480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a3fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e36480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50164/0 (socket says 172.21.15.116:50164) 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.918+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4264957507 learned_addr learned my addr 172.21.15.116:0/4264957507 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a47400640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e316b0 0x55f010e3d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a47400640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2bb50 0x55f010e35f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4264957507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2bb50 msgr2=0x55f010e35f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2bb50 0x55f010e35f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e316b0 msgr2=0x55f010e3d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e316b0 0x55f010e3d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4264957507 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f010aa53f0 con 0x55f010e2c540 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a47400640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2bb50 0x55f010e35f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e36480 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3a300126e0 tx=0x7f3a30012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584906106337936d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a34c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a34c00640 1 -- 172.21.15.116:0/4264957507 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000f8c0 con 0x55f010e2c540 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a34c00640 1 -- 172.21.15.116:0/4264957507 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3a3000fa60 con 0x55f010e2c540 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.919+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a40004d20 con 0x55f010e2c540 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a34c00640 1 -- 172.21.15.116:0/4264957507 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a3000fc60 con 0x55f010e2c540 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 10 client.17922 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f010a650f0 con 0x55f010e2c540 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a34c00640 1 -- 172.21.15.116:0/4264957507 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a40004d20 con 0x55f010e2c540 2024-03-04T07:51:37.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a34c00640 1 -- 172.21.15.116:0/4264957507 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3a3001c420 con 0x55f010e2c540 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 10 client.17922 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 10 client.17922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 20 client.17922 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010e41ad0 0x55f010e43f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 4 client.17922 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.920+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f010d9e270 con 0x55f010e41ad0 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.921+0000 7f3a47400640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010e41ad0 0x55f010e43f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.921+0000 7f3a47400640 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010e41ad0 0x55f010e43f10 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8e2dce568a10bdc server_cookie=6071e3bda8777873 in_seq=0 out_seq=0 2024-03-04T07:51:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.921+0000 7f3a34c00640 10 client.17922 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a34c00640 1 -- 172.21.15.116:0/4264957507 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f010d9e270 con 0x55f010e41ad0 2024-03-04T07:51:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a34c00640 10 client.17922 handle_command_reply: tid=0 2024-03-04T07:51:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 client.17922 shutdown 2024-03-04T07:51:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 msgr2=0x55f010e36480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e36480 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3a300126e0 tx=0x7f3a30012ba0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 shutdown_connections 2024-03-04T07:51:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010e41ad0 0x55f010e43f10 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e316b0 0x55f010e3d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e36480 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a34c00640 0 client.17922 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 --2- 172.21.15.116:0/4264957507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2bb50 0x55f010e35f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 >> 172.21.15.116:0/4264957507 conn(0x55f010e270c0 msgr2=0x55f010e26360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 shutdown_connections 2024-03-04T07:51:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 1 -- 172.21.15.116:0/4264957507 wait complete. 2024-03-04T07:51:37.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.922+0000 7f3a50793740 20 client.17922 trim_cache size 0 max 16384 2024-03-04T07:51:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 Processor -- start 2024-03-04T07:51:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 -- start start 2024-03-04T07:51:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e35560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2c540 0x55f010e3d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e316b0 0x55f010e25ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a481a8d80 con 0x55f010e316b0 2024-03-04T07:51:37.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a481ab360 con 0x55f010e2bb50 2024-03-04T07:51:37.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a50793740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a481a9380 con 0x55f010e2c540 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a47400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e35560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.940+0000 7f3a47400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e35560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41706/0 (socket says 172.21.15.116:41706) 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 -- 172.21.15.116:0/290287243 learned_addr learned my addr 172.21.15.116:0/290287243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a3fe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2c540 0x55f010e3d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e316b0 0x55f010e25ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 -- 172.21.15.116:0/290287243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2c540 msgr2=0x55f010e3d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2c540 0x55f010e3d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 -- 172.21.15.116:0/290287243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e316b0 msgr2=0x55f010e25ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e316b0 0x55f010e25ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 -- 172.21.15.116:0/290287243 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f010e2cdd0 con 0x55f010e2bb50 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a47400640 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e35560 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3a40011010 tx=0x7f3a40010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5dcce970f8dc462 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:37.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a37400640 1 -- 172.21.15.116:0/290287243 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40016040 con 0x55f010e2bb50 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.941+0000 7f3a37400640 1 -- 172.21.15.116:0/290287243 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a40003e50 con 0x55f010e2bb50 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a37400640 1 -- 172.21.15.116:0/290287243 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40004310 con 0x55f010e2bb50 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 -- 172.21.15.116:0/290287243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 msgr2=0x55f010e35560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e35560 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3a40011010 tx=0x7f3a40010830 comp rx=0 tx=0).stop 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 -- 172.21.15.116:0/290287243 shutdown_connections 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e316b0 0x55f010e25ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e2c540 0x55f010e3d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 --2- 172.21.15.116:0/290287243 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e35560 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 -- 172.21.15.116:0/290287243 >> 172.21.15.116:0/290287243 conn(0x55f010e270c0 msgr2=0x55f010e2da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:37.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 -- 172.21.15.116:0/290287243 shutdown_connections 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.942+0000 7f3a50793740 1 -- 172.21.15.116:0/290287243 wait complete. 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 -- start start 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 Processor -- start 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e3ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e3d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e3b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a481a99e0 con 0x55f010e2c540 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a481aad00 con 0x55f010e2bb50 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a481aa040 con 0x55f010e316b0 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a50793740 10 client.0 _ll_register_callbacks cb 0x7f3a30009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.943+0000 7f3a3fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e3d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e3d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50196/0 (socket says 172.21.15.116:50196) 2024-03-04T07:51:37.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4194580423 learned_addr learned my addr 172.21.15.116:0/4194580423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a47400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e3ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a47400640 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e3b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4194580423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 msgr2=0x55f010e3b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e3b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 msgr2=0x55f010e3ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e3ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 -- 172.21.15.116:0/4194580423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f010e3bd30 con 0x55f010e2c540 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a47400640 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e3ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a3fe00640 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e3d3f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000c00 tx=0x7f3a3001c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c247a51cd324e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a34c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a34c00640 1 -- 172.21.15.116:0/4194580423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000f9a0 con 0x55f010e2c540 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a34c00640 1 -- 172.21.15.116:0/4194580423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3a3001ca80 con 0x55f010e2c540 2024-03-04T07:51:37.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.944+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a40003e50 con 0x55f010e2c540 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a34c00640 1 -- 172.21.15.116:0/4194580423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a30038050 con 0x55f010e2c540 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 10 client.17928 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55f010ea6750 con 0x55f010e2c540 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a34c00640 1 -- 172.21.15.116:0/4194580423 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a40003e50 con 0x55f010e2c540 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a34c00640 1 -- 172.21.15.116:0/4194580423 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3a30003710 con 0x55f010e2c540 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 10 client.17928 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 10 client.17928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 20 client.17928 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010ea70d0 0x55f010ea94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 4 client.17928 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.945+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f010e35d30 con 0x55f010ea70d0 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a47400640 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010ea70d0 0x55f010ea94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:37.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a47400640 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010ea70d0 0x55f010ea94f0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=607d8c0822f43d88 server_cookie=1caedd9fc5f54f53 in_seq=0 out_seq=0 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a34c00640 10 client.17928 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a34c00640 1 -- 172.21.15.116:0/4194580423 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f010e35d30 con 0x55f010ea70d0 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a34c00640 10 client.17928 handle_command_reply: tid=0 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 client.17928 shutdown 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 msgr2=0x55f010e3d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e3d3f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3a30000c00 tx=0x7f3a3001c6d0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 shutdown_connections 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55f010ea70d0 0x55f010ea94f0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f010e316b0 0x55f010e3b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f010e2c540 0x55f010e3d3f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 --2- 172.21.15.116:0/4194580423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55f010e2bb50 0x55f010e3ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a34c00640 0 client.17928 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 >> 172.21.15.116:0/4194580423 conn(0x55f010e270c0 msgr2=0x55f010e2abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:37.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.946+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 shutdown_connections 2024-03-04T07:51:37.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a50793740 1 -- 172.21.15.116:0/4194580423 wait complete. 2024-03-04T07:51:37.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a50793740 20 client.17928 trim_cache size 0 max 16384 2024-03-04T07:51:37.954 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:51:37.954 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:51:37.954 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:51:37.954 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:51:37.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a2c0a1d80 msgr2=0x7f3a2c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a2c0a1d80 0x7f3a2c0a4240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3a380bcaf0 tx=0x7f3a380f38b0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481015e0 msgr2=0x7f3a48199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:37.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481015e0 0x7f3a48199bc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3a38000c00 tx=0x7f3a3802c6c0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 shutdown_connections 2024-03-04T07:51:37.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a48103190 0x7f3a481a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a481027d0 0x7f3a4819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f3a2c0a1d80 0x7f3a2c0a4240 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 --2- 172.21.15.116:0/1252947818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a481015e0 0x7f3a48199bc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:37.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 >> 172.21.15.116:0/1252947818 conn(0x7f3a480fcb70 msgr2=0x7f3a48073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:37.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.947+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 shutdown_connections 2024-03-04T07:51:37.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:37.948+0000 7f3a4da00640 1 -- 172.21.15.116:0/1252947818 wait complete. 2024-03-04T07:51:37.965 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:51:37.966 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.175+0000 7fd3ed600640 1 -- 172.21.15.116:0/1611030038 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 msgr2=0x7fd3e8073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.175+0000 7fd3ed600640 1 --2- 172.21.15.116:0/1611030038 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 0x7fd3e8073a60 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009930 tx=0x7fd3d802bc30 comp rx=0 tx=0).stop 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.176+0000 7fd3ed600640 1 -- 172.21.15.116:0/1611030038 shutdown_connections 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.176+0000 7fd3ed600640 1 --2- 172.21.15.116:0/1611030038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e810d9e0 0x7fd3e810fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.176+0000 7fd3ed600640 1 --2- 172.21.15.116:0/1611030038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3e8074030 0x7fd3e80744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.176+0000 7fd3ed600640 1 --2- 172.21.15.116:0/1611030038 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 0x7fd3e8073a60 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.176+0000 7fd3ed600640 1 -- 172.21.15.116:0/1611030038 >> 172.21.15.116:0/1611030038 conn(0x7fd3e80fc960 msgr2=0x7fd3e80fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.176+0000 7fd3ed600640 1 -- 172.21.15.116:0/1611030038 shutdown_connections 2024-03-04T07:51:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.176+0000 7fd3ed600640 1 -- 172.21.15.116:0/1611030038 wait complete. 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 Processor -- start 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 -- start start 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e8074030 0x7fd3e819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 0x7fd3e819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3e810d9e0 0x7fd3e81a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81136d0 con 0x7fd3e8074030 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e8113550 con 0x7fd3e8075610 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3ed600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e8113850 con 0x7fd3e810d9e0 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3e7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e8074030 0x7fd3e819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3e7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e8074030 0x7fd3e819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50212/0 (socket says 172.21.15.116:50212) 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3e7e00640 1 -- 172.21.15.116:0/227627759 learned_addr learned my addr 172.21.15.116:0/227627759 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.177+0000 7fd3e7400640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 0x7fd3e819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.178+0000 7fd3ecc00640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3e810d9e0 0x7fd3e81a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.178+0000 7fd3e7e00640 1 -- 172.21.15.116:0/227627759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3e810d9e0 msgr2=0x7fd3e81a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.178+0000 7fd3e7e00640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3e810d9e0 0x7fd3e81a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.178+0000 7fd3e7e00640 1 -- 172.21.15.116:0/227627759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 msgr2=0x7fd3e819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.178+0000 7fd3e7e00640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 0x7fd3e819e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.178+0000 7fd3e7e00640 1 -- 172.21.15.116:0/227627759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e81a5980 con 0x7fd3e8074030 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.178+0000 7fd3e7e00640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e8074030 0x7fd3e819dd60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8000c00 tx=0x7fd3d802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.179+0000 7fd3e4c00640 1 -- 172.21.15.116:0/227627759 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8043020 con 0x7fd3e8074030 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.179+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e81a5c10 con 0x7fd3e8074030 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.179+0000 7fd3e4c00640 1 -- 172.21.15.116:0/227627759 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3d80054e0 con 0x7fd3e8074030 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.179+0000 7fd3e4c00640 1 -- 172.21.15.116:0/227627759 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8003c40 con 0x7fd3e8074030 2024-03-04T07:51:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.179+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e81a61b0 con 0x7fd3e8074030 2024-03-04T07:51:38.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.180+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e8103bb0 con 0x7fd3e8074030 2024-03-04T07:51:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.182+0000 7fd3e4c00640 1 -- 172.21.15.116:0/227627759 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd3d8046020 con 0x7fd3e8074030 2024-03-04T07:51:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.183+0000 7fd3e4c00640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd3cc0a1d80 0x7fd3cc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.183+0000 7fd3e4c00640 1 -- 172.21.15.116:0/227627759 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd3d80efd70 con 0x7fd3e8074030 2024-03-04T07:51:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.184+0000 7fd3e7400640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd3cc0a1d80 0x7fd3cc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.184+0000 7fd3e7400640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd3cc0a1d80 0x7fd3cc0a4240 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd3e819f270 tx=0x7fd3d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.188+0000 7fd3e4c00640 1 -- 172.21.15.116:0/227627759 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd3d804d020 con 0x7fd3e8074030 2024-03-04T07:51:38.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.426+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e802d210 con 0x7fd3e8074030 2024-03-04T07:51:38.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.427+0000 7fd3e4c00640 1 -- 172.21.15.116:0/227627759 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fd3d80bd160 con 0x7fd3e8074030 2024-03-04T07:51:38.429 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:51:38.429 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:51:38.429 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:51:38.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd3cc0a1d80 msgr2=0x7fd3cc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd3cc0a1d80 0x7fd3cc0a4240 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd3e819f270 tx=0x7fd3d4005f90 comp rx=0 tx=0).stop 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e8074030 msgr2=0x7fd3e819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e8074030 0x7fd3e819dd60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8000c00 tx=0x7fd3d802c650 comp rx=0 tx=0).stop 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 shutdown_connections 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd3cc0a1d80 0x7fd3cc0a4240 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3e810d9e0 0x7fd3e81a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3e8075610 0x7fd3e819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 --2- 172.21.15.116:0/227627759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3e8074030 0x7fd3e819dd60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 >> 172.21.15.116:0/227627759 conn(0x7fd3e80fc960 msgr2=0x7fd3e8100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 shutdown_connections 2024-03-04T07:51:38.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.430+0000 7fd3ed600640 1 -- 172.21.15.116:0/227627759 wait complete. 2024-03-04T07:51:38.451 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:51:38.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:51:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.657+0000 7f40a2e00640 1 -- 172.21.15.116:0/84408397 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 msgr2=0x7f409c109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.657+0000 7f40a2e00640 1 --2- 172.21.15.116:0/84408397 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 0x7f409c109a20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4094009900 tx=0x7f409402bc30 comp rx=0 tx=0).stop 2024-03-04T07:51:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.658+0000 7f40a2e00640 1 -- 172.21.15.116:0/84408397 shutdown_connections 2024-03-04T07:51:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.658+0000 7f40a2e00640 1 --2- 172.21.15.116:0/84408397 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 0x7f409c109a20 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.658+0000 7f40a2e00640 1 --2- 172.21.15.116:0/84408397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f409c1027d0 0x7f409c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.658+0000 7f40a2e00640 1 --2- 172.21.15.116:0/84408397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1015e0 0x7f409c101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.658+0000 7f40a2e00640 1 -- 172.21.15.116:0/84408397 >> 172.21.15.116:0/84408397 conn(0x7f409c0fcb70 msgr2=0x7f409c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.658+0000 7f40a2e00640 1 -- 172.21.15.116:0/84408397 shutdown_connections 2024-03-04T07:51:38.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.658+0000 7f40a2e00640 1 -- 172.21.15.116:0/84408397 wait complete. 2024-03-04T07:51:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 Processor -- start 2024-03-04T07:51:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 -- start start 2024-03-04T07:51:38.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f409c1015e0 0x7f409c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1027d0 0x7f409c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 0x7f409c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c10d2c0 con 0x7f409c1027d0 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c10d140 con 0x7f409c103190 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a2e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c10d440 con 0x7f409c1015e0 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f409b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1027d0 0x7f409c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f409b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1027d0 0x7f409c19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50228/0 (socket says 172.21.15.116:50228) 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f409b400640 1 -- 172.21.15.116:0/617350458 learned_addr learned my addr 172.21.15.116:0/617350458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:38.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f409be00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f409c1015e0 0x7f409c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.659+0000 7f40a0e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 0x7f409c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f409b400640 1 -- 172.21.15.116:0/617350458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f409c1015e0 msgr2=0x7f409c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f409b400640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f409c1015e0 0x7f409c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f409b400640 1 -- 172.21.15.116:0/617350458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 msgr2=0x7f409c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f409b400640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 0x7f409c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f409b400640 1 -- 172.21.15.116:0/617350458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c1a1790 con 0x7f409c1027d0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f40a0e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 0x7f409c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f409b400640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1027d0 0x7f409c19a0b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f408800d770 tx=0x7f408800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f4098c00640 1 -- 172.21.15.116:0/617350458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088017070 con 0x7f409c1027d0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c1a1a80 con 0x7f409c1027d0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f4098c00640 1 -- 172.21.15.116:0/617350458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4088005500 con 0x7f409c1027d0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f4098c00640 1 -- 172.21.15.116:0/617350458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088004e00 con 0x7f409c1027d0 2024-03-04T07:51:38.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.660+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409c1a1fd0 con 0x7f409c1027d0 2024-03-04T07:51:38.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.664+0000 7f40a5b74740 1 Processor -- start 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a5b74740 1 -- start start 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b070660c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b0706b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b0706dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a5b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b06fcad60 con 0x560b0706b800 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a5b74740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b06f73cf0 con 0x560b07065ca0 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a5b74740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b06fc6e00 con 0x560b07066690 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a0e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b070660c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f409be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b0706b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f409be00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b0706b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:38598/0 (socket says 172.21.15.116:38598) 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f409be00640 1 -- 172.21.15.116:0/3048692564 learned_addr learned my addr 172.21.15.116:0/3048692564 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f40a0e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b070660c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41766/0 (socket says 172.21.15.116:41766) 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.665+0000 7f4098c00640 1 -- 172.21.15.116:0/617350458 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4088004fa0 con 0x7f409c1027d0 2024-03-04T07:51:38.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f40a0e00640 1 -- 172.21.15.116:0/3048692564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 msgr2=0x560b0706b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f409b400640 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b0706dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f40a0e00640 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b0706b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f40a0e00640 1 -- 172.21.15.116:0/3048692564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 msgr2=0x560b0706dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f40a0e00640 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b0706dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f40a0e00640 1 -- 172.21.15.116:0/3048692564 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b06cdf3f0 con 0x560b07065ca0 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f409be00640 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b0706b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f409b400640 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b0706dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f40a0e00640 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b070660c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f409402c170 tx=0x7f4094009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acb3163310234214 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f4091600640 1 -- 172.21.15.116:0/3048692564 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094045070 con 0x560b07065ca0 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f4091600640 1 -- 172.21.15.116:0/3048692564 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409402cd30 con 0x560b07065ca0 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.666+0000 7f4091600640 1 -- 172.21.15.116:0/3048692564 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094005150 con 0x560b07065ca0 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f4098c00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f40800a1e00 0x7f40800a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 -- 172.21.15.116:0/3048692564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 msgr2=0x560b070660c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b070660c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f409402c170 tx=0x7f4094009560 comp rx=0 tx=0).stop 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 -- 172.21.15.116:0/3048692564 shutdown_connections 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b0706dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b0706b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 --2- 172.21.15.116:0/3048692564 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b070660c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 -- 172.21.15.116:0/3048692564 >> 172.21.15.116:0/3048692564 conn(0x560b07061210 msgr2=0x560b07061610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f4098c00640 1 -- 172.21.15.116:0/617350458 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f40880cd670 con 0x7f409c1027d0 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 -- 172.21.15.116:0/3048692564 shutdown_connections 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f409be00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f40800a1e00 0x7f40800a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.667+0000 7f40a5b74740 1 -- 172.21.15.116:0/3048692564 wait complete. 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 -- start start 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f409be00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f40800a1e00 0x7f40800a42c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f408c005fd0 tx=0x7f408c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 Processor -- start 2024-03-04T07:51:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07065ca0 0x560b07070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b07070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070777c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b0706e750 con 0x560b0706b800 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b06f69220 con 0x560b07066690 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.668+0000 7f40a5b74740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b0706e8d0 con 0x560b07065ca0 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f40a5b74740 10 client.0 _ll_register_callbacks cb 0x560b06cdebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f40a0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070777c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b07070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b07070780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41782/0 (socket says 172.21.15.116:41782) 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f40a0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070777c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50234/0 (socket says 172.21.15.116:50234) 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 -- 172.21.15.116:0/341216953 learned_addr learned my addr 172.21.15.116:0/341216953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409be00640 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07065ca0 0x560b07070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 -- 172.21.15.116:0/341216953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07065ca0 msgr2=0x560b07070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07065ca0 0x560b07070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 -- 172.21.15.116:0/341216953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 msgr2=0x560b070777c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070777c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 -- 172.21.15.116:0/341216953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b06da97d0 con 0x560b07066690 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f40a0e00640 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070777c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f409b400640 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b07070780 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4088096870 tx=0x7f4088099d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdb5ab276d1c136a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f4086000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f4086000640 1 -- 172.21.15.116:0/341216953 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088020050 con 0x560b07066690 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.669+0000 7f4086000640 1 -- 172.21.15.116:0/341216953 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4088009cc0 con 0x560b07066690 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f409402cd30 con 0x560b07066690 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f4086000640 1 -- 172.21.15.116:0/341216953 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f408809e4b0 con 0x560b07066690 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 10 client.28088 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560b0707ab10 con 0x560b07066690 2024-03-04T07:51:38.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f4086000640 1 -- 172.21.15.116:0/341216953 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409402cd30 con 0x560b07066690 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f4086000640 1 -- 172.21.15.116:0/341216953 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f40880a2670 con 0x560b07066690 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 10 client.28088 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 10 client.28088 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 20 client.28088 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b0707c1b0 0x560b0707e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 4 client.28088 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b06fd82a0 con 0x560b0707c1b0 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.670+0000 7f409be00640 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b0707c1b0 0x560b0707e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.671+0000 7f409be00640 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b0707c1b0 0x560b0707e5f0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd94c7ebaf5301ff server_cookie=c08981fb0546598a in_seq=0 out_seq=0 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.671+0000 7f4086000640 10 client.28088 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.671+0000 7f4086000640 1 -- 172.21.15.116:0/341216953 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560b06fd82a0 con 0x560b0707c1b0 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.671+0000 7f4086000640 10 client.28088 handle_command_reply: tid=0 2024-03-04T07:51:38.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 client.28088 shutdown 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 msgr2=0x560b07070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b07070780 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4088096870 tx=0x7f4088099d70 comp rx=0 tx=0).stop 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 shutdown_connections 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b0707c1b0 0x560b0707e5f0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070777c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f4086000640 0 client.28088 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b07070780 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 --2- 172.21.15.116:0/341216953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07065ca0 0x560b07070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 >> 172.21.15.116:0/341216953 conn(0x560b07061210 msgr2=0x560b070604b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 shutdown_connections 2024-03-04T07:51:38.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 1 -- 172.21.15.116:0/341216953 wait complete. 2024-03-04T07:51:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.672+0000 7f40a5b74740 20 client.28088 trim_cache size 0 max 16384 2024-03-04T07:51:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.685+0000 7f40a5b74740 1 Processor -- start 2024-03-04T07:51:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.685+0000 7f40a5b74740 1 -- start start 2024-03-04T07:51:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.685+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b07082eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.685+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b07077780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.685+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070833f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f40a5b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c1a8c80 con 0x560b0706b800 2024-03-04T07:51:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f40a5b74740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c1ab260 con 0x560b07065ca0 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f40a5b74740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c1a9280 con 0x560b07066690 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b07082eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b07082eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41798/0 (socket says 172.21.15.116:41798) 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 -- 172.21.15.116:0/282389804 learned_addr learned my addr 172.21.15.116:0/282389804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f40a0e00640 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070833f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409b400640 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b07077780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 -- 172.21.15.116:0/282389804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 msgr2=0x560b07077780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b07077780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 -- 172.21.15.116:0/282389804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 msgr2=0x560b070833f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070833f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 -- 172.21.15.116:0/282389804 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b07061c80 con 0x560b07065ca0 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409b400640 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b07077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:51:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.686+0000 7f409be00640 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b07082eb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x560b07066a90 tx=0x7f408c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa9d7b57e7c94be server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f4090c00640 1 -- 172.21.15.116:0/282389804 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c004620 con 0x560b07065ca0 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f4090c00640 1 -- 172.21.15.116:0/282389804 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f408c0047c0 con 0x560b07065ca0 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f4090c00640 1 -- 172.21.15.116:0/282389804 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c005020 con 0x560b07065ca0 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 -- 172.21.15.116:0/282389804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 msgr2=0x560b07082eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b07082eb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x560b07066a90 tx=0x7f408c016000 comp rx=0 tx=0).stop 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 -- 172.21.15.116:0/282389804 shutdown_connections 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b0706b800 0x560b070833f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b07066690 0x560b07077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 --2- 172.21.15.116:0/282389804 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07065ca0 0x560b07082eb0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 -- 172.21.15.116:0/282389804 >> 172.21.15.116:0/282389804 conn(0x560b07061210 msgr2=0x560b07070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 -- 172.21.15.116:0/282389804 shutdown_connections 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.687+0000 7f40a5b74740 1 -- 172.21.15.116:0/282389804 wait complete. 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 -- start start 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 Processor -- start 2024-03-04T07:51:38.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b07065ca0 0x560b07074db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b070752f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b0706b800 0x560b070762b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c1a98e0 con 0x560b07065ca0 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c1aac00 con 0x560b07066690 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c1a9f40 con 0x560b0706b800 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f40a5b74740 10 client.0 _ll_register_callbacks cb 0x7f4088005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f409be00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b07065ca0 0x560b07074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f409b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b070752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f409b400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b070752f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41808/0 (socket says 172.21.15.116:41808) 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.688+0000 7f409b400640 1 -- 172.21.15.116:0/697506216 learned_addr learned my addr 172.21.15.116:0/697506216 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f40a0e00640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b0706b800 0x560b070762b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f409be00640 1 -- 172.21.15.116:0/697506216 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b0706b800 msgr2=0x560b070762b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f409be00640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b0706b800 0x560b070762b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f409be00640 1 -- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 msgr2=0x560b070752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f409be00640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b070752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f409be00640 1 -- 172.21.15.116:0/697506216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b070768a0 con 0x560b07065ca0 2024-03-04T07:51:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f40a0e00640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b0706b800 0x560b070762b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f409b400640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b070752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f409be00640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b07065ca0 0x560b07074db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f408c0049d0 tx=0x7f408c004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8c62d5cd2dd47 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f4086000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f4086000640 1 -- 172.21.15.116:0/697506216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c037070 con 0x560b07065ca0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f408c0047c0 con 0x560b07065ca0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.689+0000 7f4086000640 1 -- 172.21.15.116:0/697506216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f408c02a680 con 0x560b07065ca0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f4086000640 1 -- 172.21.15.116:0/697506216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f408c043050 con 0x560b07065ca0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 10 client.17949 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x560b070d7a80 con 0x560b07065ca0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f4086000640 1 -- 172.21.15.116:0/697506216 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f408c0047c0 con 0x560b07065ca0 2024-03-04T07:51:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f4086000640 1 -- 172.21.15.116:0/697506216 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f408c005410 con 0x560b07065ca0 2024-03-04T07:51:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 10 client.17949 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 10 client.17949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 20 client.17949 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b070d8220 0x560b070da640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 4 client.17949 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:51:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560b06fd25f0 con 0x560b070d8220 2024-03-04T07:51:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.690+0000 7f409b400640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b070d8220 0x560b070da640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f409b400640 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b070d8220 0x560b070da640 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77c761512670992b server_cookie=d7629c1138617efb in_seq=0 out_seq=0 2024-03-04T07:51:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f4086000640 10 client.17949 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f4086000640 1 -- 172.21.15.116:0/697506216 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560b06fd25f0 con 0x560b070d8220 2024-03-04T07:51:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f4086000640 10 client.17949 handle_command_reply: tid=0 2024-03-04T07:51:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 client.17949 shutdown 2024-03-04T07:51:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b07065ca0 msgr2=0x560b07074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b07065ca0 0x560b07074db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f408c0049d0 tx=0x7f408c004620 comp rx=0 tx=0).stop 2024-03-04T07:51:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 shutdown_connections 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x560b070d8220 0x560b070da640 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b0706b800 0x560b070762b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x560b07066690 0x560b070752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f4086000640 0 client.17949 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 --2- 172.21.15.116:0/697506216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b07065ca0 0x560b07074db0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 >> 172.21.15.116:0/697506216 conn(0x560b07061210 msgr2=0x560b07064470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 shutdown_connections 2024-03-04T07:51:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.691+0000 7f40a5b74740 1 -- 172.21.15.116:0/697506216 wait complete. 2024-03-04T07:51:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a5b74740 20 client.17949 trim_cache size 0 max 16384 2024-03-04T07:51:38.702 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:51:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f40800a1e00 msgr2=0x7f40800a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f40800a1e00 0x7f40800a42c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f408c005fd0 tx=0x7f408c009500 comp rx=0 tx=0).stop 2024-03-04T07:51:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1027d0 msgr2=0x7f409c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1027d0 0x7f409c19a0b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f408800d770 tx=0x7f408800dc30 comp rx=0 tx=0).stop 2024-03-04T07:51:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 shutdown_connections 2024-03-04T07:51:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f409c103190 0x7f409c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c1027d0 0x7f409c19a0b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f40800a1e00 0x7f40800a42c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 --2- 172.21.15.116:0/617350458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f409c1015e0 0x7f409c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 >> 172.21.15.116:0/617350458 conn(0x7f409c0fcb70 msgr2=0x7f409c073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 shutdown_connections 2024-03-04T07:51:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:38.692+0000 7f40a2e00640 1 -- 172.21.15.116:0/617350458 wait complete. 2024-03-04T07:51:39.008 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:38 smithi116 ceph-mon[32150]: pgmap v727: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:51:39.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:38 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/227627759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:51:39.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:38 smithi152 ceph-mon[37154]: pgmap v727: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:51:39.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:38 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/227627759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:51:39.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:38 smithi133 ceph-mon[36505]: pgmap v727: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:51:39.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:38 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/227627759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:51:39.710 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T07:51:39.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:51:39.711 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T07:51:39.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 -- 172.21.15.116:0/2037255203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 msgr2=0x7f0970108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 --2- 172.21.15.116:0/2037255203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f0970108010 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0960009900 tx=0x7f096002bc30 comp rx=0 tx=0).stop 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 -- 172.21.15.116:0/2037255203 shutdown_connections 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 --2- 172.21.15.116:0/2037255203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09701097a0 0x7f0970110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 --2- 172.21.15.116:0/2037255203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0970108de0 0x7f0970109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 --2- 172.21.15.116:0/2037255203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f0970108010 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 -- 172.21.15.116:0/2037255203 >> 172.21.15.116:0/2037255203 conn(0x7f0970076b90 msgr2=0x7f0970078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 -- 172.21.15.116:0/2037255203 shutdown_connections 2024-03-04T07:51:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.937+0000 7f0975c00640 1 -- 172.21.15.116:0/2037255203 wait complete. 2024-03-04T07:51:39.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.938+0000 7f0975c00640 1 Processor -- start 2024-03-04T07:51:39.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.938+0000 7f0975c00640 1 -- start start 2024-03-04T07:51:39.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f0975c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f097019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f0975c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0970108de0 0x7f097019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f0975c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09701097a0 0x7f09701a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f0975c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09701138b0 con 0x7f09701097a0 2024-03-04T07:51:39.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f0975c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970113730 con 0x7f0970107bf0 2024-03-04T07:51:39.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f0975c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970113a30 con 0x7f0970108de0 2024-03-04T07:51:39.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f097019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f097019df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41820/0 (socket says 172.21.15.116:41820) 2024-03-04T07:51:39.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 -- 172.21.15.116:0/3842936720 learned_addr learned my addr 172.21.15.116:0/3842936720 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:39.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096f400640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09701097a0 0x7f09701a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f0967e00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0970108de0 0x7f097019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 -- 172.21.15.116:0/3842936720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0970108de0 msgr2=0x7f097019e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0970108de0 0x7f097019e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 -- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09701097a0 msgr2=0x7f09701a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09701097a0 0x7f09701a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.939+0000 7f096ea00640 1 -- 172.21.15.116:0/3842936720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09701a5ba0 con 0x7f0970107bf0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.940+0000 7f096ea00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f097019df80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0960000c00 tx=0x7f096002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.940+0000 7f0967400640 1 -- 172.21.15.116:0/3842936720 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960043020 con 0x7f0970107bf0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.940+0000 7f0967400640 1 -- 172.21.15.116:0/3842936720 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09600054e0 con 0x7f0970107bf0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.940+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09701a5e30 con 0x7f0970107bf0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.940+0000 7f0967400640 1 -- 172.21.15.116:0/3842936720 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960003bd0 con 0x7f0970107bf0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.940+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09701a6290 con 0x7f0970107bf0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.943+0000 7f0967400640 1 -- 172.21.15.116:0/3842936720 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f0960046020 con 0x7f0970107bf0 2024-03-04T07:51:39.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.943+0000 7f0978a19740 1 Processor -- start 2024-03-04T07:51:39.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0978a19740 1 -- start start 2024-03-04T07:51:39.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0978a19740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0978a19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0978a19740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2edc30 0x55a29b2f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0978a19740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a29b1fbe20 con 0x55a29b2e8ac0 2024-03-04T07:51:39.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0978a19740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a29b24da20 con 0x55a29b2e80d0 2024-03-04T07:51:39.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0978a19740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a29b1f6b00 con 0x55a29b2edc30 2024-03-04T07:51:39.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f096f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f096f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41830/0 (socket says 172.21.15.116:41830) 2024-03-04T07:51:39.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f096f400640 1 -- 172.21.15.116:0/698177217 learned_addr learned my addr 172.21.15.116:0/698177217 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:39.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f096ea00640 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.944+0000 7f0967e00640 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2edc30 0x55a29b2f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096f400640 1 -- 172.21.15.116:0/698177217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2edc30 msgr2=0x55a29b2f00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096f400640 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2edc30 0x55a29b2f00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096f400640 1 -- 172.21.15.116:0/698177217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 msgr2=0x55a29b2ed5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096f400640 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2ed5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096f400640 1 -- 172.21.15.116:0/698177217 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a29b20be10 con 0x55a29b2e80d0 2024-03-04T07:51:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096ea00640 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2ed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f0967400640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09540a1d80 0x7f09540a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096f400640 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2e84f0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0968011440 tx=0x7f0968011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df47f81f5ec94d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:39.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f0967400640 1 -- 172.21.15.116:0/3842936720 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f09600efdf0 con 0x7f0970107bf0 2024-03-04T07:51:39.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f095fe00640 1 -- 172.21.15.116:0/698177217 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968003c30 con 0x55a29b2e80d0 2024-03-04T07:51:39.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f096ea00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09540a1d80 0x7f09540a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f095fe00640 1 -- 172.21.15.116:0/698177217 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0968003dd0 con 0x55a29b2e80d0 2024-03-04T07:51:39.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.945+0000 7f095fe00640 1 -- 172.21.15.116:0/698177217 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096800fb50 con 0x55a29b2e80d0 2024-03-04T07:51:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 -- 172.21.15.116:0/698177217 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 msgr2=0x55a29b2e84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2e84f0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0968011440 tx=0x7f0968011900 comp rx=0 tx=0).stop 2024-03-04T07:51:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f096ea00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09540a1d80 0x7f09540a4240 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f09600b94b0 tx=0x7f0960009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 -- 172.21.15.116:0/698177217 shutdown_connections 2024-03-04T07:51:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2edc30 0x55a29b2f00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2ed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 --2- 172.21.15.116:0/698177217 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2e84f0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 -- 172.21.15.116:0/698177217 >> 172.21.15.116:0/698177217 conn(0x55a29b2e3640 msgr2=0x55a29b2e3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:39.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 -- 172.21.15.116:0/698177217 shutdown_connections 2024-03-04T07:51:39.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.946+0000 7f0978a19740 1 -- 172.21.15.116:0/698177217 wait complete. 2024-03-04T07:51:39.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 -- start start 2024-03-04T07:51:39.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 Processor -- start 2024-03-04T07:51:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2f9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a29b209f40 con 0x55a29b2e8ac0 2024-03-04T07:51:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a29b249ac0 con 0x55a29b2edc30 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a29b2f0be0 con 0x55a29b2e80d0 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0978a19740 10 client.0 _ll_register_callbacks cb 0x55a29b248aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f096f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f0967e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.947+0000 7f096f400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2f24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:38686/0 (socket says 172.21.15.116:38686) 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f096f400640 1 -- 172.21.15.116:0/3723093831 learned_addr learned my addr 172.21.15.116:0/3723093831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f096f400640 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2f9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f0967e00640 1 -- 172.21.15.116:0/3723093831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 msgr2=0x55a29b2f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f0967e00640 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f0967e00640 1 -- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 msgr2=0x55a29b2f9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f0967e00640 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2f9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f0967e00640 1 -- 172.21.15.116:0/3723093831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a29b02a7e0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f096f400640 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2f9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f0967e00640 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f29e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f097019f490 tx=0x7f0958012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3a70258449f8c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f095cc00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f095cc00640 1 -- 172.21.15.116:0/3723093831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0958004020 con 0x55a29b2e8ac0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.948+0000 7f095cc00640 1 -- 172.21.15.116:0/3723093831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f09580041c0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0968003dd0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f095cc00640 1 -- 172.21.15.116:0/3723093831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09580043c0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 10 client.17961 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a29b08af70 con 0x55a29b2e8ac0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f095cc00640 1 -- 172.21.15.116:0/3723093831 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0968003dd0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f095cc00640 1 -- 172.21.15.116:0/3723093831 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0958003b80 con 0x55a29b2e8ac0 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 10 client.17961 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 10 client.17961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 20 client.17961 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b2fe430 0x55a29b300850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 4 client.17961 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.949+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a29b25af60 con 0x55a29b2fe430 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.950+0000 7f096f400640 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b2fe430 0x55a29b300850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.950+0000 7f096f400640 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b2fe430 0x55a29b300850 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=645bdf9d85fdeb37 server_cookie=ba49f3f3b1392ccf in_seq=0 out_seq=0 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.950+0000 7f095cc00640 10 client.17961 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f095cc00640 1 -- 172.21.15.116:0/3723093831 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a29b25af60 con 0x55a29b2fe430 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f095cc00640 10 client.17961 handle_command_reply: tid=0 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 client.17961 shutdown 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 msgr2=0x55a29b2f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f29e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f097019f490 tx=0x7f0958012b60 comp rx=0 tx=0).stop 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 shutdown_connections 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b2fe430 0x55a29b300850 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2f9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f29e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 --2- 172.21.15.116:0/3723093831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f095cc00640 0 client.17961 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 >> 172.21.15.116:0/3723093831 conn(0x55a29b2e3640 msgr2=0x55a29b2e4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 shutdown_connections 2024-03-04T07:51:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 1 -- 172.21.15.116:0/3723093831 wait complete. 2024-03-04T07:51:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.951+0000 7f0978a19740 20 client.17961 trim_cache size 0 max 16384 2024-03-04T07:51:39.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 Processor -- start 2024-03-04T07:51:39.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 -- start start 2024-03-04T07:51:39.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2e2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09701ad030 con 0x55a29b2e8ac0 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09701af610 con 0x55a29b2edc30 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0978a19740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09701ad630 con 0x55a29b2e80d0 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f0967e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.969+0000 7f096f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f096f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2e6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41854/0 (socket says 172.21.15.116:41854) 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f096f400640 1 -- 172.21.15.116:0/1482307056 learned_addr learned my addr 172.21.15.116:0/1482307056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f096f400640 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2e2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f0967e00640 1 -- 172.21.15.116:0/1482307056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 msgr2=0x55a29b2e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f0967e00640 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2e2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f0967e00640 1 -- 172.21.15.116:0/1482307056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 msgr2=0x55a29b2e6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f0967e00640 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2e6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f0967e00640 1 -- 172.21.15.116:0/1482307056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a29af22170 con 0x55a29b2e8ac0 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f096f400640 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f0967e00640 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f99e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0958009de0 tx=0x7f0958009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd44f0613cf2d424 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.970+0000 7f095f400640 1 -- 172.21.15.116:0/1482307056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095801c5e0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f095f400640 1 -- 172.21.15.116:0/1482307056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0958003f70 con 0x55a29b2e8ac0 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f095f400640 1 -- 172.21.15.116:0/1482307056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095800fdd0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 -- 172.21.15.116:0/1482307056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 msgr2=0x55a29b2f99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f99e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0958009de0 tx=0x7f0958009e80 comp rx=0 tx=0).stop 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 -- 172.21.15.116:0/1482307056 shutdown_connections 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2edc30 0x55a29b2e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f99e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 --2- 172.21.15.116:0/1482307056 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e80d0 0x55a29b2e2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 -- 172.21.15.116:0/1482307056 >> 172.21.15.116:0/1482307056 conn(0x55a29b2e3640 msgr2=0x55a29b2e68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:39.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 -- 172.21.15.116:0/1482307056 shutdown_connections 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.971+0000 7f0978a19740 1 -- 172.21.15.116:0/1482307056 wait complete. 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 -- start start 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 Processor -- start 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2f93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2edc30 0x55a29b2f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09701adc90 con 0x55a29b2edc30 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09701aefb0 con 0x55a29b2e80d0 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09701ae2f0 con 0x55a29b2e8ac0 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.972+0000 7f0978a19740 10 client.0 _ll_register_callbacks cb 0x7f0958009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2f93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2f93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41864/0 (socket says 172.21.15.116:41864) 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 -- 172.21.15.116:0/2377735192 learned_addr learned my addr 172.21.15.116:0/2377735192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2edc30 0x55a29b2f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f0967e00640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 -- 172.21.15.116:0/2377735192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e8ac0 msgr2=0x55a29b2f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 -- 172.21.15.116:0/2377735192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2edc30 msgr2=0x55a29b2f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2edc30 0x55a29b2f79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 -- 172.21.15.116:0/2377735192 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a29b2f8200 con 0x55a29b2e80d0 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f0967e00640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f096f400640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2f93e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0968011e00 tx=0x7f0958009e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=162b64711cf1340e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f095cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.973+0000 7f095cc00640 1 -- 172.21.15.116:0/2377735192 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968004610 con 0x55a29b2e80d0 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f095cc00640 1 -- 172.21.15.116:0/2377735192 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f09680047b0 con 0x55a29b2e80d0 2024-03-04T07:51:39.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0958003f70 con 0x55a29b2e80d0 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f095cc00640 1 -- 172.21.15.116:0/2377735192 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0968004a00 con 0x55a29b2e80d0 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 10 client.28112 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55a29b35a800 con 0x55a29b2e80d0 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f095cc00640 1 -- 172.21.15.116:0/2377735192 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0958003f70 con 0x55a29b2e80d0 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f095cc00640 1 -- 172.21.15.116:0/2377735192 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f096800f540 con 0x55a29b2e80d0 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 10 client.28112 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 10 client.28112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 20 client.28112 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b35b000 0x55a29b35d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 4 client.28112 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.974+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a29b2552b0 con 0x55a29b35b000 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.975+0000 7f0967e00640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b35b000 0x55a29b35d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.975+0000 7f0967e00640 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b35b000 0x55a29b35d420 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d16b5c8579647664 server_cookie=cba5357668207922 in_seq=0 out_seq=0 2024-03-04T07:51:39.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.975+0000 7f095cc00640 10 client.28112 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.976+0000 7f095cc00640 1 -- 172.21.15.116:0/2377735192 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a29b2552b0 con 0x55a29b35b000 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.976+0000 7f095cc00640 10 client.28112 handle_command_reply: tid=0 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.976+0000 7f0978a19740 1 client.28112 shutdown 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 msgr2=0x55a29b2f93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2f93e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0968011e00 tx=0x7f0958009e10 comp rx=0 tx=0).stop 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 shutdown_connections 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55a29b35b000 0x55a29b35d420 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a29b2edc30 0x55a29b2f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a29b2e8ac0 0x55a29b2f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 --2- 172.21.15.116:0/2377735192 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55a29b2e80d0 0x55a29b2f93e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f095cc00640 0 client.28112 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 >> 172.21.15.116:0/2377735192 conn(0x55a29b2e3640 msgr2=0x55a29b2e7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 shutdown_connections 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 1 -- 172.21.15.116:0/2377735192 wait complete. 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.977+0000 7f0978a19740 20 client.28112 trim_cache size 0 max 16384 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "793c6118-b9c4-40c3-8d77-09db7d6a8a79", 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-04T07:51:39.984 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:51:39.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09540a1d80 msgr2=0x7f09540a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09540a1d80 0x7f09540a4240 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f09600b94b0 tx=0x7f0960009260 comp rx=0 tx=0).stop 2024-03-04T07:51:39.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 msgr2=0x7f097019df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:39.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f097019df80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0960000c00 tx=0x7f096002c650 comp rx=0 tx=0).stop 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 shutdown_connections 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09701097a0 0x7f09701a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f09540a1d80 0x7f09540a4240 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0970108de0 0x7f097019e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 --2- 172.21.15.116:0/3842936720 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0970107bf0 0x7f097019df80 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 >> 172.21.15.116:0/3842936720 conn(0x7f0970076b90 msgr2=0x7f097010c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 shutdown_connections 2024-03-04T07:51:39.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:39.978+0000 7f0975c00640 1 -- 172.21.15.116:0/3842936720 wait complete. 2024-03-04T07:51:39.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:51:39.996 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:51:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.205+0000 7fa499e00640 1 -- 172.21.15.116:0/566716104 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941052a0 msgr2=0x7fa49410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.205+0000 7fa499e00640 1 --2- 172.21.15.116:0/566716104 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941052a0 0x7fa49410bb30 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009ca0 tx=0x7fa48c02c100 comp rx=0 tx=0).stop 2024-03-04T07:51:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.206+0000 7fa499e00640 1 -- 172.21.15.116:0/566716104 shutdown_connections 2024-03-04T07:51:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.206+0000 7fa499e00640 1 --2- 172.21.15.116:0/566716104 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941052a0 0x7fa49410bb30 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.206+0000 7fa499e00640 1 --2- 172.21.15.116:0/566716104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4941048e0 0x7fa494104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.206+0000 7fa499e00640 1 --2- 172.21.15.116:0/566716104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4941036f0 0x7fa494103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.206+0000 7fa499e00640 1 -- 172.21.15.116:0/566716104 >> 172.21.15.116:0/566716104 conn(0x7fa4940fec80 msgr2=0x7fa4941010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:40.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.206+0000 7fa499e00640 1 -- 172.21.15.116:0/566716104 shutdown_connections 2024-03-04T07:51:40.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.206+0000 7fa499e00640 1 -- 172.21.15.116:0/566716104 wait complete. 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.207+0000 7fa499e00640 1 Processor -- start 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.207+0000 7fa499e00640 1 -- start start 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa499e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4941036f0 0x7fa494199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa499e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941048e0 0x7fa494199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa499e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4941052a0 0x7fa4941a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa499e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49410f350 con 0x7fa4941052a0 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa499e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49410f1d0 con 0x7fa4941048e0 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa499e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49410f4d0 con 0x7fa4941036f0 2024-03-04T07:51:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941048e0 0x7fa494199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941048e0 0x7fa494199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41876/0 (socket says 172.21.15.116:41876) 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 -- 172.21.15.116:0/3436894327 learned_addr learned my addr 172.21.15.116:0/3436894327 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa498a00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4941036f0 0x7fa494199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa499400640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4941052a0 0x7fa4941a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 -- 172.21.15.116:0/3436894327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4941036f0 msgr2=0x7fa494199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4941036f0 0x7fa494199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 -- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4941052a0 msgr2=0x7fa4941a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4941052a0 0x7fa4941a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.208+0000 7fa493e00640 1 -- 172.21.15.116:0/3436894327 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4941a1590 con 0x7fa4941048e0 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.209+0000 7fa493e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941048e0 0x7fa494199eb0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e9f0 tx=0x7fa48000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.209+0000 7fa491600640 1 -- 172.21.15.116:0/3436894327 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48000cd80 con 0x7fa4941048e0 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.209+0000 7fa491600640 1 -- 172.21.15.116:0/3436894327 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa480010910 con 0x7fa4941048e0 2024-03-04T07:51:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.209+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4941a1880 con 0x7fa4941048e0 2024-03-04T07:51:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.209+0000 7fa491600640 1 -- 172.21.15.116:0/3436894327 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800188f0 con 0x7fa4941048e0 2024-03-04T07:51:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.209+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4941a1dd0 con 0x7fa4941048e0 2024-03-04T07:51:40.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.212+0000 7fa491600640 1 -- 172.21.15.116:0/3436894327 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa480010ac0 con 0x7fa4941048e0 2024-03-04T07:51:40.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.212+0000 7fa49e18c740 1 Processor -- start 2024-03-04T07:51:40.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa49e18c740 1 -- start start 2024-03-04T07:51:40.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337102d10 0x565337103130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337103700 0x565337108200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x56533710ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa49e18c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x565337030090 con 0x565337103700 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa49e18c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x565337067d30 con 0x565337102d10 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa49e18c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565337010a30 con 0x565337108870 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa499400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337102d10 0x565337103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa499400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337102d10 0x565337103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41892/0 (socket says 172.21.15.116:41892) 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa499400640 1 -- 172.21.15.116:0/4193962144 learned_addr learned my addr 172.21.15.116:0/4193962144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa498a00640 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337103700 0x565337108200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.213+0000 7fa493e00640 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x56533710ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa499400640 1 -- 172.21.15.116:0/4193962144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 msgr2=0x56533710ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa499400640 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x56533710ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa499400640 1 -- 172.21.15.116:0/4193962144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337103700 msgr2=0x565337108200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa499400640 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337103700 0x565337108200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa499400640 1 -- 172.21.15.116:0/4193962144 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565336e467d0 con 0x565337102d10 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa498a00640 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337103700 0x565337108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa499400640 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337102d10 0x565337103130 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa48c02c710 tx=0x7fa48c0323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=966bb51198ccb165 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa491600640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa47c0a1d80 0x7fa47c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa485c00640 1 -- 172.21.15.116:0/4193962144 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c043070 con 0x565337102d10 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa485c00640 1 -- 172.21.15.116:0/4193962144 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa48c032c90 con 0x565337102d10 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa491600640 1 -- 172.21.15.116:0/3436894327 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa4800ce0d0 con 0x7fa4941048e0 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa498a00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa47c0a1d80 0x7fa47c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.214+0000 7fa485c00640 1 -- 172.21.15.116:0/4193962144 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c039040 con 0x565337102d10 2024-03-04T07:51:40.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 -- 172.21.15.116:0/4193962144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337102d10 msgr2=0x565337103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337102d10 0x565337103130 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa48c02c710 tx=0x7fa48c0323d0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa498a00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa47c0a1d80 0x7fa47c0a4240 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x565337103b00 tx=0x7fa48800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 -- 172.21.15.116:0/4193962144 shutdown_connections 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x56533710ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337103700 0x565337108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4193962144 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337102d10 0x565337103130 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 -- 172.21.15.116:0/4193962144 >> 172.21.15.116:0/4193962144 conn(0x5653370fe280 msgr2=0x5653370fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 -- 172.21.15.116:0/4193962144 shutdown_connections 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.215+0000 7fa49e18c740 1 -- 172.21.15.116:0/4193962144 wait complete. 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 -- start start 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 Processor -- start 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337102d10 0x56533710d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x56533710d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x565337114600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653370060c0 con 0x565337102d10 2024-03-04T07:51:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x565337063dd0 con 0x565337103700 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56533710b7c0 con 0x565337108870 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.216+0000 7fa49e18c740 10 client.0 _ll_register_callbacks cb 0x565337023860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x56533710d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa498a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337102d10 0x56533710d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x56533710d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41902/0 (socket says 172.21.15.116:41902) 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 -- 172.21.15.116:0/3869024839 learned_addr learned my addr 172.21.15.116:0/3869024839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 -- 172.21.15.116:0/3869024839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 msgr2=0x565337114600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa499400640 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x565337114600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x565337114600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 -- 172.21.15.116:0/3869024839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337102d10 msgr2=0x56533710d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337102d10 0x56533710d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 -- 172.21.15.116:0/3869024839 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565336d7c3f0 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa493e00640 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x56533710d5c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fa480097310 tx=0x7fa48009aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5765c3425689e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.217+0000 7fa47a000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa47a000640 1 -- 172.21.15.116:0/3869024839 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800d02b0 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa47a000640 1 -- 172.21.15.116:0/3869024839 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa48009a300 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa48c032c90 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa47a000640 1 -- 172.21.15.116:0/3869024839 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4800b8050 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 10 client.28124 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x565336d3c0f0 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa47a000640 1 -- 172.21.15.116:0/3869024839 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa48c032c90 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa47a000640 1 -- 172.21.15.116:0/3869024839 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa4800b3b30 con 0x565337103700 2024-03-04T07:51:40.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 10 client.28124 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 10 client.28124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 20 client.28124 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x565337118a20 0x56533711aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 4 client.28124 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.218+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565337075270 con 0x565337118a20 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.219+0000 7fa498a00640 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x565337118a20 0x56533711aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.219+0000 7fa498a00640 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x565337118a20 0x56533711aea0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5a5f4d44e43f862 server_cookie=c421f0f890222b01 in_seq=0 out_seq=0 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.219+0000 7fa47a000640 10 client.28124 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa47a000640 1 -- 172.21.15.116:0/3869024839 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565337075270 con 0x565337118a20 2024-03-04T07:51:40.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa47a000640 10 client.28124 handle_command_reply: tid=0 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 client.28124 shutdown 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 msgr2=0x56533710d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x56533710d5c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fa480097310 tx=0x7fa48009aaf0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 shutdown_connections 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x565337118a20 0x56533711aea0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337108870 0x565337114600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x56533710d5c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa47a000640 0 client.28124 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 --2- 172.21.15.116:0/3869024839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337102d10 0x56533710d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 >> 172.21.15.116:0/3869024839 conn(0x5653370fe280 msgr2=0x5653370fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 shutdown_connections 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 1 -- 172.21.15.116:0/3869024839 wait complete. 2024-03-04T07:51:40.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.220+0000 7fa49e18c740 20 client.28124 trim_cache size 0 max 16384 2024-03-04T07:51:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 Processor -- start 2024-03-04T07:51:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 -- start start 2024-03-04T07:51:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x5653370fcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x5653371145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x5653370fd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941a8a90 con 0x565337108870 2024-03-04T07:51:40.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941ab070 con 0x565337103700 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa49e18c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941a9090 con 0x565337102d10 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x5653371145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa499400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x5653370fd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x5653371145c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41910/0 (socket says 172.21.15.116:41910) 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa493e00640 1 -- 172.21.15.116:0/17907838 learned_addr learned my addr 172.21.15.116:0/17907838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.238+0000 7fa498a00640 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x5653370fcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa493e00640 1 -- 172.21.15.116:0/17907838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 msgr2=0x5653370fcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa493e00640 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x5653370fcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa493e00640 1 -- 172.21.15.116:0/17907838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 msgr2=0x5653370fd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa493e00640 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x5653370fd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa493e00640 1 -- 172.21.15.116:0/17907838 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565337102950 con 0x565337103700 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa493e00640 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x5653371145c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa480097880 tx=0x7fa48009a5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffb75ca8bf6f1c4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa485200640 1 -- 172.21.15.116:0/17907838 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800a35a0 con 0x565337103700 2024-03-04T07:51:40.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa485200640 1 -- 172.21.15.116:0/17907838 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4800a8020 con 0x565337103700 2024-03-04T07:51:40.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.239+0000 7fa485200640 1 -- 172.21.15.116:0/17907838 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480010690 con 0x565337103700 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 -- 172.21.15.116:0/17907838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 msgr2=0x5653371145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x5653371145c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa480097880 tx=0x7fa48009a5e0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 -- 172.21.15.116:0/17907838 shutdown_connections 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x5653370fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x5653371145c0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 --2- 172.21.15.116:0/17907838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x5653370fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 -- 172.21.15.116:0/17907838 >> 172.21.15.116:0/17907838 conn(0x5653370fe280 msgr2=0x56533710d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 -- 172.21.15.116:0/17907838 shutdown_connections 2024-03-04T07:51:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 -- 172.21.15.116:0/17907838 wait complete. 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 -- start start 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.240+0000 7fa49e18c740 1 Processor -- start 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x565337113f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x565337114450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa49e18c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x565337112570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa49e18c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941a96f0 con 0x565337108870 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa49e18c740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941aaa10 con 0x565337103700 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa49e18c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4941a9d50 con 0x565337102d10 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa49e18c740 10 client.0 _ll_register_callbacks cb 0x7fa48009ac80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x565337114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x565337114450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41916/0 (socket says 172.21.15.116:41916) 2024-03-04T07:51:40.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 -- 172.21.15.116:0/4034987077 learned_addr learned my addr 172.21.15.116:0/4034987077 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa498a00640 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x565337113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa499400640 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x565337112570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 -- 172.21.15.116:0/4034987077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 msgr2=0x565337113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x565337113f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 -- 172.21.15.116:0/4034987077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 msgr2=0x565337112570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x565337112570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.241+0000 7fa493e00640 1 -- 172.21.15.116:0/4034987077 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565337112da0 con 0x565337103700 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa493e00640 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x565337114450 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fa48000cfd0 tx=0x7fa480010850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6641cf43bc172525 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa47a000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa47a000640 1 -- 172.21.15.116:0/4034987077 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800b3ca0 con 0x565337103700 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4800a8020 con 0x565337103700 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa47a000640 1 -- 172.21.15.116:0/4034987077 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa480096e10 con 0x565337103700 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa47a000640 1 -- 172.21.15.116:0/4034987077 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4800b9240 con 0x565337103700 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 10 client.28130 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x565337174de0 con 0x565337103700 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa47a000640 1 -- 172.21.15.116:0/4034987077 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4800a8020 con 0x565337103700 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa47a000640 1 -- 172.21.15.116:0/4034987077 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa480087b00 con 0x565337103700 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 10 client.28130 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 10 client.28130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 20 client.28130 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:40.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5653371754f0 0x565337177910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.242+0000 7fa49e18c740 4 client.28130 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.243+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56533706f5c0 con 0x5653371754f0 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.243+0000 7fa498a00640 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5653371754f0 0x565337177910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.243+0000 7fa47a000640 10 client.28130 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.243+0000 7fa498a00640 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5653371754f0 0x565337177910 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60065fa704cf1b8 server_cookie=fb854d68eb28ef7c in_seq=0 out_seq=0 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.243+0000 7fa47a000640 1 -- 172.21.15.116:0/4034987077 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56533706f5c0 con 0x5653371754f0 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.243+0000 7fa47a000640 10 client.28130 handle_command_reply: tid=0 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.243+0000 7fa49e18c740 1 client.28130 shutdown 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 msgr2=0x565337114450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x565337114450 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fa48000cfd0 tx=0x7fa480010850 comp rx=0 tx=0).stop 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 shutdown_connections 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5653371754f0 0x565337177910 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565337108870 0x565337112570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x565337103700 0x565337114450 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa47a000640 0 client.28130 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 --2- 172.21.15.116:0/4034987077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565337102d10 0x565337113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 >> 172.21.15.116:0/4034987077 conn(0x5653370fe280 msgr2=0x5653371014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 shutdown_connections 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 1 -- 172.21.15.116:0/4034987077 wait complete. 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.244+0000 7fa49e18c740 20 client.28130 trim_cache size 0 max 16384 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stdout: "793c6118-b9c4-40c3-8d77-09db7d6a8a79": { 2024-03-04T07:51:40.251 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-04T07:51:40.252 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "793c6118-b9c4-40c3-8d77-09db7d6a8a79", 2024-03-04T07:51:40.252 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-04T07:51:40.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:51:40.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:51:40.252 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa47c0a1d80 msgr2=0x7fa47c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa47c0a1d80 0x7fa47c0a4240 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x565337103b00 tx=0x7fa48800b410 comp rx=0 tx=0).stop 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941048e0 msgr2=0x7fa494199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941048e0 0x7fa494199eb0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e9f0 tx=0x7fa48000eeb0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 shutdown_connections 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4941052a0 0x7fa4941a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa4941048e0 0x7fa494199eb0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa47c0a1d80 0x7fa47c0a4240 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 --2- 172.21.15.116:0/3436894327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4941036f0 0x7fa494199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:40.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 >> 172.21.15.116:0/3436894327 conn(0x7fa4940fec80 msgr2=0x7fa494107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:40.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 shutdown_connections 2024-03-04T07:51:40.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:51:40.245+0000 7fa499e00640 1 -- 172.21.15.116:0/3436894327 wait complete. 2024-03-04T07:51:40.263 INFO:tasks.cephfs.filesystem:scrub status for tag:793c6118-b9c4-40c3-8d77-09db7d6a8a79 - {'path': '/', 'tag': '793c6118-b9c4-40c3-8d77-09db7d6a8a79', 'options': 'recursive,force'} 2024-03-04T07:51:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:40 smithi116 ceph-mon[32150]: pgmap v728: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:51:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:40 smithi116 ceph-mon[32150]: scrub queued for path: / 2024-03-04T07:51:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:40 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:40 smithi116 ceph-mon[32150]: scrub summary: active paths [/] 2024-03-04T07:51:41.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:40 smithi116 ceph-mon[32150]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:40 smithi152 ceph-mon[37154]: pgmap v728: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:51:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:40 smithi152 ceph-mon[37154]: scrub queued for path: / 2024-03-04T07:51:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:40 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:41.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:40 smithi152 ceph-mon[37154]: scrub summary: active paths [/] 2024-03-04T07:51:41.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:40 smithi152 ceph-mon[37154]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:41.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:40 smithi133 ceph-mon[36505]: pgmap v728: 97 pgs: 97 active+clean; 4.2 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-03-04T07:51:41.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:40 smithi133 ceph-mon[36505]: scrub queued for path: / 2024-03-04T07:51:41.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:40 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:41.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:40 smithi133 ceph-mon[36505]: scrub summary: active paths [/] 2024-03-04T07:51:41.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:40 smithi133 ceph-mon[36505]: scrub summary: idle+waiting paths [/] 2024-03-04T07:51:43.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:42 smithi116 ceph-mon[32150]: pgmap v729: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:51:43.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:42 smithi116 ceph-mon[32150]: scrub summary: idle 2024-03-04T07:51:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:42 smithi152 ceph-mon[37154]: pgmap v729: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:51:43.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:42 smithi152 ceph-mon[37154]: scrub summary: idle 2024-03-04T07:51:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:42 smithi133 ceph-mon[36505]: pgmap v729: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 23 KiB/s wr, 13 op/s 2024-03-04T07:51:43.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:42 smithi133 ceph-mon[36505]: scrub summary: idle 2024-03-04T07:51:44.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 -- 172.21.15.116:0/154073352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 msgr2=0x7f22741092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 --2- 172.21.15.116:0/154073352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f22741092a0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2260009ca0 tx=0x7f226002c100 comp rx=0 tx=0).stop 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 -- 172.21.15.116:0/154073352 shutdown_connections 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 --2- 172.21.15.116:0/154073352 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22741097e0 0x7f2274110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 --2- 172.21.15.116:0/154073352 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f22741092a0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 --2- 172.21.15.116:0/154073352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2274107c30 0x7f2274108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 -- 172.21.15.116:0/154073352 >> 172.21.15.116:0/154073352 conn(0x7f2274076be0 msgr2=0x7f2274079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 -- 172.21.15.116:0/154073352 shutdown_connections 2024-03-04T07:51:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.735+0000 7f227a200640 1 -- 172.21.15.116:0/154073352 wait complete. 2024-03-04T07:51:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.736+0000 7f227a200640 1 Processor -- start 2024-03-04T07:51:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.736+0000 7f227a200640 1 -- start start 2024-03-04T07:51:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f227a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2274107c30 0x7f227419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f227a200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f227419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f227a200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22741097e0 0x7f22741a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f227a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22741138b0 con 0x7f2274107c30 2024-03-04T07:51:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f227a200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2274113730 con 0x7f2274108e20 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f227a200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2274113a30 con 0x7f22741097e0 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f227419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2273400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2274107c30 0x7f227419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f227419e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41922/0 (socket says 172.21.15.116:41922) 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 -- 172.21.15.116:0/2723907179 learned_addr learned my addr 172.21.15.116:0/2723907179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2273e00640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22741097e0 0x7f22741a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 -- 172.21.15.116:0/2723907179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22741097e0 msgr2=0x7f22741a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22741097e0 0x7f22741a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 -- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2274107c30 msgr2=0x7f227419de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2274107c30 0x7f227419de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.737+0000 7f2272a00640 1 -- 172.21.15.116:0/2723907179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22741a5ab0 con 0x7f2274108e20 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.738+0000 7f2272a00640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f227419e3d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2260009dc0 tx=0x7f2260035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.738+0000 7f226be00640 1 -- 172.21.15.116:0/2723907179 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260039040 con 0x7f2274108e20 2024-03-04T07:51:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.738+0000 7f226be00640 1 -- 172.21.15.116:0/2723907179 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2260044070 con 0x7f2274108e20 2024-03-04T07:51:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.738+0000 7f226be00640 1 -- 172.21.15.116:0/2723907179 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22600044a0 con 0x7f2274108e20 2024-03-04T07:51:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.738+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22741a5d40 con 0x7f2274108e20 2024-03-04T07:51:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.738+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22741a6180 con 0x7f2274108e20 2024-03-04T07:51:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.741+0000 7f227cf80740 1 Processor -- start 2024-03-04T07:51:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.741+0000 7f227cf80740 1 -- start start 2024-03-04T07:51:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f226be00640 1 -- 172.21.15.116:0/2723907179 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f2260004640 con 0x7f2274108e20 2024-03-04T07:51:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f227cf80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d348ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f227cf80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d348fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f227cf80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d34903b0 0x5606d3492840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f227cf80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d33f0370 con 0x5606d34903b0 2024-03-04T07:51:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f227cf80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d3399550 con 0x5606d348a850 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f227cf80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d33ec410 con 0x5606d348b240 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d348ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d348ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41928/0 (socket says 172.21.15.116:41928) 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 -- 172.21.15.116:0/816219606 learned_addr learned my addr 172.21.15.116:0/816219606 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2272a00640 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d34903b0 0x5606d3492840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273400640 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d348fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 -- 172.21.15.116:0/816219606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 msgr2=0x5606d348fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d348fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 -- 172.21.15.116:0/816219606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d34903b0 msgr2=0x5606d3492840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d34903b0 0x5606d3492840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2273e00640 1 -- 172.21.15.116:0/816219606 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606d33ad3c0 con 0x5606d348a850 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.742+0000 7f2272a00640 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d34903b0 0x5606d3492840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.743+0000 7f2273e00640 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d348ac70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f226c009ca0 tx=0x7f226c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f0cca1dc5210e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.743+0000 7f225fe00640 1 -- 172.21.15.116:0/816219606 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c01e070 con 0x5606d348a850 2024-03-04T07:51:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.743+0000 7f225fe00640 1 -- 172.21.15.116:0/816219606 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f226c009e50 con 0x5606d348a850 2024-03-04T07:51:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.743+0000 7f225fe00640 1 -- 172.21.15.116:0/816219606 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c00ea60 con 0x5606d348a850 2024-03-04T07:51:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.743+0000 7f226be00640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22580a1ea0 0x7f22580a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.743+0000 7f2273400640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22580a1ea0 0x7f22580a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.743+0000 7f226be00640 1 -- 172.21.15.116:0/2723907179 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f22600f03e0 con 0x7f2274108e20 2024-03-04T07:51:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 -- 172.21.15.116:0/816219606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 msgr2=0x5606d348ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d348ac70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f226c009ca0 tx=0x7f226c011810 comp rx=0 tx=0).stop 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 -- 172.21.15.116:0/816219606 shutdown_connections 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d34903b0 0x5606d3492840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d348fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 --2- 172.21.15.116:0/816219606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d348ac70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 -- 172.21.15.116:0/816219606 >> 172.21.15.116:0/816219606 conn(0x5606d3485dc0 msgr2=0x5606d34861c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f2273400640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22580a1ea0 0x7f22580a4360 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x5606d348b640 tx=0x7f2264005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 -- 172.21.15.116:0/816219606 shutdown_connections 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.744+0000 7f227cf80740 1 -- 172.21.15.116:0/816219606 wait complete. 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 -- start start 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 Processor -- start 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3494be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d3495120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d349c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d3493360 con 0x5606d348b240 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d33ac9b0 con 0x5606d348a850 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d34934e0 con 0x5606d34903b0 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f227cf80740 10 client.0 _ll_register_callbacks cb 0x5606d33e8340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.745+0000 7f2273400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3494be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3494be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41940/0 (socket says 172.21.15.116:41940) 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2272a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d3495120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 -- 172.21.15.116:0/2280358202 learned_addr learned my addr 172.21.15.116:0/2280358202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273e00640 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d349c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 -- 172.21.15.116:0/2280358202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 msgr2=0x5606d349c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d349c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 -- 172.21.15.116:0/2280358202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 msgr2=0x5606d3495120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d3495120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 -- 172.21.15.116:0/2280358202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606d3105210 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f2273400640 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3494be0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2264004600 tx=0x7f2264004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865f3c4e1365adec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f225cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f225cc00640 1 -- 172.21.15.116:0/2280358202 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264009e60 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f226c009e50 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.746+0000 7f225cc00640 1 -- 172.21.15.116:0/2280358202 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f226401f630 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f225cc00640 1 -- 172.21.15.116:0/2280358202 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f226401f830 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 10 client.28142 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5606d31d1620 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f225cc00640 1 -- 172.21.15.116:0/2280358202 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f226c009e50 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f225cc00640 1 -- 172.21.15.116:0/2280358202 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2264027070 con 0x5606d348a850 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 10 client.28142 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 10 client.28142 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 20 client.28142 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34a09f0 0x5606d34a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 4 client.28142 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.747+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606d33fd8b0 con 0x5606d34a09f0 2024-03-04T07:51:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.748+0000 7f2272a00640 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34a09f0 0x5606d34a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.748+0000 7f2272a00640 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34a09f0 0x5606d34a2e20 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66117fcff5781311 server_cookie=8f6a4bbdd97e7d93 in_seq=0 out_seq=0 2024-03-04T07:51:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.748+0000 7f225cc00640 10 client.28142 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.749+0000 7f225cc00640 1 -- 172.21.15.116:0/2280358202 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5606d33fd8b0 con 0x5606d34a09f0 2024-03-04T07:51:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.749+0000 7f225cc00640 10 client.28142 handle_command_reply: tid=0 2024-03-04T07:51:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.749+0000 7f227cf80740 1 client.28142 shutdown 2024-03-04T07:51:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.749+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 msgr2=0x5606d3494be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.749+0000 7f227cf80740 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3494be0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2264004600 tx=0x7f2264004730 comp rx=0 tx=0).stop 2024-03-04T07:51:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 shutdown_connections 2024-03-04T07:51:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34a09f0 0x5606d34a2e20 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d349c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d3495120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 --2- 172.21.15.116:0/2280358202 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3494be0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f225cc00640 0 client.28142 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 >> 172.21.15.116:0/2280358202 conn(0x5606d3485dc0 msgr2=0x5606d3484fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 shutdown_connections 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 1 -- 172.21.15.116:0/2280358202 wait complete. 2024-03-04T07:51:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.750+0000 7f227cf80740 20 client.28142 trim_cache size 0 max 16384 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 Processor -- start 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 -- start start 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3495080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d349c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d3489730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22741ad020 con 0x5606d348b240 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22741af600 con 0x5606d348a850 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f227cf80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22741ad620 con 0x5606d34903b0 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f2272a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d349c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.765+0000 7f2273400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3495080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3495080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41942/0 (socket says 172.21.15.116:41942) 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 -- 172.21.15.116:0/2976935132 learned_addr learned my addr 172.21.15.116:0/2976935132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273e00640 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d3489730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 -- 172.21.15.116:0/2976935132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 msgr2=0x5606d3489730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d3489730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 -- 172.21.15.116:0/2976935132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 msgr2=0x5606d349c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d349c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 -- 172.21.15.116:0/2976935132 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606d348ac30 con 0x5606d348a850 2024-03-04T07:51:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f2273400640 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3495080 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f2264029d00 tx=0x7f2264009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5198e008d0d9487 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f225f400640 1 -- 172.21.15.116:0/2976935132 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264038030 con 0x5606d348a850 2024-03-04T07:51:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f225f400640 1 -- 172.21.15.116:0/2976935132 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2264009e60 con 0x5606d348a850 2024-03-04T07:51:44.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.766+0000 7f225f400640 1 -- 172.21.15.116:0/2976935132 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264029950 con 0x5606d348a850 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 -- 172.21.15.116:0/2976935132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 msgr2=0x5606d3495080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3495080 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f2264029d00 tx=0x7f2264009c00 comp rx=0 tx=0).stop 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 -- 172.21.15.116:0/2976935132 shutdown_connections 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d34903b0 0x5606d3489730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348b240 0x5606d349c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 --2- 172.21.15.116:0/2976935132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d348a850 0x5606d3495080 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 -- 172.21.15.116:0/2976935132 >> 172.21.15.116:0/2976935132 conn(0x5606d3485dc0 msgr2=0x5606d3489020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 -- 172.21.15.116:0/2976935132 shutdown_connections 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 -- 172.21.15.116:0/2976935132 wait complete. 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.767+0000 7f227cf80740 1 -- start start 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 1 Processor -- start 2024-03-04T07:51:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348a850 0x5606d3499cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d349a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d34903b0 0x5606d349b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22741adc80 con 0x5606d348a850 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22741aefa0 con 0x5606d34903b0 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22741ae2e0 con 0x5606d348b240 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f227cf80740 10 client.0 _ll_register_callbacks cb 0x7f2264009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f2273e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d34903b0 0x5606d349b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f2273400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348a850 0x5606d3499cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f2273e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d34903b0 0x5606d349b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:41946/0 (socket says 172.21.15.116:41946) 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f2273e00640 1 -- 172.21.15.116:0/551721248 learned_addr learned my addr 172.21.15.116:0/551721248 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.768+0000 7f2272a00640 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d349a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f2273e00640 1 -- 172.21.15.116:0/551721248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 msgr2=0x5606d349a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f2273e00640 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d349a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f2273e00640 1 -- 172.21.15.116:0/551721248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348a850 msgr2=0x5606d3499cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f2273e00640 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348a850 0x5606d3499cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f2273e00640 1 -- 172.21.15.116:0/551721248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606d349b7e0 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f2273400640 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348a850 0x5606d3499cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f2273e00640 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d34903b0 0x5606d349b1f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f226c009ca0 tx=0x7f226c0050b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9e890969f398a7a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f225cc00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f225cc00640 1 -- 172.21.15.116:0/551721248 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c01e040 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f225cc00640 1 -- 172.21.15.116:0/551721248 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f226c0035f0 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2264009e60 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f225cc00640 1 -- 172.21.15.116:0/551721248 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f226c0037f0 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.769+0000 7f227cf80740 10 client.28148 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5606d34fc7b0 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f225cc00640 1 -- 172.21.15.116:0/551721248 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2264009e60 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f225cc00640 1 -- 172.21.15.116:0/551721248 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f226c024a00 con 0x5606d34903b0 2024-03-04T07:51:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f227cf80740 10 client.28148 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f227cf80740 10 client.28148 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f227cf80740 20 client.28148 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f227cf80740 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34fcfa0 0x5606d34ff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f227cf80740 4 client.28148 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5606d33f7c00 con 0x5606d34fcfa0 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.770+0000 7f2273400640 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34fcfa0 0x5606d34ff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.771+0000 7f225cc00640 10 client.28148 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.771+0000 7f2273400640 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34fcfa0 0x5606d34ff3c0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7007a7becd0ef55f server_cookie=5fb03d76d8671226 in_seq=0 out_seq=0 2024-03-04T07:51:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f225cc00640 1 -- 172.21.15.116:0/551721248 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5606d33f7c00 con 0x5606d34fcfa0 2024-03-04T07:51:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f225cc00640 10 client.28148 handle_command_reply: tid=0 2024-03-04T07:51:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 client.28148 shutdown 2024-03-04T07:51:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d34903b0 msgr2=0x5606d349b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d34903b0 0x5606d349b1f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f226c009ca0 tx=0x7f226c0050b0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 shutdown_connections 2024-03-04T07:51:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5606d34fcfa0 0x5606d34ff3c0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5606d34903b0 0x5606d349b1f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5606d348b240 0x5606d349a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 --2- 172.21.15.116:0/551721248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606d348a850 0x5606d3499cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f225cc00640 0 client.28148 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 >> 172.21.15.116:0/551721248 conn(0x5606d3485dc0 msgr2=0x5606d34898b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 shutdown_connections 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 1 -- 172.21.15.116:0/551721248 wait complete. 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.786+0000 7f227cf80740 20 client.28148 trim_cache size 0 max 16384 2024-03-04T07:51:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:51:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:51:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:51:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:51:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22580a1ea0 msgr2=0x7f22580a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22580a1ea0 0x7f22580a4360 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x5606d348b640 tx=0x7f2264005f90 comp rx=0 tx=0).stop 2024-03-04T07:51:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 msgr2=0x7f227419e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f227419e3d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2260009dc0 tx=0x7f2260035d90 comp rx=0 tx=0).stop 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 shutdown_connections 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f22580a1ea0 0x7f22580a4360 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22741097e0 0x7f22741a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2274108e20 0x7f227419e3d0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 --2- 172.21.15.116:0/2723907179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2274107c30 0x7f227419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 >> 172.21.15.116:0/2723907179 conn(0x7f2274076be0 msgr2=0x7f227410c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 shutdown_connections 2024-03-04T07:51:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:44.787+0000 7f227a200640 1 -- 172.21.15.116:0/2723907179 wait complete. 2024-03-04T07:51:45.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:44 smithi116 ceph-mon[32150]: pgmap v730: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:51:45.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:44 smithi152 ceph-mon[37154]: pgmap v730: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:51:45.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:44 smithi133 ceph-mon[36505]: pgmap v730: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-04T07:51:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:46 smithi116 ceph-mon[32150]: pgmap v731: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-04T07:51:47.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:46 smithi152 ceph-mon[37154]: pgmap v731: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-04T07:51:47.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:46 smithi133 ceph-mon[36505]: pgmap v731: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-04T07:51:49.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:48 smithi116 ceph-mon[32150]: pgmap v732: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:51:49.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:48 smithi152 ceph-mon[37154]: pgmap v732: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:51:49.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:48 smithi133 ceph-mon[36505]: pgmap v732: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:51:51.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:50 smithi116 ceph-mon[32150]: pgmap v733: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:51:51.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:50 smithi152 ceph-mon[37154]: pgmap v733: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:51:51.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:50 smithi133 ceph-mon[36505]: pgmap v733: 97 pgs: 97 active+clean; 4.3 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 11 op/s 2024-03-04T07:51:53.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:52 smithi116 ceph-mon[32150]: pgmap v734: 97 pgs: 97 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:51:53.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:52 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:51:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:52 smithi152 ceph-mon[37154]: pgmap v734: 97 pgs: 97 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:51:53.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:52 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:51:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:52 smithi133 ceph-mon[36505]: pgmap v734: 97 pgs: 97 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 21 KiB/s wr, 13 op/s 2024-03-04T07:51:53.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:52 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T07:51:54.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:51:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:51:54.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:53 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:51:54.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:51:54.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:53 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T07:51:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T07:51:54.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:53 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' 2024-03-04T07:51:55.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:54 smithi116 ceph-mon[32150]: pgmap v735: 97 pgs: 97 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:55.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:54 smithi152 ceph-mon[37154]: pgmap v735: 97 pgs: 97 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:55.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:54 smithi133 ceph-mon[36505]: pgmap v735: 97 pgs: 97 active+clean; 4.4 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:51:56.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:56 smithi133 ceph-mon[36505]: pgmap v736: 97 pgs: 97 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:51:56.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:56 smithi116 ceph-mon[32150]: pgmap v736: 97 pgs: 97 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:51:56.521 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:56 smithi152 ceph-mon[37154]: pgmap v736: 97 pgs: 97 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-04T07:51:57.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.926+0000 7f99d3600640 1 -- 172.21.15.116:0/1004424870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 msgr2=0x7f99cc0763b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.926+0000 7f99d3600640 1 --2- 172.21.15.116:0/1004424870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc0763b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009ca0 tx=0x7f99bc02c100 comp rx=0 tx=0).stop 2024-03-04T07:51:57.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.927+0000 7f99d3600640 1 -- 172.21.15.116:0/1004424870 shutdown_connections 2024-03-04T07:51:57.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.927+0000 7f99d3600640 1 --2- 172.21.15.116:0/1004424870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99cc077b70 0x7f99cc078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.927+0000 7f99d3600640 1 --2- 172.21.15.116:0/1004424870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99cc0771d0 0x7f99cc077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.927+0000 7f99d3600640 1 --2- 172.21.15.116:0/1004424870 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc0763b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.927+0000 7f99d3600640 1 -- 172.21.15.116:0/1004424870 >> 172.21.15.116:0/1004424870 conn(0x7f99cc0fedc0 msgr2=0x7f99cc1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:57.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.927+0000 7f99d3600640 1 -- 172.21.15.116:0/1004424870 shutdown_connections 2024-03-04T07:51:57.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.927+0000 7f99d3600640 1 -- 172.21.15.116:0/1004424870 wait complete. 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 Processor -- start 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 -- start start 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99cc0771d0 0x7f99cc19e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99cc077b70 0x7f99cc1a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1138c0 con 0x7f99cc0771d0 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc113740 con 0x7f99cc075f90 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.928+0000 7f99d3600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc113a40 con 0x7f99cc077b70 2024-03-04T07:51:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc19de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56180/0 (socket says 172.21.15.116:56180) 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 -- 172.21.15.116:0/12549305 learned_addr learned my addr 172.21.15.116:0/12549305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d1600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99cc077b70 0x7f99cc1a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99cbe00640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99cc0771d0 0x7f99cc19e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 -- 172.21.15.116:0/12549305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99cc077b70 msgr2=0x7f99cc1a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99cc077b70 0x7f99cc1a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 -- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99cc0771d0 msgr2=0x7f99cc19e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99cc0771d0 0x7f99cc19e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 -- 172.21.15.116:0/12549305 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc1a5a40 con 0x7f99cc075f90 2024-03-04T07:51:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99d0c00640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc19de20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0325f0 tx=0x7f99bc035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.929+0000 7f99c9600640 1 -- 172.21.15.116:0/12549305 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc039040 con 0x7f99cc075f90 2024-03-04T07:51:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.930+0000 7f99c9600640 1 -- 172.21.15.116:0/12549305 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99bc044070 con 0x7f99cc075f90 2024-03-04T07:51:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.930+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc1a5cd0 con 0x7f99cc075f90 2024-03-04T07:51:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.930+0000 7f99c9600640 1 -- 172.21.15.116:0/12549305 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc004540 con 0x7f99cc075f90 2024-03-04T07:51:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.930+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc1a6190 con 0x7f99cc075f90 2024-03-04T07:51:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.934+0000 7f99d6207740 1 Processor -- start 2024-03-04T07:51:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.934+0000 7f99c9600640 1 -- 172.21.15.116:0/12549305 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f99bc032890 con 0x7f99cc075f90 2024-03-04T07:51:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.934+0000 7f99d6207740 1 -- start start 2024-03-04T07:51:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.934+0000 7f99d6207740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12d3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d6207740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12d8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d6207740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12db840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d6207740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638e1239370 con 0x5638e12d3850 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d6207740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638e11e2550 con 0x5638e12d4240 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d6207740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638e1235410 con 0x5638e12d93b0 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12d3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d0c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12d8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12d3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56910/0 (socket says 172.21.15.116:56910) 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d0c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12d8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:56190/0 (socket says 172.21.15.116:56190) 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 -- 172.21.15.116:0/3084983342 learned_addr learned my addr 172.21.15.116:0/3084983342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99cbe00640 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12db840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 -- 172.21.15.116:0/3084983342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 msgr2=0x5638e12db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12db840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 -- 172.21.15.116:0/3084983342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 msgr2=0x5638e12d8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12d8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d1600640 1 -- 172.21.15.116:0/3084983342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638e11f63c0 con 0x5638e12d3850 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.935+0000 7f99d0c00640 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12d8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.936+0000 7f99d1600640 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12d3c70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009c10 tx=0x7f99c40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732e7e8a1634bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.936+0000 7f99c9600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f99b00a1ea0 0x7f99b00a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.936+0000 7f99c1c00640 1 -- 172.21.15.116:0/3084983342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c4012450 con 0x5638e12d3850 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.936+0000 7f99c9600640 1 -- 172.21.15.116:0/12549305 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f99bc0f0470 con 0x7f99cc075f90 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.936+0000 7f99d0c00640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f99b00a1ea0 0x7f99b00a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.936+0000 7f99c1c00640 1 -- 172.21.15.116:0/3084983342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99c4012ad0 con 0x5638e12d3850 2024-03-04T07:51:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.936+0000 7f99c1c00640 1 -- 172.21.15.116:0/3084983342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c400fb60 con 0x5638e12d3850 2024-03-04T07:51:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 -- 172.21.15.116:0/3084983342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 msgr2=0x5638e12d3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d0c00640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f99b00a1ea0 0x7f99b00a4360 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006010 tx=0x7f99bc0f3ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12d3c70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009c10 tx=0x7f99c40117b0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 -- 172.21.15.116:0/3084983342 shutdown_connections 2024-03-04T07:51:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12db840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12d8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 --2- 172.21.15.116:0/3084983342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12d3c70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 -- 172.21.15.116:0/3084983342 >> 172.21.15.116:0/3084983342 conn(0x5638e12cedc0 msgr2=0x5638e12cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 -- 172.21.15.116:0/3084983342 shutdown_connections 2024-03-04T07:51:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.937+0000 7f99d6207740 1 -- 172.21.15.116:0/3084983342 wait complete. 2024-03-04T07:51:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 -- start start 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 Processor -- start 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12ddb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d4240 0x5638e12de0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d93b0 0x5638e12e5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638e12dc360 con 0x5638e12d93b0 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638e11f59b0 con 0x5638e12d3850 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638e12dc4e0 con 0x5638e12d4240 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.938+0000 7f99d6207740 10 client.0 _ll_register_callbacks cb 0x5638e1231340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99d1600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12ddb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99d1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d93b0 0x5638e12e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d4240 0x5638e12de0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d4240 0x5638e12de0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.116:44868/0 (socket says 172.21.15.116:44868) 2024-03-04T07:51:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99d1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d93b0 0x5638e12e5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56916/0 (socket says 172.21.15.116:56916) 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 -- 172.21.15.116:0/2689141013 learned_addr learned my addr 172.21.15.116:0/2689141013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 -- 172.21.15.116:0/2689141013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 msgr2=0x5638e12ddb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12ddb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 -- 172.21.15.116:0/2689141013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d93b0 msgr2=0x5638e12e5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d93b0 0x5638e12e5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 -- 172.21.15.116:0/2689141013 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638e0f4e210 con 0x5638e12d4240 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99d1600640 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d93b0 0x5638e12e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99cbe00640 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d4240 0x5638e12de0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99b400bec0 tx=0x7f99b4012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33f30346deed001c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.939+0000 7f99ba000640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-04T07:51:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99ba000640 1 -- 172.21.15.116:0/2689141013 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b401f060 con 0x5638e12d4240 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99ba000640 1 -- 172.21.15.116:0/2689141013 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f99b40108a0 con 0x5638e12d4240 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99c4012ad0 con 0x5638e12d4240 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99ba000640 1 -- 172.21.15.116:0/2689141013 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99b4010aa0 con 0x5638e12d4240 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99d6207740 10 client.28006 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5638e101a620 con 0x5638e12d4240 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99ba000640 1 -- 172.21.15.116:0/2689141013 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99c4012ad0 con 0x5638e12d4240 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99ba000640 1 -- 172.21.15.116:0/2689141013 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f99b4024070 con 0x5638e12d4240 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99d6207740 10 client.28006 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99d6207740 10 client.28006 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99d6207740 20 client.28006 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.940+0000 7f99d6207740 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e12e9b40 0x5638e12ebf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.941+0000 7f99d6207740 4 client.28006 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.941+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638e12468b0 con 0x5638e12e9b40 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.941+0000 7f99d1600640 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e12e9b40 0x5638e12ebf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.941+0000 7f99d1600640 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e12e9b40 0x5638e12ebf60 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89b4cc6fd2e27f86 server_cookie=1f7de4da0680589e in_seq=0 out_seq=0 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.941+0000 7f99ba000640 10 client.28006 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99ba000640 1 -- 172.21.15.116:0/2689141013 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5638e12468b0 con 0x5638e12e9b40 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99ba000640 10 client.28006 handle_command_reply: tid=0 2024-03-04T07:51:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 client.28006 shutdown 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d4240 msgr2=0x5638e12de0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d4240 0x5638e12de0d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99b400bec0 tx=0x7f99b4012820 comp rx=0 tx=0).stop 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 shutdown_connections 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e12e9b40 0x5638e12ebf60 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d93b0 0x5638e12e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d4240 0x5638e12de0d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99ba000640 0 client.28006 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 --2- 172.21.15.116:0/2689141013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12ddb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 >> 172.21.15.116:0/2689141013 conn(0x5638e12cedc0 msgr2=0x5638e12cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 shutdown_connections 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.942+0000 7f99d6207740 1 -- 172.21.15.116:0/2689141013 wait complete. 2024-03-04T07:51:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.943+0000 7f99d6207740 20 client.28006 trim_cache size 0 max 16384 2024-03-04T07:51:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.957+0000 7f99d6207740 1 Processor -- start 2024-03-04T07:51:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d6207740 1 -- start start 2024-03-04T07:51:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d6207740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d6207740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d4240 0x5638e12e50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d6207740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12d2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d6207740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1aceb0 con 0x5638e12d4240 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d6207740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1af490 con 0x5638e12d3850 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d6207740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1ad4b0 con 0x5638e12d93b0 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d4240 0x5638e12e50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99d1600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d4240 0x5638e12e50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56922/0 (socket says 172.21.15.116:56922) 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 -- 172.21.15.116:0/476018105 learned_addr learned my addr 172.21.15.116:0/476018105 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 -- 172.21.15.116:0/476018105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 msgr2=0x5638e12d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 --2- 172.21.15.116:0/476018105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12d2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 -- 172.21.15.116:0/476018105 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 msgr2=0x5638e12d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 --2- 172.21.15.116:0/476018105 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.958+0000 7f99cbe00640 1 -- 172.21.15.116:0/476018105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638e12d3640 con 0x5638e12d4240 2024-03-04T07:51:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.959+0000 7f99cbe00640 1 --2- 172.21.15.116:0/476018105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d4240 0x5638e12e50d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f99b400ba10 tx=0x7f99b4005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e70471ea100756 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.959+0000 7f99c1200640 1 -- 172.21.15.116:0/476018105 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4033070 con 0x5638e12d4240 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.959+0000 7f99c1200640 1 -- 172.21.15.116:0/476018105 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99b4010e40 con 0x5638e12d4240 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.959+0000 7f99c1200640 1 -- 172.21.15.116:0/476018105 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b402a070 con 0x5638e12d4240 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 -- 172.21.15.116:0/476018105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d4240 msgr2=0x5638e12e50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 --2- 172.21.15.116:0/476018105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d4240 0x5638e12e50d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f99b400ba10 tx=0x7f99b4005970 comp rx=0 tx=0).stop 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 -- 172.21.15.116:0/476018105 shutdown_connections 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 --2- 172.21.15.116:0/476018105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12d2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 --2- 172.21.15.116:0/476018105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d4240 0x5638e12e50d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 --2- 172.21.15.116:0/476018105 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d3850 0x5638e12d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 -- 172.21.15.116:0/476018105 >> 172.21.15.116:0/476018105 conn(0x5638e12cedc0 msgr2=0x5638e12d5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 -- 172.21.15.116:0/476018105 shutdown_connections 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 -- 172.21.15.116:0/476018105 wait complete. 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 -- start start 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.960+0000 7f99d6207740 1 Processor -- start 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d6207740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12e31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d6207740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12e36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d6207740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12e4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d6207740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1adb10 con 0x5638e12d3850 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d6207740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1aee30 con 0x5638e12d4240 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d6207740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1ae170 con 0x5638e12d93b0 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d6207740 10 client.0 _ll_register_callbacks cb 0x7f99b400b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:51:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12e31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99cbe00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12e36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12e31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56928/0 (socket says 172.21.15.116:56928) 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d1600640 1 -- 172.21.15.116:0/3959076886 learned_addr learned my addr 172.21.15.116:0/3959076886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d1600640 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99cbe00640 1 -- 172.21.15.116:0/3959076886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 msgr2=0x5638e12e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99cbe00640 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12e4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99cbe00640 1 -- 172.21.15.116:0/3959076886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 msgr2=0x5638e12e31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99cbe00640 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12e31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99cbe00640 1 -- 172.21.15.116:0/3959076886 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638e12e4c40 con 0x5638e12d4240 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.961+0000 7f99d1600640 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99cbe00640 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12e36e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f99b4031040 tx=0x7f99b401a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c48343ebf9b6c048 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99ba000640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99ba000640 1 -- 172.21.15.116:0/3959076886 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b402a070 con 0x5638e12d4240 2024-03-04T07:51:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99b4010e40 con 0x5638e12d4240 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99ba000640 1 -- 172.21.15.116:0/3959076886 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f99b4025520 con 0x5638e12d4240 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99ba000640 1 -- 172.21.15.116:0/3959076886 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99b4033040 con 0x5638e12d4240 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 10 client.28166 fetch_fsmap learned FSMap version 19 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x5638e1345cc0 con 0x5638e12d4240 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99ba000640 1 -- 172.21.15.116:0/3959076886 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99b4010e40 con 0x5638e12d4240 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99ba000640 1 -- 172.21.15.116:0/3959076886 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f99b4036020 con 0x5638e12d4240 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 10 client.28166 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 10 client.28166 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 20 client.28166 populate_metadata read hostname 'smithi116' 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e1346560 0x5638e1348980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 4 client.28166 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.962+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5638e1240c00 con 0x5638e1346560 2024-03-04T07:51:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.963+0000 7f99d1600640 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e1346560 0x5638e1348980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:51:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.963+0000 7f99d1600640 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e1346560 0x5638e1348980 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c8eb88729b0c7be server_cookie=d7d2bdc43a09d62c in_seq=0 out_seq=0 2024-03-04T07:51:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.963+0000 7f99ba000640 10 client.28166 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99ba000640 1 -- 172.21.15.116:0/3959076886 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5638e1240c00 con 0x5638e1346560 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99ba000640 10 client.28166 handle_command_reply: tid=0 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 client.28166 shutdown 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 msgr2=0x5638e12e36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12e36e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f99b4031040 tx=0x7f99b401a680 comp rx=0 tx=0).stop 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 shutdown_connections 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x5638e1346560 0x5638e1348980 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638e12d93b0 0x5638e12e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x5638e12d4240 0x5638e12e36e0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 --2- 172.21.15.116:0/3959076886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5638e12d3850 0x5638e12e31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99ba000640 0 client.28166 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:51:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 >> 172.21.15.116:0/3959076886 conn(0x5638e12cedc0 msgr2=0x5638e12dd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 shutdown_connections 2024-03-04T07:51:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 1 -- 172.21.15.116:0/3959076886 wait complete. 2024-03-04T07:51:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d6207740 20 client.28166 trim_cache size 0 max 16384 2024-03-04T07:51:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-04T07:51:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-04T07:51:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-04T07:51:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-04T07:51:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f99b00a1ea0 msgr2=0x7f99b00a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d3600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f99b00a1ea0 0x7f99b00a4360 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f99bc006010 tx=0x7f99bc0f3ef0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 msgr2=0x7f99cc19de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.982+0000 7f99d3600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc19de20 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0325f0 tx=0x7f99bc035d10 comp rx=0 tx=0).stop 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 shutdown_connections 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f99b00a1ea0 0x7f99b00a4360 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99cc077b70 0x7f99cc1a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99cc0771d0 0x7f99cc19e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 --2- 172.21.15.116:0/12549305 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99cc075f90 0x7f99cc19de20 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 >> 172.21.15.116:0/12549305 conn(0x7f99cc0fedc0 msgr2=0x7f99cc1008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 shutdown_connections 2024-03-04T07:51:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-04T07:51:57.983+0000 7f99d3600640 1 -- 172.21.15.116:0/12549305 wait complete. 2024-03-04T07:51:59.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:51:58 smithi116 ceph-mon[32150]: pgmap v737: 97 pgs: 97 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:51:59.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:51:58 smithi152 ceph-mon[37154]: pgmap v737: 97 pgs: 97 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:51:59.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:51:58 smithi133 ceph-mon[36505]: pgmap v737: 97 pgs: 97 active+clean; 4.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:52:00.360 INFO:tasks.workunit.client.0.smithi116.stdout:561/600: open fd = 3 2024-03-04T07:52:00.360 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.360 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.360 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:562/600: open fd = 3 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:563/600: open fd = 3 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:564/600: open fd = 3 2024-03-04T07:52:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:565/600: open fd = 3 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:566/600: open fd = 3 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:567/600: open fd = 3 2024-03-04T07:52:00.362 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:568/600: open fd = 3 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:569/600: open fd = 3 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:570/600: open fd = 3 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.363 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:571/600: open fd = 3 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:572/600: open fd = 3 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:573/600: open fd = 3 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:574/600: open fd = 3 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.364 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:575/600: open fd = 3 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:576/600: open fd = 3 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:577/600: open fd = 3 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.365 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:578/600: open fd = 3 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:579/600: open fd = 3 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:580/600: open fd = 3 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:581/600: open fd = 3 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.366 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:582/600: open fd = 3 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:583/600: open fd = 3 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:584/600: open fd = 3 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.367 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:585/600: open fd = 3 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:586/600: open fd = 3 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:587/600: open fd = 3 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.368 INFO:tasks.workunit.client.0.smithi116.stdout:588/600: open fd = 3 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:589/600: open fd = 3 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:590/600: open fd = 3 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:591/600: open fd = 3 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.369 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:592/600: open fd = 3 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:593/600: open fd = 3 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:594/600: open fd = 3 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.370 INFO:tasks.workunit.client.0.smithi116.stdout:595/600: open fd = 3 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:596/600: open fd = 3 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:597/600: open fd = 3 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:598/600: open fd = 3 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.371 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:599/600: open fd = 3 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:600/600: open fd = 3 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:write ret = 32 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:pread ret = 64 2024-03-04T07:52:00.372 INFO:tasks.workunit.client.0.smithi116.stdout:OK 2024-03-04T07:52:00.373 INFO:tasks.workunit.client.0.smithi116.stderr:+ echo OK 2024-03-04T07:52:00.373 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-04T07:52:00.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-04T07:52:00.527 INFO:tasks.workunit:Stopping ['fs/test_o_trunc.sh'] on client.0... 2024-03-04T07:52:00.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-04T07:52:01.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:00 smithi116 ceph-mon[32150]: pgmap v738: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:52:01.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:00 smithi152 ceph-mon[37154]: pgmap v738: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:52:01.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:00 smithi133 ceph-mon[36505]: pgmap v738: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-03-04T07:52:01.516 DEBUG:teuthology.parallel:result is None 2024-03-04T07:52:01.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-04T07:52:01.613 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-04T07:52:01.613 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-04T07:52:01.623 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T07:52:01.648 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-04T07:52:01.658 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-04T07:52:03.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:02 smithi116 ceph-mon[32150]: pgmap v739: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:52:03.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:02 smithi152 ceph-mon[37154]: pgmap v739: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:52:03.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:02 smithi133 ceph-mon[36505]: pgmap v739: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-03-04T07:52:05.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:04 smithi116 ceph-mon[32150]: pgmap v740: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:52:05.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:04 smithi152 ceph-mon[37154]: pgmap v740: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:52:05.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:04 smithi133 ceph-mon[36505]: pgmap v740: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 9 op/s 2024-03-04T07:52:07.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:06 smithi116 ceph-mon[32150]: pgmap v741: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-03-04T07:52:07.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:06 smithi152 ceph-mon[37154]: pgmap v741: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-03-04T07:52:07.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:06 smithi133 ceph-mon[36505]: pgmap v741: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 KiB/s wr, 11 op/s 2024-03-04T07:52:09.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:08 smithi116 ceph-mon[32150]: pgmap v742: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2024-03-04T07:52:09.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:08 smithi152 ceph-mon[37154]: pgmap v742: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2024-03-04T07:52:09.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:08 smithi133 ceph-mon[36505]: pgmap v742: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2024-03-04T07:52:10.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T07:52:10.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T07:52:10.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 -- 172.21.15.116:0/1539312523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2681007b0 msgr2=0x7fa268100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 --2- 172.21.15.116:0/1539312523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2681007b0 0x7fa268100c10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa254009860 tx=0x7fa25402bb70 comp rx=0 tx=0).stop 2024-03-04T07:52:10.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 -- 172.21.15.116:0/1539312523 shutdown_connections 2024-03-04T07:52:10.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 --2- 172.21.15.116:0/1539312523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa268101150 0x7fa2681015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 --2- 172.21.15.116:0/1539312523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2681007b0 0x7fa268100c10 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 --2- 172.21.15.116:0/1539312523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2680ffe50 0x7fa268100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 -- 172.21.15.116:0/1539312523 >> 172.21.15.116:0/1539312523 conn(0x7fa2680fba70 msgr2=0x7fa2680fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:10.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 -- 172.21.15.116:0/1539312523 shutdown_connections 2024-03-04T07:52:10.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.490+0000 7fa26e000640 1 -- 172.21.15.116:0/1539312523 wait complete. 2024-03-04T07:52:10.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.491+0000 7fa26e000640 1 Processor -- start 2024-03-04T07:52:10.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.491+0000 7fa26e000640 1 -- start start 2024-03-04T07:52:10.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa26e000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2680ffe50 0x7fa268199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa26e000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2681007b0 0x7fa26819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa26e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa268101150 0x7fa2681a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa26e000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26810f450 con 0x7fa268101150 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa26e000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26810f2d0 con 0x7fa2680ffe50 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa26e000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26810f5d0 con 0x7fa2681007b0 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2680ffe50 0x7fa268199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa268101150 0x7fa2681a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2680ffe50 0x7fa268199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42154/0 (socket says 172.21.15.116:42154) 2024-03-04T07:52:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267400640 1 -- 172.21.15.116:0/691038739 learned_addr learned my addr 172.21.15.116:0/691038739 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa266a00640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2681007b0 0x7fa26819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267e00640 1 -- 172.21.15.116:0/691038739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2681007b0 msgr2=0x7fa26819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267e00640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2681007b0 0x7fa26819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267e00640 1 -- 172.21.15.116:0/691038739 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2680ffe50 msgr2=0x7fa268199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.492+0000 7fa267e00640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2680ffe50 0x7fa268199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa267e00640 1 -- 172.21.15.116:0/691038739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2681a1740 con 0x7fa268101150 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa267400640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2680ffe50 0x7fa268199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa267e00640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa268101150 0x7fa2681a10d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e940 tx=0x7fa26000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa25fe00640 1 -- 172.21.15.116:0/691038739 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260004d20 con 0x7fa268101150 2024-03-04T07:52:10.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2681a1a30 con 0x7fa268101150 2024-03-04T07:52:10.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa25fe00640 1 -- 172.21.15.116:0/691038739 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa260005020 con 0x7fa268101150 2024-03-04T07:52:10.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa25fe00640 1 -- 172.21.15.116:0/691038739 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600116d0 con 0x7fa268101150 2024-03-04T07:52:10.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.493+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2681a1f50 con 0x7fa268101150 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.496+0000 7fa270df1740 1 Processor -- start 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.496+0000 7fa25fe00640 1 -- 172.21.15.116:0/691038739 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa2600054d0 con 0x7fa268101150 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa270df1740 1 -- start start 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa270df1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb15f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa270df1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb1b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa270df1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb1b6b0 0x55724eb1db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa270df1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ea43090 con 0x55724eb1b6b0 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa270df1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ea7ad30 con 0x55724eb15b50 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa270df1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ea23a30 con 0x55724eb16540 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.497+0000 7fa266a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb15f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa266a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb15f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42162/0 (socket says 172.21.15.116:42162) 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa266a00640 1 -- 172.21.15.116:0/4058558966 learned_addr learned my addr 172.21.15.116:0/4058558966 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa267400640 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb1b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa266a00640 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb1b6b0 0x55724eb1db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa267400640 1 -- 172.21.15.116:0/4058558966 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 msgr2=0x55724eb15f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa267400640 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb15f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa267400640 1 -- 172.21.15.116:0/4058558966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb1b6b0 msgr2=0x55724eb1db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa267400640 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb1b6b0 0x55724eb1db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa267400640 1 -- 172.21.15.116:0/4058558966 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55724e8597d0 con 0x55724eb16540 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa25fe00640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa24c0a1d80 0x7fa24c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.498+0000 7fa267400640 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb1b040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa258011510 tx=0x7fa2580119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c011c0b09daf7d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.499+0000 7fa253e00640 1 -- 172.21.15.116:0/4058558966 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25801e070 con 0x55724eb16540 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.499+0000 7fa25fe00640 1 -- 172.21.15.116:0/691038739 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa2600ce170 con 0x7fa268101150 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.499+0000 7fa266a00640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa24c0a1d80 0x7fa24c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.499+0000 7fa253e00640 1 -- 172.21.15.116:0/4058558966 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa25800faa0 con 0x55724eb16540 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.499+0000 7fa253e00640 1 -- 172.21.15.116:0/4058558966 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258019b80 con 0x55724eb16540 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 -- 172.21.15.116:0/4058558966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 msgr2=0x55724eb1b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb1b040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa258011510 tx=0x7fa2580119d0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa266a00640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa24c0a1d80 0x7fa24c0a4240 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55724eb16da0 tx=0x7fa25402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 -- 172.21.15.116:0/4058558966 shutdown_connections 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb1b6b0 0x55724eb1db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb1b040 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 --2- 172.21.15.116:0/4058558966 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb15f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 -- 172.21.15.116:0/4058558966 >> 172.21.15.116:0/4058558966 conn(0x55724eb110c0 msgr2=0x55724eb114c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 -- 172.21.15.116:0/4058558966 shutdown_connections 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 -- 172.21.15.116:0/4058558966 wait complete. 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.500+0000 7fa270df1740 1 -- start start 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 1 Processor -- start 2024-03-04T07:52:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb1ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb20460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb274a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ea190c0 con 0x55724eb16540 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724ea76dd0 con 0x55724eb15b50 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55724eb1e660 con 0x55724eb1b6b0 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa270df1740 10 client.0 _ll_register_callbacks cb 0x55724ea36860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa266a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb20460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa267400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb1ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa267400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb1ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42172/0 (socket says 172.21.15.116:42172) 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.501+0000 7fa267400640 1 -- 172.21.15.116:0/2425689340 learned_addr learned my addr 172.21.15.116:0/2425689340 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa267e00640 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb274a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa267400640 1 -- 172.21.15.116:0/2425689340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 msgr2=0x55724eb274a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa267400640 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb274a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa267400640 1 -- 172.21.15.116:0/2425689340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 msgr2=0x55724eb20460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa267400640 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb20460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa267400640 1 -- 172.21.15.116:0/2425689340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55724e78f3f0 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa267400640 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb1ff20 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa258000c00 tx=0x7fa25800f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfd6306c4eb4faf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa250c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa250c00640 1 -- 172.21.15.116:0/2425689340 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258026020 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa250c00640 1 -- 172.21.15.116:0/2425689340 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa258004040 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.502+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa25800faa0 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa250c00640 1 -- 172.21.15.116:0/2425689340 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa25801e040 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 10 client.28178 fetch_fsmap learned FSMap version 19 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55724e74f0f0 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa250c00640 1 -- 172.21.15.116:0/2425689340 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa25800faa0 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa250c00640 1 -- 172.21.15.116:0/2425689340 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa258021030 con 0x55724eb15b50 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 10 client.28178 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:52:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 10 client.28178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:52:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 20 client.28178 populate_metadata read hostname 'smithi116' 2024-03-04T07:52:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb2b650 0x55724eb2daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 4 client.28178 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:52:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.503+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55724ea88270 con 0x55724eb2b650 2024-03-04T07:52:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.504+0000 7fa266a00640 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb2b650 0x55724eb2daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.504+0000 7fa266a00640 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb2b650 0x55724eb2daa0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=819f5de5051fa3b3 server_cookie=274482e0e8c7a39e in_seq=0 out_seq=0 2024-03-04T07:52:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.504+0000 7fa250c00640 10 client.28178 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa250c00640 1 -- 172.21.15.116:0/2425689340 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55724ea88270 con 0x55724eb2b650 2024-03-04T07:52:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa250c00640 10 client.28178 handle_command_reply: tid=0 2024-03-04T07:52:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 client.28178 shutdown 2024-03-04T07:52:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 msgr2=0x55724eb1ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb1ff20 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa258000c00 tx=0x7fa25800f870 comp rx=0 tx=0).stop 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 shutdown_connections 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb2b650 0x55724eb2daa0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb274a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb20460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa250c00640 0 client.28178 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 --2- 172.21.15.116:0/2425689340 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb1ff20 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 >> 172.21.15.116:0/2425689340 conn(0x55724eb110c0 msgr2=0x55724eb10360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 shutdown_connections 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 1 -- 172.21.15.116:0/2425689340 wait complete. 2024-03-04T07:52:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.505+0000 7fa270df1740 20 client.28178 trim_cache size 0 max 16384 2024-03-04T07:52:10.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 Processor -- start 2024-03-04T07:52:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 -- start start 2024-03-04T07:52:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb15b50 0x55724eb14df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb27460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb1b6b0 0x55724eb1f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681a8d10 con 0x55724eb15b50 2024-03-04T07:52:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681ab2f0 con 0x55724eb1b6b0 2024-03-04T07:52:10.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.523+0000 7fa270df1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681a9310 con 0x55724eb16540 2024-03-04T07:52:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb1b6b0 0x55724eb1f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb15b50 0x55724eb14df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb15b50 0x55724eb14df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48952/0 (socket says 172.21.15.116:48952) 2024-03-04T07:52:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267400640 1 -- 172.21.15.116:0/2113336417 learned_addr learned my addr 172.21.15.116:0/2113336417 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb1b6b0 0x55724eb1f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42182/0 (socket says 172.21.15.116:42182) 2024-03-04T07:52:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa266a00640 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb27460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267e00640 1 -- 172.21.15.116:0/2113336417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 msgr2=0x55724eb27460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267e00640 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb27460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267e00640 1 -- 172.21.15.116:0/2113336417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb15b50 msgr2=0x55724eb14df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267e00640 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb15b50 0x55724eb14df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267e00640 1 -- 172.21.15.116:0/2113336417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55724eb11e30 con 0x55724eb1b6b0 2024-03-04T07:52:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.524+0000 7fa267400640 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb15b50 0x55724eb14df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa267e00640 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb1b6b0 0x55724eb1f540 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa260005c40 tx=0x7fa26009a870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75cc575f9dc8e12e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa253400640 1 -- 172.21.15.116:0/2113336417 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600b2070 con 0x55724eb1b6b0 2024-03-04T07:52:10.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa253400640 1 -- 172.21.15.116:0/2113336417 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2600a3600 con 0x55724eb1b6b0 2024-03-04T07:52:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa253400640 1 -- 172.21.15.116:0/2113336417 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26001f070 con 0x55724eb1b6b0 2024-03-04T07:52:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa270df1740 1 -- 172.21.15.116:0/2113336417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb1b6b0 msgr2=0x55724eb1f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa270df1740 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb1b6b0 0x55724eb1f540 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa260005c40 tx=0x7fa26009a870 comp rx=0 tx=0).stop 2024-03-04T07:52:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa270df1740 1 -- 172.21.15.116:0/2113336417 shutdown_connections 2024-03-04T07:52:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.525+0000 7fa270df1740 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb1b6b0 0x55724eb1f540 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.526+0000 7fa270df1740 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb16540 0x55724eb27460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.526+0000 7fa270df1740 1 --2- 172.21.15.116:0/2113336417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb15b50 0x55724eb14df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.526+0000 7fa270df1740 1 -- 172.21.15.116:0/2113336417 >> 172.21.15.116:0/2113336417 conn(0x55724eb110c0 msgr2=0x55724eb10360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.526+0000 7fa270df1740 1 -- 172.21.15.116:0/2113336417 shutdown_connections 2024-03-04T07:52:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.526+0000 7fa270df1740 1 -- 172.21.15.116:0/2113336417 wait complete. 2024-03-04T07:52:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.526+0000 7fa270df1740 1 -- start start 2024-03-04T07:52:10.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.526+0000 7fa270df1740 1 Processor -- start 2024-03-04T07:52:10.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa270df1740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb27140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa270df1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb257c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa270df1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb25d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa270df1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681a9970 con 0x55724eb16540 2024-03-04T07:52:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa270df1740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681aac90 con 0x55724eb15b50 2024-03-04T07:52:10.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa270df1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681a9fd0 con 0x55724eb1b6b0 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa270df1740 10 client.0 _ll_register_callbacks cb 0x7fa258005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb27140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa266a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb257c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb27140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42186/0 (socket says 172.21.15.116:42186) 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 -- 172.21.15.116:0/3785093379 learned_addr learned my addr 172.21.15.116:0/3785093379 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267e00640 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb25d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 -- 172.21.15.116:0/3785093379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 msgr2=0x55724eb25d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb25d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 -- 172.21.15.116:0/3785093379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 msgr2=0x55724eb257c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb257c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa267400640 1 -- 172.21.15.116:0/3785093379 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55724eb26340 con 0x55724eb15b50 2024-03-04T07:52:10.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.527+0000 7fa266a00640 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb257c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa267400640 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb27140 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa258000c00 tx=0x7fa258003c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d08a998334a1cf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa250c00640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa250c00640 1 -- 172.21.15.116:0/3785093379 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258026070 con 0x55724eb15b50 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa250c00640 1 -- 172.21.15.116:0/3785093379 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa258019450 con 0x55724eb15b50 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2600a3600 con 0x55724eb15b50 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa250c00640 1 -- 172.21.15.116:0/3785093379 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa25803c050 con 0x55724eb15b50 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa270df1740 10 client.28184 fetch_fsmap learned FSMap version 19 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x55724eb87d70 con 0x55724eb15b50 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa250c00640 1 -- 172.21.15.116:0/3785093379 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2600a3600 con 0x55724eb15b50 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.528+0000 7fa250c00640 1 -- 172.21.15.116:0/3785093379 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fa258034020 con 0x55724eb15b50 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa270df1740 10 client.28184 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa270df1740 10 client.28184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa270df1740 20 client.28184 populate_metadata read hostname 'smithi116' 2024-03-04T07:52:10.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa270df1740 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb88630 0x55724eb8aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa270df1740 4 client.28184 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55724ea825c0 con 0x55724eb88630 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa266a00640 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb88630 0x55724eb8aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa266a00640 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb88630 0x55724eb8aa50 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da92b1e3bf728899 server_cookie=3e3c84b3f5673b2b in_seq=0 out_seq=0 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa250c00640 10 client.28184 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa250c00640 1 -- 172.21.15.116:0/3785093379 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55724ea825c0 con 0x55724eb88630 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa250c00640 10 client.28184 handle_command_reply: tid=0 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.529+0000 7fa270df1740 1 client.28184 shutdown 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 msgr2=0x55724eb27140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb27140 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa258000c00 tx=0x7fa258003c60 comp rx=0 tx=0).stop 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 shutdown_connections 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x55724eb88630 0x55724eb8aa50 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55724eb1b6b0 0x55724eb25d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa250c00640 0 client.28184 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55724eb16540 0x55724eb257c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 --2- 172.21.15.116:0/3785093379 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x55724eb15b50 0x55724eb27140 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 >> 172.21.15.116:0/3785093379 conn(0x55724eb110c0 msgr2=0x55724eb32360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 shutdown_connections 2024-03-04T07:52:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 1 -- 172.21.15.116:0/3785093379 wait complete. 2024-03-04T07:52:10.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.530+0000 7fa270df1740 20 client.28184 trim_cache size 0 max 16384 2024-03-04T07:52:10.546 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:52:10.546 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-04T07:52:10.546 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-04T07:52:10.546 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa24c0a1d80 msgr2=0x7fa24c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa24c0a1d80 0x7fa24c0a4240 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55724eb16da0 tx=0x7fa25402e040 comp rx=0 tx=0).stop 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa268101150 msgr2=0x7fa2681a10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa268101150 0x7fa2681a10d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e940 tx=0x7fa26000ee00 comp rx=0 tx=0).stop 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 shutdown_connections 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa268101150 0x7fa2681a10d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa24c0a1d80 0x7fa24c0a4240 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa2681007b0 0x7fa26819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 --2- 172.21.15.116:0/691038739 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2680ffe50 0x7fa268199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 >> 172.21.15.116:0/691038739 conn(0x7fa2680fba70 msgr2=0x7fa2680fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:10.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 shutdown_connections 2024-03-04T07:52:10.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.531+0000 7fa26e000640 1 -- 172.21.15.116:0/691038739 wait complete. 2024-03-04T07:52:10.549 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T07:52:10.549 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:52:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.765+0000 7f63cc200640 1 -- 172.21.15.116:0/1131948294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4107c10 msgr2=0x7f63c4108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.765+0000 7f63cc200640 1 --2- 172.21.15.116:0/1131948294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4107c10 0x7f63c4108030 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009900 tx=0x7f63b802bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.766+0000 7f63cc200640 1 -- 172.21.15.116:0/1131948294 shutdown_connections 2024-03-04T07:52:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.766+0000 7f63cc200640 1 --2- 172.21.15.116:0/1131948294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c41097c0 0x7f63c4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.766+0000 7f63cc200640 1 --2- 172.21.15.116:0/1131948294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63c4108e00 0x7f63c4109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.766+0000 7f63cc200640 1 --2- 172.21.15.116:0/1131948294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4107c10 0x7f63c4108030 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.766+0000 7f63cc200640 1 -- 172.21.15.116:0/1131948294 >> 172.21.15.116:0/1131948294 conn(0x7f63c4076be0 msgr2=0x7f63c4079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.766+0000 7f63cc200640 1 -- 172.21.15.116:0/1131948294 shutdown_connections 2024-03-04T07:52:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.766+0000 7f63cc200640 1 -- 172.21.15.116:0/1131948294 wait complete. 2024-03-04T07:52:10.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.767+0000 7f63cc200640 1 Processor -- start 2024-03-04T07:52:10.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.767+0000 7f63cc200640 1 -- start start 2024-03-04T07:52:10.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.767+0000 7f63cc200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c4107c10 0x7f63c419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.767+0000 7f63cc200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4108e00 0x7f63c419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63cc200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63c41097c0 0x7f63c41a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63cc200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c41138d0 con 0x7f63c4107c10 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63cc200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4113750 con 0x7f63c4108e00 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63cc200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4113a50 con 0x7f63c41097c0 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c8e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4108e00 0x7f63c419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c9800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c4107c10 0x7f63c419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c8e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4108e00 0x7f63c419e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42200/0 (socket says 172.21.15.116:42200) 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c8e00640 1 -- 172.21.15.116:0/4047528983 learned_addr learned my addr 172.21.15.116:0/4047528983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63ca200640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63c41097c0 0x7f63c41a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c9800640 1 -- 172.21.15.116:0/4047528983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63c41097c0 msgr2=0x7f63c41a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c9800640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63c41097c0 0x7f63c41a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c9800640 1 -- 172.21.15.116:0/4047528983 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4108e00 msgr2=0x7f63c419e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c9800640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4108e00 0x7f63c419e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.768+0000 7f63c9800640 1 -- 172.21.15.116:0/4047528983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c41a5b50 con 0x7f63c4107c10 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.769+0000 7f63c9800640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c4107c10 0x7f63c419df80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009230 tx=0x7f63b802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.769+0000 7f63c2000640 1 -- 172.21.15.116:0/4047528983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8043020 con 0x7f63c4107c10 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.769+0000 7f63c2000640 1 -- 172.21.15.116:0/4047528983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63b80054e0 con 0x7f63c4107c10 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.769+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c41a5de0 con 0x7f63c4107c10 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.769+0000 7f63c2000640 1 -- 172.21.15.116:0/4047528983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8003b60 con 0x7f63c4107c10 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.769+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c41a6240 con 0x7f63c4107c10 2024-03-04T07:52:10.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.770+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c410efb0 con 0x7f63c4107c10 2024-03-04T07:52:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.772+0000 7f63c2000640 1 -- 172.21.15.116:0/4047528983 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f63b8046020 con 0x7f63c4107c10 2024-03-04T07:52:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.773+0000 7f63c2000640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63ac0a1d80 0x7f63ac0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.774+0000 7f63c2000640 1 -- 172.21.15.116:0/4047528983 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f63b80efe50 con 0x7f63c4107c10 2024-03-04T07:52:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.774+0000 7f63c8e00640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63ac0a1d80 0x7f63ac0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.775+0000 7f63c8e00640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63ac0a1d80 0x7f63ac0a4240 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f63c419f440 tx=0x7f63b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:10.778+0000 7f63c2000640 1 -- 172.21.15.116:0/4047528983 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f63b804d020 con 0x7f63c4107c10 2024-03-04T07:52:11.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.016+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c4062270 con 0x7f63c4107c10 2024-03-04T07:52:11.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.018+0000 7f63c2000640 1 -- 172.21.15.116:0/4047528983 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f63b80bd240 con 0x7f63c4107c10 2024-03-04T07:52:11.018 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:52:11.019 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:52:11.019 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.020+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63ac0a1d80 msgr2=0x7f63ac0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63ac0a1d80 0x7f63ac0a4240 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f63c419f440 tx=0x7f63b0005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c4107c10 msgr2=0x7f63c419df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c4107c10 0x7f63c419df80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009230 tx=0x7f63b802c5d0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 shutdown_connections 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f63ac0a1d80 0x7f63ac0a4240 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63c41097c0 0x7f63c41a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f63c4108e00 0x7f63c419e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 --2- 172.21.15.116:0/4047528983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63c4107c10 0x7f63c419df80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 >> 172.21.15.116:0/4047528983 conn(0x7f63c4076be0 msgr2=0x7f63c410c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:11.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 shutdown_connections 2024-03-04T07:52:11.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.021+0000 7f63cc200640 1 -- 172.21.15.116:0/4047528983 wait complete. 2024-03-04T07:52:11.042 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T07:52:11.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T07:52:11.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:10 smithi116 ceph-mon[32150]: pgmap v743: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-03-04T07:52:11.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 -- 172.21.15.116:0/341077475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 msgr2=0x7f36f8074430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 --2- 172.21.15.116:0/341077475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8074430 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f36e0009900 tx=0x7f36e002bcc0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 -- 172.21.15.116:0/341077475 shutdown_connections 2024-03-04T07:52:11.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 --2- 172.21.15.116:0/341077475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36f8109470 0x7f36f810b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 --2- 172.21.15.116:0/341077475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8074430 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 --2- 172.21.15.116:0/341077475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f80755b0 0x7f36f8073a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 -- 172.21.15.116:0/341077475 >> 172.21.15.116:0/341077475 conn(0x7f36f80fcb70 msgr2=0x7f36f80fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:11.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 -- 172.21.15.116:0/341077475 shutdown_connections 2024-03-04T07:52:11.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.261+0000 7f36fe600640 1 -- 172.21.15.116:0/341077475 wait complete. 2024-03-04T07:52:11.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.262+0000 7f36fe600640 1 Processor -- start 2024-03-04T07:52:11.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36fe600640 1 -- start start 2024-03-04T07:52:11.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36fe600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36fe600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36f80755b0 0x7f36f819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36fe600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f8109470 0x7f36f81a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36fe600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f810f4b0 con 0x7f36f8109470 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36fe600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f810f330 con 0x7f36f8073fd0 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36fe600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f810f630 con 0x7f36f80755b0 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36f7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36f7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f8109470 0x7f36f81a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36f7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42226/0 (socket says 172.21.15.116:42226) 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36f7400640 1 -- 172.21.15.116:0/1840847594 learned_addr learned my addr 172.21.15.116:0/1840847594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.263+0000 7f36f6a00640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36f80755b0 0x7f36f819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36f7e00640 1 -- 172.21.15.116:0/1840847594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36f80755b0 msgr2=0x7f36f819a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36f7e00640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36f80755b0 0x7f36f819a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36f7e00640 1 -- 172.21.15.116:0/1840847594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 msgr2=0x7f36f8199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36f7e00640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36f7e00640 1 -- 172.21.15.116:0/1840847594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f81a1710 con 0x7f36f8109470 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36f7400640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36f7e00640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f8109470 0x7f36f81a10a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009fd0 tx=0x7f36f000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36efe00640 1 -- 172.21.15.116:0/1840847594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f000ce20 con 0x7f36f8109470 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f81a1a00 con 0x7f36f8109470 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.264+0000 7f36efe00640 1 -- 172.21.15.116:0/1840847594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36f0005020 con 0x7f36f8109470 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.265+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f81a1f70 con 0x7f36f8109470 2024-03-04T07:52:11.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.265+0000 7f36efe00640 1 -- 172.21.15.116:0/1840847594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0010720 con 0x7f36f8109470 2024-03-04T07:52:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 Processor -- start 2024-03-04T07:52:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f36efe00640 1 -- 172.21.15.116:0/1840847594 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f36f0005540 con 0x7f36f8109470 2024-03-04T07:52:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 -- start start 2024-03-04T07:52:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f4130c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f418190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f418800 0x56029f41ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029f377d60 con 0x56029f412ca0 2024-03-04T07:52:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029f320cf0 con 0x56029f418800 2024-03-04T07:52:11.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.268+0000 7f37011f7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029f373e00 con 0x56029f413690 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f4130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f4130c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49018/0 (socket says 172.21.15.116:49018) 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f6a00640 1 -- 172.21.15.116:0/2545748088 learned_addr learned my addr 172.21.15.116:0/2545748088 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f7400640 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f418190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f6a00640 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f418800 0x56029f41ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f7400640 1 -- 172.21.15.116:0/2545748088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f418800 msgr2=0x56029f41ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f7400640 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f418800 0x56029f41ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f7400640 1 -- 172.21.15.116:0/2545748088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 msgr2=0x56029f4130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f7400640 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f4130c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f7400640 1 -- 172.21.15.116:0/2545748088 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56029f08c3f0 con 0x56029f413690 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36efe00640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f36dc0a1e00 0x7f36dc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.269+0000 7f36f7400640 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f418190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f36e80114f0 tx=0x7f36e80119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88a20b9a6ac9c1db server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:11.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.270+0000 7f36efe00640 1 -- 172.21.15.116:0/1840847594 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f36f00ce360 con 0x7f36f8109470 2024-03-04T07:52:11.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.270+0000 7f36f6a00640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f36dc0a1e00 0x7f36dc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.270+0000 7f36ecc00640 1 -- 172.21.15.116:0/2545748088 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e801e070 con 0x56029f413690 2024-03-04T07:52:11.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.270+0000 7f36ecc00640 1 -- 172.21.15.116:0/2545748088 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36e80035f0 con 0x56029f413690 2024-03-04T07:52:11.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.270+0000 7f36ecc00640 1 -- 172.21.15.116:0/2545748088 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e800f820 con 0x56029f413690 2024-03-04T07:52:11.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 -- 172.21.15.116:0/2545748088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 msgr2=0x56029f418190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f36f6a00640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f36dc0a1e00 0x7f36dc0a42c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005ec0 tx=0x7f36e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f418190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f36e80114f0 tx=0x7f36e80119b0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 -- 172.21.15.116:0/2545748088 shutdown_connections 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f418800 0x56029f41ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f418190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 --2- 172.21.15.116:0/2545748088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f4130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 -- 172.21.15.116:0/2545748088 >> 172.21.15.116:0/2545748088 conn(0x56029f40e210 msgr2=0x56029f40e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 -- 172.21.15.116:0/2545748088 shutdown_connections 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 -- 172.21.15.116:0/2545748088 wait complete. 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.271+0000 7f37011f7740 1 -- start start 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 1 Processor -- start 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f41d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f41d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f418800 0x56029f4246d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029f41b7b0 con 0x56029f418800 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029f316220 con 0x56029f412ca0 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029f41b930 con 0x56029f413690 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f37011f7740 10 client.0 _ll_register_callbacks cb 0x56029f08bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f36f7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f41d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f36f7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f41d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42248/0 (socket says 172.21.15.116:42248) 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f36f7400640 1 -- 172.21.15.116:0/3756849042 learned_addr learned my addr 172.21.15.116:0/3756849042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f36f7e00640 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f418800 0x56029f4246d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.272+0000 7f36f6a00640 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f41d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36f7400640 1 -- 172.21.15.116:0/3756849042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 msgr2=0x56029f41d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36f7400640 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f41d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36f7400640 1 -- 172.21.15.116:0/3756849042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f418800 msgr2=0x56029f4246d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36f7400640 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f418800 0x56029f4246d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36f7400640 1 -- 172.21.15.116:0/3756849042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56029f1567d0 con 0x56029f412ca0 2024-03-04T07:52:11.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36f7400640 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f41d150 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f36e8000c00 tx=0x7f36e8003a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98477121938d3fed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:11.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36e5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:52:11.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36e5600640 1 -- 172.21.15.116:0/3756849042 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e801e030 con 0x56029f412ca0 2024-03-04T07:52:11.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36e80035f0 con 0x56029f412ca0 2024-03-04T07:52:11.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.273+0000 7f36e5600640 1 -- 172.21.15.116:0/3756849042 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f36e8005260 con 0x56029f412ca0 2024-03-04T07:52:11.277 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:10 smithi152 ceph-mon[37154]: pgmap v743: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-03-04T07:52:11.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f36e5600640 1 -- 172.21.15.116:0/3756849042 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36e8005c50 con 0x56029f412ca0 2024-03-04T07:52:11.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 10 client.28202 fetch_fsmap learned FSMap version 19 2024-03-04T07:52:11.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56029f427a40 con 0x56029f412ca0 2024-03-04T07:52:11.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f36e5600640 1 -- 172.21.15.116:0/3756849042 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36e80035f0 con 0x56029f412ca0 2024-03-04T07:52:11.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f36e5600640 1 -- 172.21.15.116:0/3756849042 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f36e800f650 con 0x56029f412ca0 2024-03-04T07:52:11.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 10 client.28202 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:52:11.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 10 client.28202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 20 client.28202 populate_metadata read hostname 'smithi116' 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f428f30 0x56029f42b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 4 client.28202 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.274+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56029f3852a0 con 0x56029f428f30 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.275+0000 7f36f6a00640 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f428f30 0x56029f42b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.275+0000 7f36f6a00640 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f428f30 0x56029f42b3b0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a7fd9aef56fd3d server_cookie=56dfcb8321d63c93 in_seq=0 out_seq=0 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.275+0000 7f36e5600640 10 client.28202 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f36e5600640 1 -- 172.21.15.116:0/3756849042 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56029f3852a0 con 0x56029f428f30 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f36e5600640 10 client.28202 handle_command_reply: tid=0 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 client.28202 shutdown 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 msgr2=0x56029f41d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f41d150 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f36e8000c00 tx=0x7f36e8003a50 comp rx=0 tx=0).stop 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 shutdown_connections 2024-03-04T07:52:11.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f428f30 0x56029f42b3b0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f418800 0x56029f4246d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f413690 0x56029f41d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 --2- 172.21.15.116:0/3756849042 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f41d150 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 >> 172.21.15.116:0/3756849042 conn(0x56029f40e210 msgr2=0x56029f40d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f36e5600640 0 client.28202 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 shutdown_connections 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 1 -- 172.21.15.116:0/3756849042 wait complete. 2024-03-04T07:52:11.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.276+0000 7f37011f7740 20 client.28202 trim_cache size 0 max 16384 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 Processor -- start 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 -- start start 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f40d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f413690 0x56029f424690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f42fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81a8cc0 con 0x56029f413690 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81ab2a0 con 0x56029f412ca0 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f37011f7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81a92c0 con 0x56029f418800 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f36f7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f40d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f36f7400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f40d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42258/0 (socket says 172.21.15.116:42258) 2024-03-04T07:52:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f36f7400640 1 -- 172.21.15.116:0/3235942987 learned_addr learned my addr 172.21.15.116:0/3235942987 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f36f6a00640 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f413690 0x56029f424690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.290+0000 7f36f7e00640 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f42fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36f7400640 1 -- 172.21.15.116:0/3235942987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 msgr2=0x56029f42fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36f7400640 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f42fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36f7400640 1 -- 172.21.15.116:0/3235942987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f413690 msgr2=0x56029f424690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36f7400640 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f413690 0x56029f424690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36f7400640 1 -- 172.21.15.116:0/3235942987 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56029f417dc0 con 0x56029f412ca0 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36f7400640 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f40d330 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f36e8003d20 tx=0x7f36e8003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b541e0979afb3968 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36e7e00640 1 -- 172.21.15.116:0/3235942987 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e801e040 con 0x56029f412ca0 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36e7e00640 1 -- 172.21.15.116:0/3235942987 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36e8028070 con 0x56029f412ca0 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.291+0000 7f36e7e00640 1 -- 172.21.15.116:0/3235942987 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8004680 con 0x56029f412ca0 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 -- 172.21.15.116:0/3235942987 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 msgr2=0x56029f40d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f40d330 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f36e8003d20 tx=0x7f36e8003ea0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 -- 172.21.15.116:0/3235942987 shutdown_connections 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f42fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f413690 0x56029f424690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 --2- 172.21.15.116:0/3235942987 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f412ca0 0x56029f40d330 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 -- 172.21.15.116:0/3235942987 >> 172.21.15.116:0/3235942987 conn(0x56029f40e210 msgr2=0x56029f40cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 -- 172.21.15.116:0/3235942987 shutdown_connections 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 -- 172.21.15.116:0/3235942987 wait complete. 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 -- start start 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.292+0000 7f37011f7740 1 Processor -- start 2024-03-04T07:52:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f37011f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f422620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f37011f7740 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f413690 0x56029f422b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f37011f7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f423b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f37011f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81a9920 con 0x56029f412ca0 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f37011f7740 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81aac40 con 0x56029f413690 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f37011f7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81a9f80 con 0x56029f418800 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f37011f7740 10 client.0 _ll_register_callbacks cb 0x7f36e801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f36f7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f422620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f36f6a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f413690 0x56029f422b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f36f7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f422620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49038/0 (socket says 172.21.15.116:49038) 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f36f7400640 1 -- 172.21.15.116:0/4263259013 learned_addr learned my addr 172.21.15.116:0/4263259013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:11.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f36f7e00640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f423b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f36f6a00640 1 -- 172.21.15.116:0/4263259013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 msgr2=0x56029f423b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.293+0000 7f36f6a00640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f423b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36f6a00640 1 -- 172.21.15.116:0/4263259013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 msgr2=0x56029f422620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36f6a00640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f422620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36f6a00640 1 -- 172.21.15.116:0/4263259013 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56029f424160 con 0x56029f413690 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36f7400640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f422620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36f7e00640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f423b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36f6a00640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f413690 0x56029f422b60 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f36e0000c00 tx=0x7f36e0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b490e1685555cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36e5600640 10 client.0 ms_handle_connect on v2:172.21.15.133:3300/0 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36e5600640 1 -- 172.21.15.116:0/4263259013 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0004f10 con 0x56029f413690 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36e5600640 1 -- 172.21.15.116:0/4263259013 <== mon.2 v2:172.21.15.133:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f36e00050b0 con 0x56029f413690 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36e8028070 con 0x56029f413690 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f36e5600640 1 -- 172.21.15.116:0/4263259013 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36e00052b0 con 0x56029f413690 2024-03-04T07:52:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f37011f7740 10 client.28208 fetch_fsmap learned FSMap version 19 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.294+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=4+}) v3 -- 0x56029f485150 con 0x56029f413690 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f36e5600640 1 -- 172.21.15.116:0/4263259013 <== mon.2 v2:172.21.15.133:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36e8028070 con 0x56029f413690 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f36e5600640 1 -- 172.21.15.116:0/4263259013 <== mon.2 v2:172.21.15.133:3300/0 5 ==== fsmap(e 19) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f36e004f450 con 0x56029f413690 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f37011f7740 10 client.28208 fetch_fsmap finished waiting for FSMap version 19 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f37011f7740 10 client.28208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f37011f7740 20 client.28208 populate_metadata read hostname 'smithi116' 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f37011f7740 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f485a50 0x56029f487e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f37011f7740 4 client.28208 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 --> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56029f37f5f0 con 0x56029f485a50 2024-03-04T07:52:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.295+0000 7f36f7400640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f485a50 0x56029f487e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:11.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f36f7400640 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f485a50 0x56029f487e70 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=621102810d346920 server_cookie=cedf0eda71177f59 in_seq=0 out_seq=0 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f36e5600640 10 client.28208 ms_handle_connect on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f36e5600640 1 -- 172.21.15.116:0/4263259013 <== mds.0 v2:172.21.15.133:6838/1914821860 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56029f37f5f0 con 0x56029f485a50 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f36e5600640 10 client.28208 handle_command_reply: tid=0 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 client.28208 shutdown 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f413690 msgr2=0x56029f422b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f413690 0x56029f422b60 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f36e0000c00 tx=0x7f36e0009260 comp rx=0 tx=0).stop 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 shutdown_connections 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:6838/1914821860,v1:172.21.15.133:6839/1914821860] conn(0x56029f485a50 0x56029f487e70 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56029f418800 0x56029f423b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x56029f413690 0x56029f422b60 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f36e5600640 0 client.28208 ms_handle_reset on v2:172.21.15.133:6838/1914821860 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 --2- 172.21.15.116:0/4263259013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56029f412ca0 0x56029f422620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 >> 172.21.15.116:0/4263259013 conn(0x56029f40e210 msgr2=0x56029f411d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:11.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 shutdown_connections 2024-03-04T07:52:11.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 1 -- 172.21.15.116:0/4263259013 wait complete. 2024-03-04T07:52:11.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.296+0000 7f37011f7740 20 client.28208 trim_cache size 0 max 16384 2024-03-04T07:52:11.303 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f36dc0a1e00 msgr2=0x7f36dc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f36dc0a1e00 0x7f36dc0a42c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005ec0 tx=0x7f36e0005fd0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f8109470 msgr2=0x7f36f81a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f8109470 0x7f36f81a10a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009fd0 tx=0x7f36f000ef10 comp rx=0 tx=0).stop 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 shutdown_connections 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f8109470 0x7f36f81a10a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f36dc0a1e00 0x7f36dc0a42c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36f80755b0 0x7f36f819a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 --2- 172.21.15.116:0/1840847594 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f36f8073fd0 0x7f36f8199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 >> 172.21.15.116:0/1840847594 conn(0x7f36f80fcb70 msgr2=0x7f36f8105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 shutdown_connections 2024-03-04T07:52:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:11.297+0000 7f36fe600640 1 -- 172.21.15.116:0/1840847594 wait complete. 2024-03-04T07:52:11.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:10 smithi133 ceph-mon[36505]: pgmap v743: 97 pgs: 97 active+clean; 4.5 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-03-04T07:52:12.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:11 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4047528983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:12.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:11 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4047528983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:12.314 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-04T07:52:12.315 INFO:tasks.fwd_scrub:done joining 2024-03-04T07:52:12.315 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-04T07:52:12.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:52:12.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:11 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4047528983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 -- 172.21.15.116:0/4276231331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c1097c0 msgr2=0x7fb61c110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 --2- 172.21.15.116:0/4276231331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c1097c0 0x7fb61c110050 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb614009900 tx=0x7fb61402bc50 comp rx=0 tx=0).stop 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 -- 172.21.15.116:0/4276231331 shutdown_connections 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 --2- 172.21.15.116:0/4276231331 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c1097c0 0x7fb61c110050 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 --2- 172.21.15.116:0/4276231331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb61c108e00 0x7fb61c109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 --2- 172.21.15.116:0/4276231331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61c107c10 0x7fb61c108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 -- 172.21.15.116:0/4276231331 >> 172.21.15.116:0/4276231331 conn(0x7fb61c076be0 msgr2=0x7fb61c079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.545+0000 7fb621e00640 1 -- 172.21.15.116:0/4276231331 shutdown_connections 2024-03-04T07:52:12.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.546+0000 7fb621e00640 1 -- 172.21.15.116:0/4276231331 wait complete. 2024-03-04T07:52:12.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.546+0000 7fb621e00640 1 Processor -- start 2024-03-04T07:52:12.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb621e00640 1 -- start start 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb621e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c107c10 0x7fb61c19dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb621e00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb61c108e00 0x7fb61c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb621e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61c1097c0 0x7fb61c1a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb621e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c1138e0 con 0x7fb61c1097c0 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb621e00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c113760 con 0x7fb61c107c10 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb621e00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c113a60 con 0x7fb61c108e00 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb61aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c107c10 0x7fb61c19dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb61b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61c1097c0 0x7fb61c1a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb61aa00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c107c10 0x7fb61c19dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42280/0 (socket says 172.21.15.116:42280) 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb61aa00640 1 -- 172.21.15.116:0/2902353205 learned_addr learned my addr 172.21.15.116:0/2902353205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.547+0000 7fb61a000640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb61c108e00 0x7fb61c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb61aa00640 1 -- 172.21.15.116:0/2902353205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb61c108e00 msgr2=0x7fb61c19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb61aa00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb61c108e00 0x7fb61c19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb61aa00640 1 -- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61c1097c0 msgr2=0x7fb61c1a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb61aa00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61c1097c0 0x7fb61c1a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb61aa00640 1 -- 172.21.15.116:0/2902353205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c1a5ac0 con 0x7fb61c107c10 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb61aa00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c107c10 0x7fb61c19dea0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009d20 tx=0x7fb60c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb620a00640 1 -- 172.21.15.116:0/2902353205 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c010630 con 0x7fb61c107c10 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb620a00640 1 -- 172.21.15.116:0/2902353205 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb60c010cb0 con 0x7fb61c107c10 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb620a00640 1 -- 172.21.15.116:0/2902353205 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c018de0 con 0x7fb61c107c10 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c1a5db0 con 0x7fb61c107c10 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.548+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c1a6220 con 0x7fb61c107c10 2024-03-04T07:52:12.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.550+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61c10ee60 con 0x7fb61c107c10 2024-03-04T07:52:12.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.551+0000 7fb620a00640 1 -- 172.21.15.116:0/2902353205 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb60c01f020 con 0x7fb61c107c10 2024-03-04T07:52:12.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.552+0000 7fb620a00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6000a1e00 0x7fb6000a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.553+0000 7fb620a00640 1 -- 172.21.15.116:0/2902353205 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb60c0ce180 con 0x7fb61c107c10 2024-03-04T07:52:12.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.553+0000 7fb61a000640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6000a1e00 0x7fb6000a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.553+0000 7fb61a000640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6000a1e00 0x7fb6000a42c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb61c19f3b0 tx=0x7fb608005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:12.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.556+0000 7fb620a00640 1 -- 172.21.15.116:0/2902353205 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb60c09b5b0 con 0x7fb61c107c10 2024-03-04T07:52:12.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.756+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb61c0622e0 con 0x7fb61c107c10 2024-03-04T07:52:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.758+0000 7fb620a00640 1 -- 172.21.15.116:0/2902353205 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7fb60c09aeb0 con 0x7fb61c107c10 2024-03-04T07:52:12.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:52:12.759 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.760+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6000a1e00 msgr2=0x7fb6000a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.760+0000 7fb621e00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6000a1e00 0x7fb6000a42c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb61c19f3b0 tx=0x7fb608005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.760+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c107c10 msgr2=0x7fb61c19dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c107c10 0x7fb61c19dea0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009d20 tx=0x7fb60c00c6a0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 shutdown_connections 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61c1097c0 0x7fb61c1a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6000a1e00 0x7fb6000a42c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb61c108e00 0x7fb61c19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 --2- 172.21.15.116:0/2902353205 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb61c107c10 0x7fb61c19dea0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 >> 172.21.15.116:0/2902353205 conn(0x7fb61c076be0 msgr2=0x7fb61c10c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 shutdown_connections 2024-03-04T07:52:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.761+0000 7fb621e00640 1 -- 172.21.15.116:0/2902353205 wait complete. 2024-03-04T07:52:12.780 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:52:12.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:52:12.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 -- 172.21.15.116:0/1234287678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941053a0 msgr2=0x7f5b9410bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 --2- 172.21.15.116:0/1234287678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941053a0 0x7f5b9410bc30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009ca0 tx=0x7f5b8c02c0f0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 -- 172.21.15.116:0/1234287678 shutdown_connections 2024-03-04T07:52:12.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 --2- 172.21.15.116:0/1234287678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941053a0 0x7f5b9410bc30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 --2- 172.21.15.116:0/1234287678 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b941049e0 0x7f5b94104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 --2- 172.21.15.116:0/1234287678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 0x7f5b94103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 -- 172.21.15.116:0/1234287678 >> 172.21.15.116:0/1234287678 conn(0x7f5b940fed80 msgr2=0x7f5b941011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:12.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 -- 172.21.15.116:0/1234287678 shutdown_connections 2024-03-04T07:52:12.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.969+0000 7f5b9b600640 1 -- 172.21.15.116:0/1234287678 wait complete. 2024-03-04T07:52:12.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.970+0000 7f5b9b600640 1 Processor -- start 2024-03-04T07:52:12.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.970+0000 7f5b9b600640 1 -- start start 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b9b600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 0x7f5b94199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b9b600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941049e0 0x7f5b94199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b9b600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b941053a0 0x7f5b941a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b9b600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9410f450 con 0x7f5b941037f0 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b9b600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9410f2d0 con 0x7f5b941053a0 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b9b600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9410f5d0 con 0x7f5b941049e0 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 0x7f5b94199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 0x7f5b94199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49074/0 (socket says 172.21.15.116:49074) 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 -- 172.21.15.116:0/751046893 learned_addr learned my addr 172.21.15.116:0/751046893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b99600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b941053a0 0x7f5b941a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b93e00640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941049e0 0x7f5b94199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 -- 172.21.15.116:0/751046893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941049e0 msgr2=0x7f5b94199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941049e0 0x7f5b94199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 -- 172.21.15.116:0/751046893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b941053a0 msgr2=0x7f5b941a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:12.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b941053a0 0x7f5b941a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:12.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.971+0000 7f5b98c00640 1 -- 172.21.15.116:0/751046893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b941a1650 con 0x7f5b941037f0 2024-03-04T07:52:12.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.972+0000 7f5b98c00640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 0x7f5b94199a30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800e9c0 tx=0x7f5b8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.972+0000 7f5b91600640 1 -- 172.21.15.116:0/751046893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8800cde0 con 0x7f5b941037f0 2024-03-04T07:52:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.972+0000 7f5b91600640 1 -- 172.21.15.116:0/751046893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b88010910 con 0x7f5b941037f0 2024-03-04T07:52:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.972+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b941a1940 con 0x7f5b941037f0 2024-03-04T07:52:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.972+0000 7f5b91600640 1 -- 172.21.15.116:0/751046893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b88018a10 con 0x7f5b941037f0 2024-03-04T07:52:12.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.972+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b941a1e90 con 0x7f5b941037f0 2024-03-04T07:52:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.973+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b94068810 con 0x7f5b941037f0 2024-03-04T07:52:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.975+0000 7f5b91600640 1 -- 172.21.15.116:0/751046893 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f5b8801f020 con 0x7f5b941037f0 2024-03-04T07:52:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.976+0000 7f5b91600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b7c0a1d80 0x7f5b7c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.976+0000 7f5b91600640 1 -- 172.21.15.116:0/751046893 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5b880cdfa0 con 0x7f5b941037f0 2024-03-04T07:52:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.977+0000 7f5b93e00640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b7c0a1d80 0x7f5b7c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:12.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.977+0000 7f5b93e00640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b7c0a1d80 0x7f5b7c0a4240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5b9419af40 tx=0x7f5b80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:12.980+0000 7f5b91600640 1 -- 172.21.15.116:0/751046893 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5b8809b390 con 0x7f5b941037f0 2024-03-04T07:52:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.165+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b940621d0 con 0x7f5b941037f0 2024-03-04T07:52:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.166+0000 7f5b91600640 1 -- 172.21.15.116:0/751046893 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b8809ac90 con 0x7f5b941037f0 2024-03-04T07:52:13.166 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:52:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b7c0a1d80 msgr2=0x7f5b7c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b7c0a1d80 0x7f5b7c0a4240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5b9419af40 tx=0x7f5b80005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 msgr2=0x7f5b94199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 0x7f5b94199a30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800e9c0 tx=0x7f5b8800ee80 comp rx=0 tx=0).stop 2024-03-04T07:52:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 shutdown_connections 2024-03-04T07:52:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5b941053a0 0x7f5b941a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f5b7c0a1d80 0x7f5b7c0a4240 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b941049e0 0x7f5b94199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 --2- 172.21.15.116:0/751046893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b941037f0 0x7f5b94199a30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 >> 172.21.15.116:0/751046893 conn(0x7f5b940fed80 msgr2=0x7f5b94107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 shutdown_connections 2024-03-04T07:52:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.170+0000 7f5b9b600640 1 -- 172.21.15.116:0/751046893 wait complete. 2024-03-04T07:52:13.185 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:12 smithi116 ceph-mon[32150]: pgmap v744: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-03-04T07:52:13.185 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:12 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2902353205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:13.189 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:52:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:12 smithi152 ceph-mon[37154]: pgmap v744: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-03-04T07:52:13.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:12 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2902353205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:12 smithi133 ceph-mon[36505]: pgmap v744: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-03-04T07:52:13.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:12 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2902353205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 -- 172.21.15.116:0/1206469523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 msgr2=0x7f628c108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 --2- 172.21.15.116:0/1206469523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c108010 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f627c009900 tx=0x7f627c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 -- 172.21.15.116:0/1206469523 shutdown_connections 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 --2- 172.21.15.116:0/1206469523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628c1097a0 0x7f628c110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 --2- 172.21.15.116:0/1206469523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f628c108de0 0x7f628c109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 --2- 172.21.15.116:0/1206469523 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c108010 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 -- 172.21.15.116:0/1206469523 >> 172.21.15.116:0/1206469523 conn(0x7f628c076b90 msgr2=0x7f628c078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 -- 172.21.15.116:0/1206469523 shutdown_connections 2024-03-04T07:52:13.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.395+0000 7f6290c00640 1 -- 172.21.15.116:0/1206469523 wait complete. 2024-03-04T07:52:13.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 Processor -- start 2024-03-04T07:52:13.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 -- start start 2024-03-04T07:52:13.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f628c108de0 0x7f628c19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628c1097a0 0x7f628c1a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c113860 con 0x7f628c1097a0 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c1136e0 con 0x7f628c107bf0 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6290c00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c1139e0 con 0x7f628c108de0 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f6289e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.397+0000 7f628a800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628c1097a0 0x7f628c1a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c19df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42320/0 (socket says 172.21.15.116:42320) 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 -- 172.21.15.116:0/238952906 learned_addr learned my addr 172.21.15.116:0/238952906 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6281400640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f628c108de0 0x7f628c19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 -- 172.21.15.116:0/238952906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f628c108de0 msgr2=0x7f628c19e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f628c108de0 0x7f628c19e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 -- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628c1097a0 msgr2=0x7f628c1a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628c1097a0 0x7f628c1a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 -- 172.21.15.116:0/238952906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c1a5b80 con 0x7f628c107bf0 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6281400640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f628c108de0 0x7f628c19e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6289e00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c19df60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f627c000c00 tx=0x7f627c02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6283400640 1 -- 172.21.15.116:0/238952906 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c041070 con 0x7f628c107bf0 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6283400640 1 -- 172.21.15.116:0/238952906 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f627c004450 con 0x7f628c107bf0 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c1a5e10 con 0x7f628c107bf0 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.398+0000 7f6283400640 1 -- 172.21.15.116:0/238952906 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c034440 con 0x7f628c107bf0 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.399+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c1a6220 con 0x7f628c107bf0 2024-03-04T07:52:13.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.400+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c10ef30 con 0x7f628c107bf0 2024-03-04T07:52:13.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.401+0000 7f6283400640 1 -- 172.21.15.116:0/238952906 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f627c044070 con 0x7f628c107bf0 2024-03-04T07:52:13.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.403+0000 7f6283400640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62700a1d80 0x7f62700a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.403+0000 7f6283400640 1 -- 172.21.15.116:0/238952906 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f627c0ef930 con 0x7f628c107bf0 2024-03-04T07:52:13.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.403+0000 7f6281400640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62700a1d80 0x7f62700a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.403+0000 7f6281400640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62700a1d80 0x7f62700a4240 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f628c19f470 tx=0x7f6274005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:13.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.407+0000 7f6283400640 1 -- 172.21.15.116:0/238952906 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f627c0bcdb0 con 0x7f628c107bf0 2024-03-04T07:52:13.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.604+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f628c062310 con 0x7f628c107bf0 2024-03-04T07:52:13.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.604+0000 7f6283400640 1 -- 172.21.15.116:0/238952906 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f627c0bc6b0 con 0x7f628c107bf0 2024-03-04T07:52:13.605 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:52:13.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62700a1d80 msgr2=0x7f62700a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62700a1d80 0x7f62700a4240 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f628c19f470 tx=0x7f6274005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:13.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 msgr2=0x7f628c19df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c19df60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f627c000c00 tx=0x7f627c02c640 comp rx=0 tx=0).stop 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 shutdown_connections 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628c1097a0 0x7f628c1a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f62700a1d80 0x7f62700a4240 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f628c108de0 0x7f628c19e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 --2- 172.21.15.116:0/238952906 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f628c107bf0 0x7f628c19df60 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 >> 172.21.15.116:0/238952906 conn(0x7f628c076b90 msgr2=0x7f628c10c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 shutdown_connections 2024-03-04T07:52:13.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.607+0000 7f6290c00640 1 -- 172.21.15.116:0/238952906 wait complete. 2024-03-04T07:52:13.624 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:52:13.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 -- 172.21.15.116:0/2736135808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd550103810 msgr2=0x7fd550103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 --2- 172.21.15.116:0/2736135808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd550103810 0x7fd550103c30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd544009ca0 tx=0x7fd54402c0f0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 -- 172.21.15.116:0/2736135808 shutdown_connections 2024-03-04T07:52:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 --2- 172.21.15.116:0/2736135808 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5501053c0 0x7fd55010bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 --2- 172.21.15.116:0/2736135808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd550104a00 0x7fd550104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 --2- 172.21.15.116:0/2736135808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd550103810 0x7fd550103c30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 -- 172.21.15.116:0/2736135808 >> 172.21.15.116:0/2736135808 conn(0x7fd5500fed80 msgr2=0x7fd5501011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 -- 172.21.15.116:0/2736135808 shutdown_connections 2024-03-04T07:52:13.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.838+0000 7fd558400640 1 -- 172.21.15.116:0/2736135808 wait complete. 2024-03-04T07:52:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.839+0000 7fd558400640 1 Processor -- start 2024-03-04T07:52:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.839+0000 7fd558400640 1 -- start start 2024-03-04T07:52:13.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd558400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd550103810 0x7fd550199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd558400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd550104a00 0x7fd55019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd558400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5501053c0 0x7fd5501a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd558400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55010f460 con 0x7fd5501053c0 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd558400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55010f2e0 con 0x7fd550103810 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd558400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55010f5e0 con 0x7fd550104a00 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd550103810 0x7fd550199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd550103810 0x7fd550199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42338/0 (socket says 172.21.15.116:42338) 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 -- 172.21.15.116:0/3090503297 learned_addr learned my addr 172.21.15.116:0/3090503297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd556400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5501053c0 0x7fd5501a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555000640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd550104a00 0x7fd55019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 -- 172.21.15.116:0/3090503297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd550104a00 msgr2=0x7fd55019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd550104a00 0x7fd55019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 -- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5501053c0 msgr2=0x7fd5501a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5501053c0 0x7fd5501a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd555a00640 1 -- 172.21.15.116:0/3090503297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5501a1750 con 0x7fd550103810 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.840+0000 7fd556400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5501053c0 0x7fd5501a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.841+0000 7fd555a00640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd550103810 0x7fd550199b30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd5440324c0 tx=0x7fd544032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.841+0000 7fd54e000640 1 -- 172.21.15.116:0/3090503297 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544041020 con 0x7fd550103810 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.841+0000 7fd54e000640 1 -- 172.21.15.116:0/3090503297 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd544034cd0 con 0x7fd550103810 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.841+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5501a19e0 con 0x7fd550103810 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.841+0000 7fd54e000640 1 -- 172.21.15.116:0/3090503297 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544039040 con 0x7fd550103810 2024-03-04T07:52:13.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.841+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5501a1f20 con 0x7fd550103810 2024-03-04T07:52:13.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.842+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55010abb0 con 0x7fd550103810 2024-03-04T07:52:13.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.844+0000 7fd54e000640 1 -- 172.21.15.116:0/3090503297 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd544044020 con 0x7fd550103810 2024-03-04T07:52:13.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.845+0000 7fd54e000640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd5380a1e00 0x7fd5380a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:13.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.845+0000 7fd54e000640 1 -- 172.21.15.116:0/3090503297 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd5440efcf0 con 0x7fd550103810 2024-03-04T07:52:13.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.845+0000 7fd555000640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd5380a1e00 0x7fd5380a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:13.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.846+0000 7fd555000640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd5380a1e00 0x7fd5380a42c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd53c005d00 tx=0x7fd53c005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:13.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:13.849+0000 7fd54e000640 1 -- 172.21.15.116:0/3090503297 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd54404b020 con 0x7fd550103810 2024-03-04T07:52:14.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.049+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd5500621d0 con 0x7fd550103810 2024-03-04T07:52:14.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.050+0000 7fd54e000640 1 -- 172.21.15.116:0/3090503297 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd5440bd170 con 0x7fd550103810 2024-03-04T07:52:14.050 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.052+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd5380a1e00 msgr2=0x7fd5380a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.052+0000 7fd558400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd5380a1e00 0x7fd5380a42c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd53c005d00 tx=0x7fd53c005c50 comp rx=0 tx=0).stop 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.052+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd550103810 msgr2=0x7fd550199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.052+0000 7fd558400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd550103810 0x7fd550199b30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd5440324c0 tx=0x7fd544032f40 comp rx=0 tx=0).stop 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 shutdown_connections 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5501053c0 0x7fd5501a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd5380a1e00 0x7fd5380a42c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd550104a00 0x7fd55019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 --2- 172.21.15.116:0/3090503297 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd550103810 0x7fd550199b30 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 >> 172.21.15.116:0/3090503297 conn(0x7fd5500fed80 msgr2=0x7fd550107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:14.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 shutdown_connections 2024-03-04T07:52:14.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.053+0000 7fd558400640 1 -- 172.21.15.116:0/3090503297 wait complete. 2024-03-04T07:52:14.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-04T07:52:14.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:13 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/751046893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:52:14.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:13 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/238952906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:52:14.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:13 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/751046893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:52:14.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:13 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/238952906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:52:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.281+0000 7fc13e800640 1 -- 172.21.15.116:0/180614308 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381027e0 msgr2=0x7fc138102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.281+0000 7fc13e800640 1 --2- 172.21.15.116:0/180614308 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381027e0 0x7fc138102c60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc124009900 tx=0x7fc12402bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.282+0000 7fc13e800640 1 -- 172.21.15.116:0/180614308 shutdown_connections 2024-03-04T07:52:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.282+0000 7fc13e800640 1 --2- 172.21.15.116:0/180614308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1381031a0 0x7fc138109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.282+0000 7fc13e800640 1 --2- 172.21.15.116:0/180614308 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381027e0 0x7fc138102c60 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.282+0000 7fc13e800640 1 --2- 172.21.15.116:0/180614308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1381015f0 0x7fc138101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.282+0000 7fc13e800640 1 -- 172.21.15.116:0/180614308 >> 172.21.15.116:0/180614308 conn(0x7fc1380fcb80 msgr2=0x7fc1380fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.282+0000 7fc13e800640 1 -- 172.21.15.116:0/180614308 shutdown_connections 2024-03-04T07:52:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.282+0000 7fc13e800640 1 -- 172.21.15.116:0/180614308 wait complete. 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 Processor -- start 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 -- start start 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381015f0 0x7fc1381978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1381027e0 0x7fc138197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1381031a0 0x7fc13819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13810d270 con 0x7fc1381027e0 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13810d0f0 con 0x7fc1381015f0 2024-03-04T07:52:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.283+0000 7fc13e800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13810d3f0 con 0x7fc1381031a0 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381015f0 0x7fc1381978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381015f0 0x7fc1381978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:42360/0 (socket says 172.21.15.116:42360) 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 -- 172.21.15.116:0/1264706957 learned_addr learned my addr 172.21.15.116:0/1264706957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc136a00640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1381027e0 0x7fc138197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137e00640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1381031a0 0x7fc13819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 -- 172.21.15.116:0/1264706957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1381031a0 msgr2=0x7fc13819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1381031a0 0x7fc13819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 -- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1381027e0 msgr2=0x7fc138197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1381027e0 0x7fc138197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 -- 172.21.15.116:0/1264706957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13819f500 con 0x7fc1381015f0 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137e00640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1381031a0 0x7fc13819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc136a00640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1381027e0 0x7fc138197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.284+0000 7fc137400640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381015f0 0x7fc1381978e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc12800e9c0 tx=0x7fc12800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.285+0000 7fc12fe00640 1 -- 172.21.15.116:0/1264706957 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12800cde0 con 0x7fc1381015f0 2024-03-04T07:52:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.285+0000 7fc12fe00640 1 -- 172.21.15.116:0/1264706957 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc128010910 con 0x7fc1381015f0 2024-03-04T07:52:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.285+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13819f7f0 con 0x7fc1381015f0 2024-03-04T07:52:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.285+0000 7fc12fe00640 1 -- 172.21.15.116:0/1264706957 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128018a00 con 0x7fc1381015f0 2024-03-04T07:52:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.285+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13819fd60 con 0x7fc1381015f0 2024-03-04T07:52:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.286+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc138108910 con 0x7fc1381015f0 2024-03-04T07:52:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.288+0000 7fc12fe00640 1 -- 172.21.15.116:0/1264706957 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fc12801f020 con 0x7fc1381015f0 2024-03-04T07:52:14.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.289+0000 7fc12fe00640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc11c0a1e00 0x7fc11c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:14.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.289+0000 7fc12fe00640 1 -- 172.21.15.116:0/1264706957 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc1280ce020 con 0x7fc1381015f0 2024-03-04T07:52:14.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.289+0000 7fc136a00640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc11c0a1e00 0x7fc11c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:14.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.290+0000 7fc136a00640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc11c0a1e00 0x7fc11c0a42c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc124006010 tx=0x7fc12402c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:14.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.293+0000 7fc12fe00640 1 -- 172.21.15.116:0/1264706957 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc12801e2c0 con 0x7fc1381015f0 2024-03-04T07:52:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:13 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/751046893' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:52:14.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:13 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/238952906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:52:14.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.541+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fc13802d2a0 con 0x7fc1381015f0 2024-03-04T07:52:14.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.551+0000 7fc12fe00640 1 -- 172.21.15.116:0/1264706957 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+736399 (secure 0 0 0) 0x7fc12809b490 con 0x7fc1381015f0 2024-03-04T07:52:14.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 18, 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout: "active_gid": 14152, 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout: "active_name": "x", 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addrs": { 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-04T07:52:14.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6800", 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 142890918 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6801", 2024-03-04T07:52:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 142890918 2024-03-04T07:52:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addr": "172.21.15.116:6801/142890918", 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "active_change": "2024-03-04T07:29:57.331964+0000", 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "active_mgr_features": 4540701547738038271, 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "available": true, 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "standbys": [ 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 24109, 2024-03-04T07:52:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "z", 2024-03-04T07:52:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-04T07:52:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-04T07:52:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-04T07:52:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.577 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.578 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.579 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.580 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.581 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-04T07:52:14.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.590 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.592 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.593 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-04T07:52:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-04T07:52:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.594 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.595 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-04T07:52:14.596 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-04T07:52:14.597 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.598 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:14.599 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.601 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.602 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.603 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.604 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.605 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.610 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.611 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.612 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.613 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-04T07:52:14.616 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.617 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.618 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-04T07:52:14.619 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.620 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.621 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.622 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-04T07:52:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-04T07:52:14.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.625 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.630 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.632 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.633 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.634 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.635 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.636 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.637 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.638 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.640 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-04T07:52:14.641 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-04T07:52:14.642 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.643 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.644 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.647 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.650 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.651 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-04T07:52:14.652 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-04T07:52:14.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.654 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.655 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.656 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.657 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-04T07:52:14.658 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.659 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.661 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.662 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-04T07:52:14.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.666 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-04T07:52:14.667 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-04T07:52:14.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-04T07:52:14.669 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.670 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.672 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.673 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.674 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.675 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.676 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-04T07:52:14.677 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.678 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.679 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.680 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.681 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.682 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-04T07:52:14.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-04T07:52:14.684 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.685 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.686 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.687 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.688 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.689 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.690 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-04T07:52:14.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.693 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.696 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-04T07:52:14.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.699 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.701 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.702 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.703 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.705 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.706 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.708 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.709 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-04T07:52:14.710 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.712 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.714 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.715 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.720 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-04T07:52:14.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-04T07:52:14.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.720 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-04T07:52:14.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-04T07:52:14.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.723 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.724 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-04T07:52:14.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-04T07:52:14.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-04T07:52:14.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-04T07:52:14.728 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.740 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.741 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.742 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.743 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.744 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.746 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.749 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.750 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.751 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.752 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.755 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.757 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-04T07:52:14.758 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-04T07:52:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-04T07:52:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.766 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.768 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.769 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.770 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-04T07:52:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-04T07:52:14.772 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.773 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.774 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-04T07:52:14.775 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-04T07:52:14.776 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.777 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.778 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.779 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.780 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.782 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-04T07:52:14.783 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:14.784 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.785 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.786 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.787 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.788 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-04T07:52:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.790 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.791 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.792 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.793 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.798 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.799 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.801 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.802 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.803 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.804 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.806 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-04T07:52:14.811 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.812 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.820 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.822 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.823 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.824 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-04T07:52:14.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.827 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-04T07:52:14.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-04T07:52:14.829 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.830 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.831 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.833 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.834 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-04T07:52:14.835 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.836 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.837 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.839 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.841 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.842 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.843 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.844 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.845 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.846 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.847 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.848 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.849 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.850 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.852 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-04T07:52:14.853 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.854 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.855 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.856 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.857 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.860 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.861 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.862 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.869 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.871 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-04T07:52:14.872 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.880 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.882 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.884 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.886 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.887 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.895 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-04T07:52:14.896 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.898 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.899 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.900 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.901 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.905 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.907 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.908 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.913 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.919 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.920 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.920 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.920 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.920 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.929 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.931 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.932 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-04T07:52:14.932 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.932 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.932 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.933 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.935 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.936 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.936 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.937 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.938 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-04T07:52:14.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.942 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.943 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.945 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.947 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-04T07:52:14.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-04T07:52:14.947 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.947 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.947 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.947 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.947 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.948 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.948 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.948 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.948 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.949 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-04T07:52:14.949 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-04T07:52:14.949 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.949 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.949 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.949 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.950 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.950 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.950 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.950 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.950 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.950 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.951 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-04T07:52:14.953 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.954 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.956 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.958 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-04T07:52:14.958 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.958 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.958 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-04T07:52:14.960 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.961 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.962 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.963 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.964 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.965 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.966 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.967 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.967 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.968 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.969 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.970 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.972 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.973 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.973 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.973 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.973 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.973 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.973 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.974 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.974 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.974 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.974 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.974 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.974 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.975 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.975 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.976 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.980 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.980 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.980 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.980 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.981 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-04T07:52:14.982 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.985 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.986 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.987 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:14.988 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-04T07:52:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.997 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:14.998 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:14.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.000 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.000 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.000 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.000 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.000 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.001 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.002 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.002 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.002 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.002 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.002 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-04T07:52:15.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.004 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.005 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.007 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.008 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.009 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.010 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.012 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.013 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.013 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.013 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.013 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-04T07:52:15.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-04T07:52:15.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-04T07:52:15.014 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.016 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.017 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.017 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.017 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.017 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.017 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.017 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.018 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.020 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.020 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.020 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.020 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.020 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.022 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.023 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.024 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.026 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.027 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.028 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.029 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.030 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.031 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.032 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.034 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.035 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.036 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.037 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.039 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.040 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.041 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.042 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.043 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-04T07:52:15.044 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 24110, 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "y", 2024-03-04T07:52:15.045 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.047 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.048 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.049 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.050 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.051 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.053 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.054 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.055 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.056 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-04T07:52:15.057 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-04T07:52:15.058 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.060 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-04T07:52:15.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-04T07:52:15.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-04T07:52:15.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-04T07:52:15.064 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-04T07:52:15.065 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-04T07:52:15.066 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-04T07:52:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.068 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.069 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.070 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.071 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.072 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.074 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.076 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-04T07:52:15.080 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-04T07:52:15.081 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.082 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.083 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.084 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.085 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.086 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.087 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.088 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.089 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.090 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-04T07:52:15.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-04T07:52:15.095 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-04T07:52:15.096 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-04T07:52:15.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-04T07:52:15.098 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-04T07:52:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-04T07:52:15.100 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-04T07:52:15.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.102 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.103 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-04T07:52:15.104 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-04T07:52:15.105 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-04T07:52:15.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.107 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-04T07:52:15.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-04T07:52:15.109 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-04T07:52:15.110 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-04T07:52:15.111 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.112 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.115 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.116 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.117 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.118 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.119 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.121 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.122 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.123 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.124 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.125 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.126 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.127 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.128 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.131 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.132 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.133 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.134 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.135 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-04T07:52:15.137 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.138 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.139 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.140 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-04T07:52:15.141 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.143 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.143 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-04T07:52:15.143 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-04T07:52:15.143 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-04T07:52:15.144 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.146 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.147 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.148 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.149 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.151 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.152 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.153 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.154 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.155 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.156 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.157 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.158 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.159 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.160 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.161 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.162 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.164 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.166 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.168 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.169 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.170 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.173 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-04T07:52:15.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-04T07:52:15.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-04T07:52:15.177 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-04T07:52:15.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-04T07:52:15.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-04T07:52:15.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-04T07:52:15.181 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-04T07:52:15.182 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.183 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.184 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.185 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-04T07:52:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.188 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.189 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.190 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.192 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.193 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.194 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.195 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.196 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.197 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.198 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.199 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.200 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.201 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.202 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.203 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.204 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.206 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.207 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.208 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.209 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-04T07:52:15.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.213 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.214 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.215 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.217 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.218 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.219 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.220 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-04T07:52:15.221 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.222 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.223 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.224 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.228 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.229 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.230 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.231 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.232 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.233 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.234 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.235 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.236 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.238 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.239 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.241 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.242 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.243 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.244 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.245 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.246 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.248 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.249 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.250 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.251 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.253 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.254 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.255 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.256 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-04T07:52:15.262 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.263 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.264 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.265 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.266 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.268 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-04T07:52:15.269 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.270 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.274 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.275 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.276 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.283 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.290 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.291 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.292 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.293 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.294 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.295 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.296 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-04T07:52:15.297 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.304 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.305 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.306 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.307 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.308 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-04T07:52:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.310 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.311 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.312 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.313 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.314 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.315 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.318 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.319 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.320 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.321 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.322 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.323 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.324 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.325 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.326 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.327 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.329 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-04T07:52:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-04T07:52:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.337 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.338 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.339 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.340 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-04T07:52:15.341 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.342 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.343 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.344 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.345 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.350 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.351 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.352 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.353 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.354 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.355 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.356 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.358 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.361 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.362 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.364 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:15.365 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:15.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.368 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.370 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.371 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.372 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.373 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.374 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.375 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.376 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.377 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-04T07:52:15.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.379 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.380 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.381 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.382 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.383 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.384 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.386 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.387 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.388 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.388 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.388 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.388 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.388 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.388 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.389 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-04T07:52:15.390 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-04T07:52:15.391 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-04T07:52:15.392 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-04T07:52:15.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.394 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.395 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-04T07:52:15.396 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.397 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.398 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-04T07:52:15.399 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.401 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.402 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.404 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.405 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.408 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.409 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.410 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.411 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.412 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.413 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.414 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.418 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.419 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.420 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.421 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.424 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.425 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.426 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.427 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.428 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.429 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.430 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.431 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.432 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.433 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.434 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.435 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.436 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.437 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.438 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.439 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.441 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.443 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.444 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.445 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.446 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.447 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.448 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.449 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.451 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.452 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.453 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.454 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.456 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.457 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.458 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-04T07:52:15.460 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.461 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.462 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.467 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.468 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-04T07:52:15.469 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.470 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "modules": [ 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-04T07:52:15.471 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "iostat", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "nfs", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "restful" 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.472 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.473 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.474 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.476 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.477 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.478 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.479 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.480 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.481 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.482 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.483 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.484 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-04T07:52:15.486 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.487 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.488 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.489 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.490 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.491 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.493 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.495 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.496 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.497 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.498 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.499 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.500 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.501 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.502 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.503 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-04T07:52:15.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-04T07:52:15.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.507 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.508 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.515 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.516 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-04T07:52:15.517 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-04T07:52:15.518 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.519 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.521 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.522 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.523 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.524 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.525 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.526 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-04T07:52:15.527 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-04T07:52:15.528 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.529 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-04T07:52:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.531 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.533 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.536 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-04T07:52:15.536 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-04T07:52:15.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.536 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.536 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-04T07:52:15.540 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-04T07:52:15.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.543 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.544 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.545 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.546 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.547 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.548 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.549 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-04T07:52:15.550 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-04T07:52:15.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.552 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.553 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.554 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.555 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-04T07:52:15.556 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-04T07:52:15.557 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.558 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.559 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.560 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.561 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.563 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.564 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.565 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.566 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.567 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.568 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.569 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.571 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.572 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.574 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.575 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.578 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.580 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.581 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.582 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.583 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.589 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.590 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-04T07:52:15.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.592 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-04T07:52:15.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-04T07:52:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-04T07:52:15.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.597 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.604 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.605 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-04T07:52:15.609 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.611 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.612 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.613 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.615 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.616 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.617 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.619 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.621 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.624 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.625 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.626 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.627 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.629 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.630 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-04T07:52:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-04T07:52:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-04T07:52:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.638 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.639 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.640 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.641 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.642 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.643 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.644 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.645 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-04T07:52:15.646 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.649 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.654 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.655 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.656 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-04T07:52:15.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.658 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.659 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-04T07:52:15.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-04T07:52:15.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-04T07:52:15.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.665 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.666 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.667 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-04T07:52:15.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-04T07:52:15.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-04T07:52:15.671 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-04T07:52:15.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.673 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-04T07:52:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.676 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.677 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.678 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.681 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.682 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.684 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.685 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-04T07:52:15.686 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-04T07:52:15.687 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.689 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.691 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.692 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.694 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.695 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.696 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.697 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-04T07:52:15.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-04T07:52:15.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-04T07:52:15.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-04T07:52:15.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-04T07:52:15.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.706 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.710 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.711 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.712 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.713 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.714 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.716 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.717 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.718 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.719 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.721 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.722 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.723 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.725 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.726 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-04T07:52:15.727 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.728 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.729 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.732 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.735 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.736 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.737 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.738 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.743 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.743 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.743 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.745 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.746 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.747 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.748 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.749 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.750 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.751 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.752 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.753 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.754 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.755 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.758 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.759 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.760 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.761 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.762 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.763 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-04T07:52:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.766 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.767 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.768 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.769 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.770 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.772 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.775 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.776 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.777 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.778 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.779 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-04T07:52:15.780 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.781 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.782 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.783 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.784 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.785 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.786 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.787 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.789 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.790 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.791 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.792 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.793 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-04T07:52:15.794 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.795 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.796 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.797 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.798 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.800 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.801 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.802 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.803 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.804 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.805 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.806 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.809 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.810 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.813 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.814 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.815 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.816 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.817 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.818 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.819 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.820 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.821 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.822 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.824 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.825 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-04T07:52:15.827 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.828 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.829 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.830 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.831 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.832 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.834 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-04T07:52:15.835 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.837 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.838 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-04T07:52:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.841 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.842 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.843 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.845 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.847 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.848 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.849 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.851 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.852 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.853 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.854 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.857 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.858 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.859 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.860 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.862 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-04T07:52:15.863 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.865 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.867 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.869 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.870 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.872 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.873 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.875 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-04T07:52:15.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.877 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.880 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.882 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-04T07:52:15.883 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.884 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.885 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.888 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-04T07:52:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-04T07:52:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.898 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.901 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.903 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.904 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.907 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.911 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-04T07:52:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.913 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.914 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.916 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.917 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.918 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.920 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.921 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.923 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-04T07:52:15.924 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-04T07:52:15.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.927 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.928 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-04T07:52:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "services": { 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard": "https://172.21.15.116:8443/" 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "always_on_modules": { 2024-03-04T07:52:15.930 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus": [ 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.931 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific": [ 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.932 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy": [ 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "reef": [ 2024-03-04T07:52:15.933 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "squid": [ 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-04T07:52:15.934 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "last_failure_osd_epoch": 3, 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: "active_clients": [ 2024-03-04T07:52:15.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1106698012 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "libcephsqlite", 2024-03-04T07:52:15.936 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3861295184 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-04T07:52:15.937 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 9312237 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 855763577 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.938 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.939 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-04T07:52:15.939 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-04T07:52:15.939 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-04T07:52:15.940 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:14 smithi133 ceph-mon[36505]: pgmap v745: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.9 KiB/s wr, 6 op/s 2024-03-04T07:52:15.940 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:14 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3090503297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:52:15.940 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:14 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1264706957' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-04T07:52:15.941 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:14 smithi152 ceph-mon[37154]: pgmap v745: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.9 KiB/s wr, 6 op/s 2024-03-04T07:52:15.941 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:14 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3090503297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:52:15.941 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:14 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1264706957' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.561+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc11c0a1e00 msgr2=0x7fc11c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.561+0000 7fc13e800640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc11c0a1e00 0x7fc11c0a42c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc124006010 tx=0x7fc12402c8a0 comp rx=0 tx=0).stop 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.561+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381015f0 msgr2=0x7fc1381978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.561+0000 7fc13e800640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381015f0 0x7fc1381978e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc12800e9c0 tx=0x7fc12800ee80 comp rx=0 tx=0).stop 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 shutdown_connections 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fc11c0a1e00 0x7fc11c0a42c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1381031a0 0x7fc13819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:15.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1381027e0 0x7fc138197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 --2- 172.21.15.116:0/1264706957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc1381015f0 0x7fc1381978e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 >> 172.21.15.116:0/1264706957 conn(0x7fc1380fcb80 msgr2=0x7fc138105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 shutdown_connections 2024-03-04T07:52:15.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:14.562+0000 7fc13e800640 1 -- 172.21.15.116:0/1264706957 wait complete. 2024-03-04T07:52:15.948 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-04T07:52:15.958 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-04T07:52:15.959 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:52:15.959 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T07:52:15.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:14 smithi116 ceph-mon[32150]: pgmap v745: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.9 KiB/s wr, 6 op/s 2024-03-04T07:52:15.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3090503297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:52:15.959 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:14 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1264706957' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-04T07:52:16.020 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-04T07:52:16.020 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T07:52:16.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:52:16.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 -- 172.21.15.116:0/535687133 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 msgr2=0x7fbca8077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 --2- 172.21.15.116:0/535687133 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 0x7fbca8077630 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fbc94009900 tx=0x7fbc9402bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 -- 172.21.15.116:0/535687133 shutdown_connections 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 --2- 172.21.15.116:0/535687133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8077b70 0x7fbca8078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 --2- 172.21.15.116:0/535687133 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 0x7fbca8077630 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 --2- 172.21.15.116:0/535687133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca8075f90 0x7fbca80763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 -- 172.21.15.116:0/535687133 >> 172.21.15.116:0/535687133 conn(0x7fbca80fed80 msgr2=0x7fbca81011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 -- 172.21.15.116:0/535687133 shutdown_connections 2024-03-04T07:52:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.274+0000 7fbcaf400640 1 -- 172.21.15.116:0/535687133 wait complete. 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 Processor -- start 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 -- start start 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8075f90 0x7fbca819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 0x7fbca819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca8077b70 0x7fbca81a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca81138e0 con 0x7fbca8075f90 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8113760 con 0x7fbca80771d0 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaf400640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8113a60 con 0x7fbca8077b70 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8075f90 0x7fbca819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbca7e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 0x7fbca819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8075f90 0x7fbca819df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41978/0 (socket says 172.21.15.116:41978) 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.276+0000 7fbcaca00640 1 -- 172.21.15.116:0/4135518188 learned_addr learned my addr 172.21.15.116:0/4135518188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbcad400640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca8077b70 0x7fbca81a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca7e00640 1 -- 172.21.15.116:0/4135518188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca8077b70 msgr2=0x7fbca81a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca7e00640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca8077b70 0x7fbca81a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca7e00640 1 -- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8075f90 msgr2=0x7fbca819df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca7e00640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8075f90 0x7fbca819df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca7e00640 1 -- 172.21.15.116:0/4135518188 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca81a5b20 con 0x7fbca80771d0 2024-03-04T07:52:16.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca7e00640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 0x7fbca819e440 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fbc94036040 tx=0x7fbc9402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:16.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca5600640 1 -- 172.21.15.116:0/4135518188 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94043020 con 0x7fbca80771d0 2024-03-04T07:52:16.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca5600640 1 -- 172.21.15.116:0/4135518188 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc940054e0 con 0x7fbca80771d0 2024-03-04T07:52:16.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbca5600640 1 -- 172.21.15.116:0/4135518188 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94003ca0 con 0x7fbca80771d0 2024-03-04T07:52:16.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.277+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca81a5db0 con 0x7fbca80771d0 2024-03-04T07:52:16.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.278+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca81a6220 con 0x7fbca80771d0 2024-03-04T07:52:16.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.279+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca8110310 con 0x7fbca80771d0 2024-03-04T07:52:16.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.280+0000 7fbca5600640 1 -- 172.21.15.116:0/4135518188 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fbc94046020 con 0x7fbca80771d0 2024-03-04T07:52:16.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.281+0000 7fbca5600640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbc900a1e00 0x7fbc900a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.282+0000 7fbca5600640 1 -- 172.21.15.116:0/4135518188 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fbc940efe40 con 0x7fbca80771d0 2024-03-04T07:52:16.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.282+0000 7fbcaca00640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbc900a1e00 0x7fbc900a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.282+0000 7fbcaca00640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbc900a1e00 0x7fbc900a42c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00bca0 tx=0x7fbc9c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.285+0000 7fbca5600640 1 -- 172.21.15.116:0/4135518188 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbc9404d020 con 0x7fbca80771d0 2024-03-04T07:52:16.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.483+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbca8062200 con 0x7fbca80771d0 2024-03-04T07:52:16.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.485+0000 7fbca5600640 1 -- 172.21.15.116:0/4135518188 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7fbc940bd230 con 0x7fbca80771d0 2024-03-04T07:52:16.486 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:52:16.488 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:52:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbc900a1e00 msgr2=0x7fbc900a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbc900a1e00 0x7fbc900a42c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00bca0 tx=0x7fbc9c00b410 comp rx=0 tx=0).stop 2024-03-04T07:52:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 msgr2=0x7fbca819e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 0x7fbca819e440 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fbc94036040 tx=0x7fbc9402c6c0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 shutdown_connections 2024-03-04T07:52:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fbc900a1e00 0x7fbc900a42c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca8077b70 0x7fbca81a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbca80771d0 0x7fbca819e440 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 --2- 172.21.15.116:0/4135518188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8075f90 0x7fbca819df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 >> 172.21.15.116:0/4135518188 conn(0x7fbca80fed80 msgr2=0x7fbca8100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 shutdown_connections 2024-03-04T07:52:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.488+0000 7fbcaf400640 1 -- 172.21.15.116:0/4135518188 wait complete. 2024-03-04T07:52:16.509 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T07:32:42.025045+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T07:37:52.748569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T07:37:54.082948+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T07:52:16.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T07:52:16.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 -- 172.21.15.116:0/2043574355 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381014d0 msgr2=0x7f48381018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 --2- 172.21.15.116:0/2043574355 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381014d0 0x7f48381018f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f4828009900 tx=0x7f482802bc50 comp rx=0 tx=0).stop 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 -- 172.21.15.116:0/2043574355 shutdown_connections 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 --2- 172.21.15.116:0/2043574355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4838103080 0x7f4838109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 --2- 172.21.15.116:0/2043574355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48381026c0 0x7f4838102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 --2- 172.21.15.116:0/2043574355 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381014d0 0x7f48381018f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 -- 172.21.15.116:0/2043574355 >> 172.21.15.116:0/2043574355 conn(0x7f48380fca40 msgr2=0x7f48380fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 -- 172.21.15.116:0/2043574355 shutdown_connections 2024-03-04T07:52:16.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.741+0000 7f483d200640 1 -- 172.21.15.116:0/2043574355 wait complete. 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 Processor -- start 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 -- start start 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48381014d0 0x7f4838199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381026c0 0x7f4838199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4838103080 0x7f48381a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483810d180 con 0x7f4838103080 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483810d000 con 0x7f48381026c0 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.742+0000 7f483d200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483810d300 con 0x7f48381014d0 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381026c0 0x7f4838199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4837400640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48381014d0 0x7f4838199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381026c0 0x7f4838199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47060/0 (socket says 172.21.15.116:47060) 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 -- 172.21.15.116:0/3495295212 learned_addr learned my addr 172.21.15.116:0/3495295212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4837e00640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4838103080 0x7f48381a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 -- 172.21.15.116:0/3495295212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48381014d0 msgr2=0x7f4838199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48381014d0 0x7f4838199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 -- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4838103080 msgr2=0x7f48381a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4838103080 0x7f48381a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.743+0000 7f4836a00640 1 -- 172.21.15.116:0/3495295212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48381a1630 con 0x7f48381026c0 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.744+0000 7f4836a00640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381026c0 0x7f4838199f50 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f482400d950 tx=0x7f482400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.744+0000 7f482fe00640 1 -- 172.21.15.116:0/3495295212 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824017030 con 0x7f48381026c0 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.744+0000 7f482fe00640 1 -- 172.21.15.116:0/3495295212 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4824004020 con 0x7f48381026c0 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.744+0000 7f482fe00640 1 -- 172.21.15.116:0/3495295212 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48240047a0 con 0x7f48381026c0 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.744+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48381a1920 con 0x7f48381026c0 2024-03-04T07:52:16.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.744+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48381a1ec0 con 0x7f48381026c0 2024-03-04T07:52:16.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.745+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48381a2150 con 0x7f48381026c0 2024-03-04T07:52:16.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.748+0000 7f482fe00640 1 -- 172.21.15.116:0/3495295212 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f4824009a30 con 0x7f48381026c0 2024-03-04T07:52:16.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.749+0000 7f482fe00640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f481c0a1e00 0x7f481c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:16.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.749+0000 7f482fe00640 1 -- 172.21.15.116:0/3495295212 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f48240cd790 con 0x7f48381026c0 2024-03-04T07:52:16.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.749+0000 7f4837400640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f481c0a1e00 0x7f481c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:16.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.750+0000 7f4837400640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f481c0a1e00 0x7f481c0a42c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f482802c190 tx=0x7f4828000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.753+0000 7f482fe00640 1 -- 172.21.15.116:0/3495295212 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f482409ab80 con 0x7f48381026c0 2024-03-04T07:52:16.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.948+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48381a2320 con 0x7f48381026c0 2024-03-04T07:52:16.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.948+0000 7f482fe00640 1 -- 172.21.15.116:0/3495295212 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f482409a480 con 0x7f48381026c0 2024-03-04T07:52:16.949 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-04T07:52:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f481c0a1e00 msgr2=0x7f481c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f481c0a1e00 0x7f481c0a42c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f482802c190 tx=0x7f4828000bc0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381026c0 msgr2=0x7f4838199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381026c0 0x7f4838199f50 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f482400d950 tx=0x7f482400de10 comp rx=0 tx=0).stop 2024-03-04T07:52:16.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 shutdown_connections 2024-03-04T07:52:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4838103080 0x7f48381a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f48381026c0 0x7f4838199f50 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f481c0a1e00 0x7f481c0a42c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 --2- 172.21.15.116:0/3495295212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48381014d0 0x7f4838199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 >> 172.21.15.116:0/3495295212 conn(0x7f48380fca40 msgr2=0x7f4838073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 shutdown_connections 2024-03-04T07:52:16.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:16.951+0000 7f483d200640 1 -- 172.21.15.116:0/3495295212 wait complete. 2024-03-04T07:52:16.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T07:52:17.047 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:16 smithi116 ceph-mon[32150]: pgmap v746: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.6 KiB/s wr, 5 op/s 2024-03-04T07:52:17.047 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:16 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/4135518188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:17.047 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:16 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3495295212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:52:17.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 -- 172.21.15.116:0/3031404039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4108de0 msgr2=0x7f73c4109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 --2- 172.21.15.116:0/3031404039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4108de0 0x7f73c4109260 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009900 tx=0x7f73ac02bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:17.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 -- 172.21.15.116:0/3031404039 shutdown_connections 2024-03-04T07:52:17.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 --2- 172.21.15.116:0/3031404039 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 0x7f73c4110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 --2- 172.21.15.116:0/3031404039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4108de0 0x7f73c4109260 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 --2- 172.21.15.116:0/3031404039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73c4107bf0 0x7f73c4108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 -- 172.21.15.116:0/3031404039 >> 172.21.15.116:0/3031404039 conn(0x7f73c4076b90 msgr2=0x7f73c4078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 -- 172.21.15.116:0/3031404039 shutdown_connections 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.177+0000 7f73ca800640 1 -- 172.21.15.116:0/3031404039 wait complete. 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 Processor -- start 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 -- start start 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4107bf0 0x7f73c419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73c4108de0 0x7f73c419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 0x7f73c41a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c41138c0 con 0x7f73c4107bf0 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c4113740 con 0x7f73c41097a0 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.178+0000 7f73ca800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c4113a40 con 0x7f73c4108de0 2024-03-04T07:52:17.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4107bf0 0x7f73c419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 0x7f73c41a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4107bf0 0x7f73c419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42012/0 (socket says 172.21.15.116:42012) 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3400640 1 -- 172.21.15.116:0/1025814451 learned_addr learned my addr 172.21.15.116:0/1025814451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 0x7f73c41a54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47072/0 (socket says 172.21.15.116:47072) 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73bbe00640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73c4108de0 0x7f73c419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 -- 172.21.15.116:0/1025814451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73c4108de0 msgr2=0x7f73c419e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73c4108de0 0x7f73c419e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 -- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4107bf0 msgr2=0x7f73c419df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4107bf0 0x7f73c419df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 -- 172.21.15.116:0/1025814451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c41a5b60 con 0x7f73c41097a0 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73bbe00640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73c4108de0 0x7f73c419e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.179+0000 7f73c3e00640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 0x7f73c41a54f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00ec50 tx=0x7f73bc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.180+0000 7f73c0c00640 1 -- 172.21.15.116:0/1025814451 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc003b70 con 0x7f73c41097a0 2024-03-04T07:52:17.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.180+0000 7f73c0c00640 1 -- 172.21.15.116:0/1025814451 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73bc003d10 con 0x7f73c41097a0 2024-03-04T07:52:17.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.180+0000 7f73c0c00640 1 -- 172.21.15.116:0/1025814451 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc010890 con 0x7f73c41097a0 2024-03-04T07:52:17.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.180+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c41a5e50 con 0x7f73c41097a0 2024-03-04T07:52:17.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.180+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c41a62c0 con 0x7f73c41097a0 2024-03-04T07:52:17.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.181+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c410ef10 con 0x7f73c41097a0 2024-03-04T07:52:17.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.182+0000 7f73c0c00640 1 -- 172.21.15.116:0/1025814451 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f73bc003fe0 con 0x7f73c41097a0 2024-03-04T07:52:17.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.184+0000 7f73c0c00640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73a80a1e00 0x7f73a80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.184+0000 7f73c0c00640 1 -- 172.21.15.116:0/1025814451 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f73bc0ce3a0 con 0x7f73c41097a0 2024-03-04T07:52:17.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.184+0000 7f73c3400640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73a80a1e00 0x7f73a80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.184+0000 7f73c3400640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73a80a1e00 0x7f73a80a42c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f73b40058c0 tx=0x7f73b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:17.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.188+0000 7f73c0c00640 1 -- 172.21.15.116:0/1025814451 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f73bc09b790 con 0x7f73c41097a0 2024-03-04T07:52:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:16 smithi152 ceph-mon[37154]: pgmap v746: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.6 KiB/s wr, 5 op/s 2024-03-04T07:52:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:16 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/4135518188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:17.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:16 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3495295212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:52:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:16 smithi133 ceph-mon[36505]: pgmap v746: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.6 KiB/s wr, 5 op/s 2024-03-04T07:52:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:16 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/4135518188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:17.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:16 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3495295212' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T07:52:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.385+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73c402cff0 con 0x7f73c41097a0 2024-03-04T07:52:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.385+0000 7f73c0c00640 1 -- 172.21.15.116:0/1025814451 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73bc09b090 con 0x7f73c41097a0 2024-03-04T07:52:17.386 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-04T07:52:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73a80a1e00 msgr2=0x7f73a80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73a80a1e00 0x7f73a80a42c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f73b40058c0 tx=0x7f73b4005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 msgr2=0x7f73c41a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 0x7f73c41a54f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00ec50 tx=0x7f73bc00c6a0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 shutdown_connections 2024-03-04T07:52:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f73c41097a0 0x7f73c41a54f0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f73a80a1e00 0x7f73a80a42c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f73c4108de0 0x7f73c419e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 --2- 172.21.15.116:0/1025814451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73c4107bf0 0x7f73c419df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 >> 172.21.15.116:0/1025814451 conn(0x7f73c4076b90 msgr2=0x7f73c410c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 shutdown_connections 2024-03-04T07:52:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.388+0000 7f73ca800640 1 -- 172.21.15.116:0/1025814451 wait complete. 2024-03-04T07:52:17.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.616+0000 7f391e800640 1 -- 172.21.15.116:0/351132937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39180771d0 msgr2=0x7f3918077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.616+0000 7f391e800640 1 --2- 172.21.15.116:0/351132937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39180771d0 0x7f3918077630 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3904009900 tx=0x7f390402bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.617+0000 7f391e800640 1 -- 172.21.15.116:0/351132937 shutdown_connections 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.617+0000 7f391e800640 1 --2- 172.21.15.116:0/351132937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3918077b70 0x7f3918078410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.617+0000 7f391e800640 1 --2- 172.21.15.116:0/351132937 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f39180771d0 0x7f3918077630 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.617+0000 7f391e800640 1 --2- 172.21.15.116:0/351132937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3918075f90 0x7f39180763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.617+0000 7f391e800640 1 -- 172.21.15.116:0/351132937 >> 172.21.15.116:0/351132937 conn(0x7f39180fed80 msgr2=0x7f39181011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.617+0000 7f391e800640 1 -- 172.21.15.116:0/351132937 shutdown_connections 2024-03-04T07:52:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.617+0000 7f391e800640 1 -- 172.21.15.116:0/351132937 wait complete. 2024-03-04T07:52:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.618+0000 7f391e800640 1 Processor -- start 2024-03-04T07:52:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.618+0000 7f391e800640 1 -- start start 2024-03-04T07:52:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f391e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3918075f90 0x7f391819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f391e800640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39180771d0 0x7f391819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f391e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3918077b70 0x7f39181a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f391e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39181138e0 con 0x7f3918077b70 2024-03-04T07:52:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f391e800640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918113760 con 0x7f3918075f90 2024-03-04T07:52:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f391e800640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918113a60 con 0x7f39180771d0 2024-03-04T07:52:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3918075f90 0x7f391819df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3918077b70 0x7f39181a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3918075f90 0x7f391819df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47094/0 (socket says 172.21.15.116:47094) 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 -- 172.21.15.116:0/2517503919 learned_addr learned my addr 172.21.15.116:0/2517503919 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3916a00640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39180771d0 0x7f391819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 -- 172.21.15.116:0/2517503919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39180771d0 msgr2=0x7f391819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39180771d0 0x7f391819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 -- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3918077b70 msgr2=0x7f39181a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3918077b70 0x7f39181a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.619+0000 7f3917400640 1 -- 172.21.15.116:0/2517503919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39181a5b40 con 0x7f3918075f90 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.620+0000 7f3917400640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3918075f90 0x7f391819df20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f390800e9f0 tx=0x7f390800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.620+0000 7f390fe00640 1 -- 172.21.15.116:0/2517503919 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390800cd80 con 0x7f3918075f90 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.620+0000 7f390fe00640 1 -- 172.21.15.116:0/2517503919 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3908010910 con 0x7f3918075f90 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.620+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39181a5e30 con 0x7f3918075f90 2024-03-04T07:52:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.620+0000 7f390fe00640 1 -- 172.21.15.116:0/2517503919 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908018990 con 0x7f3918075f90 2024-03-04T07:52:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.620+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39181a6290 con 0x7f3918075f90 2024-03-04T07:52:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.621+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3918110180 con 0x7f3918075f90 2024-03-04T07:52:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.623+0000 7f390fe00640 1 -- 172.21.15.116:0/2517503919 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f3908010ac0 con 0x7f3918075f90 2024-03-04T07:52:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.624+0000 7f390fe00640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f38fc0a1d80 0x7f38fc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.625+0000 7f390fe00640 1 -- 172.21.15.116:0/2517503919 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f39080ce040 con 0x7f3918075f90 2024-03-04T07:52:17.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.625+0000 7f3916a00640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f38fc0a1d80 0x7f38fc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:17.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.626+0000 7f3916a00640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f38fc0a1d80 0x7f38fc0a4240 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f390402c240 tx=0x7f390402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:17.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.633+0000 7f390fe00640 1 -- 172.21.15.116:0/2517503919 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f390809ae60 con 0x7f3918075f90 2024-03-04T07:52:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.818+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f391802d280 con 0x7f3918075f90 2024-03-04T07:52:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.819+0000 7f390fe00640 1 -- 172.21.15.116:0/2517503919 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f390809ef00 con 0x7f3918075f90 2024-03-04T07:52:17.819 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-04T07:52:17.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f38fc0a1d80 msgr2=0x7f38fc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f38fc0a1d80 0x7f38fc0a4240 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f390402c240 tx=0x7f390402df90 comp rx=0 tx=0).stop 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3918075f90 msgr2=0x7f391819df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3918075f90 0x7f391819df20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f390800e9f0 tx=0x7f390800eeb0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 shutdown_connections 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3918077b70 0x7f39181a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f38fc0a1d80 0x7f38fc0a4240 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39180771d0 0x7f391819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 --2- 172.21.15.116:0/2517503919 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3918075f90 0x7f391819df20 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 >> 172.21.15.116:0/2517503919 conn(0x7f39180fed80 msgr2=0x7f3918100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.821+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 shutdown_connections 2024-03-04T07:52:17.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:17.822+0000 7f391e800640 1 -- 172.21.15.116:0/2517503919 wait complete. 2024-03-04T07:52:17.838 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T07:52:18.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.040+0000 7fb70f600640 1 -- 172.21.15.116:0/3002030321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081053a0 msgr2=0x7fb70810bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.040+0000 7fb70f600640 1 --2- 172.21.15.116:0/3002030321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081053a0 0x7fb70810bc30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb700009ca0 tx=0x7fb70002c100 comp rx=0 tx=0).stop 2024-03-04T07:52:18.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.041+0000 7fb70f600640 1 -- 172.21.15.116:0/3002030321 shutdown_connections 2024-03-04T07:52:18.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.041+0000 7fb70f600640 1 --2- 172.21.15.116:0/3002030321 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081053a0 0x7fb70810bc30 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.041+0000 7fb70f600640 1 --2- 172.21.15.116:0/3002030321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7081049e0 0x7fb708104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.041+0000 7fb70f600640 1 --2- 172.21.15.116:0/3002030321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081037f0 0x7fb708103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.041+0000 7fb70f600640 1 -- 172.21.15.116:0/3002030321 >> 172.21.15.116:0/3002030321 conn(0x7fb7080fed80 msgr2=0x7fb7081011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.041+0000 7fb70f600640 1 -- 172.21.15.116:0/3002030321 shutdown_connections 2024-03-04T07:52:18.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.041+0000 7fb70f600640 1 -- 172.21.15.116:0/3002030321 wait complete. 2024-03-04T07:52:18.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 Processor -- start 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 -- start start 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7081037f0 0x7fb708199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081049e0 0x7fb708199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081053a0 0x7fb7081a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70810f450 con 0x7fb7081053a0 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70810f2d0 con 0x7fb7081049e0 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.042+0000 7fb70f600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70810f5d0 con 0x7fb7081037f0 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081053a0 0x7fb7081a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081053a0 0x7fb7081a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42040/0 (socket says 172.21.15.116:42040) 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 -- 172.21.15.116:0/3589836881 learned_addr learned my addr 172.21.15.116:0/3589836881 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:18.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb707e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081049e0 0x7fb708199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70cc00640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7081037f0 0x7fb708199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 -- 172.21.15.116:0/3589836881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7081037f0 msgr2=0x7fb708199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7081037f0 0x7fb708199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 -- 172.21.15.116:0/3589836881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081049e0 msgr2=0x7fb708199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081049e0 0x7fb708199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 -- 172.21.15.116:0/3589836881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7081a1690 con 0x7fb7081053a0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb707e00640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081049e0 0x7fb708199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.043+0000 7fb70d600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081053a0 0x7fb7081a1020 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb70002c640 tx=0x7fb700035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.044+0000 7fb705600640 1 -- 172.21.15.116:0/3589836881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700039040 con 0x7fb7081053a0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.044+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7081a1920 con 0x7fb7081053a0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.044+0000 7fb705600640 1 -- 172.21.15.116:0/3589836881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb700044070 con 0x7fb7081053a0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.044+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7081a1e10 con 0x7fb7081053a0 2024-03-04T07:52:18.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.044+0000 7fb705600640 1 -- 172.21.15.116:0/3589836881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb700004400 con 0x7fb7081053a0 2024-03-04T07:52:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.045+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70810aab0 con 0x7fb7081053a0 2024-03-04T07:52:18.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.047+0000 7fb705600640 1 -- 172.21.15.116:0/3589836881 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb7000045a0 con 0x7fb7081053a0 2024-03-04T07:52:18.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.048+0000 7fb705600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6f00a1d80 0x7fb6f00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.048+0000 7fb705600640 1 -- 172.21.15.116:0/3589836881 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb7000f04a0 con 0x7fb7081053a0 2024-03-04T07:52:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.048+0000 7fb70cc00640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6f00a1d80 0x7fb6f00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.049+0000 7fb70cc00640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6f00a1d80 0x7fb6f00a4240 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb708104840 tx=0x7fb6fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:18.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.052+0000 7fb705600640 1 -- 172.21.15.116:0/3589836881 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb7000bd890 con 0x7fb7081053a0 2024-03-04T07:52:18.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.256+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb70802cff0 con 0x7fb7081053a0 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.256+0000 7fb705600640 1 -- 172.21.15.116:0/3589836881 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+601 (secure 0 0 0) 0x7fb7000098b0 con 0x7fb7081053a0 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1782984174 2024-03-05T07:29:57.331810+0000 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3439724466 2024-03-05T07:29:57.331810+0000 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2846507920 2024-03-05T07:29:57.331810+0000 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3742476358 2024-03-05T07:29:22.911402+0000 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3218338096 2024-03-05T07:29:57.331810+0000 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2843178631 2024-03-05T07:29:22.911402+0000 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3218338096 2024-03-05T07:29:57.331810+0000 2024-03-04T07:52:18.257 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1181502267 2024-03-05T07:29:22.911402+0000 2024-03-04T07:52:18.258 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1181502267 2024-03-05T07:29:22.911402+0000 2024-03-04T07:52:18.258 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/740422381 2024-03-05T07:29:22.911402+0000 2024-03-04T07:52:18.258 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-04T07:52:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:17 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1025814451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:52:18.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:17 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2517503919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:52:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6f00a1d80 msgr2=0x7fb6f00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6f00a1d80 0x7fb6f00a4240 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb708104840 tx=0x7fb6fc005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081053a0 msgr2=0x7fb7081a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081053a0 0x7fb7081a1020 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb70002c640 tx=0x7fb700035d10 comp rx=0 tx=0).stop 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 shutdown_connections 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7081053a0 0x7fb7081a1020 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7081049e0 0x7fb708199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb6f00a1d80 0x7fb6f00a4240 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 --2- 172.21.15.116:0/3589836881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7081037f0 0x7fb708199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 >> 172.21.15.116:0/3589836881 conn(0x7fb7080fed80 msgr2=0x7fb708107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.259+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 shutdown_connections 2024-03-04T07:52:18.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.260+0000 7fb70f600640 1 -- 172.21.15.116:0/3589836881 wait complete. 2024-03-04T07:52:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:17 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1025814451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:52:18.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:17 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2517503919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:52:18.278 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T07:52:18.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:52:18.278 DEBUG:teuthology.orchestra.run.smithi116:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T07:52:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:17 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1025814451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T07:52:18.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:17 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2517503919' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T07:52:18.395 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T07:52:18.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:52:18.395 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T07:52:18.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:52:18.456 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-04T07:52:18.493 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T07:52:18.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:52:18.494 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T07:52:18.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T07:52:18.608 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-04T07:52:18.728 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-04T07:52:18.739 INFO:tasks.cephadm:Teardown begin 2024-03-04T07:52:18.739 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T07:52:18.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:52:18.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 -- 172.21.15.116:0/3051345226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8101097c0 msgr2=0x7fd810110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 --2- 172.21.15.116:0/3051345226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8101097c0 0x7fd810110050 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd808009ca0 tx=0x7fd80802c0f0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 -- 172.21.15.116:0/3051345226 shutdown_connections 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 --2- 172.21.15.116:0/3051345226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8101097c0 0x7fd810110050 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 --2- 172.21.15.116:0/3051345226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd810108e00 0x7fd810109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 --2- 172.21.15.116:0/3051345226 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd810107c10 0x7fd810108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 -- 172.21.15.116:0/3051345226 >> 172.21.15.116:0/3051345226 conn(0x7fd810076be0 msgr2=0x7fd810079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 -- 172.21.15.116:0/3051345226 shutdown_connections 2024-03-04T07:52:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.928+0000 7fd814a00640 1 -- 172.21.15.116:0/3051345226 wait complete. 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 Processor -- start 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 -- start start 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd810107c10 0x7fd81019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd810108e00 0x7fd81019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8101097c0 0x7fd8101a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810113870 con 0x7fd810108e00 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101136f0 con 0x7fd8101097c0 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.929+0000 7fd814a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101139f0 con 0x7fd810107c10 2024-03-04T07:52:18.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8101097c0 0x7fd8101a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd805400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd810108e00 0x7fd81019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8101097c0 0x7fd8101a5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47120/0 (socket says 172.21.15.116:47120) 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 -- 172.21.15.116:0/137338578 learned_addr learned my addr 172.21.15.116:0/137338578 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80de00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd810107c10 0x7fd81019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 -- 172.21.15.116:0/137338578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd810107c10 msgr2=0x7fd81019de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd810107c10 0x7fd81019de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 -- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd810108e00 msgr2=0x7fd81019e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd810108e00 0x7fd81019e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 -- 172.21.15.116:0/137338578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8101a5a70 con 0x7fd8101097c0 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd805400640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd810108e00 0x7fd81019e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.930+0000 7fd80e800640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8101097c0 0x7fd8101a5400 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd80802c630 tx=0x7fd808032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.931+0000 7fd807400640 1 -- 172.21.15.116:0/137338578 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808041020 con 0x7fd8101097c0 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.931+0000 7fd807400640 1 -- 172.21.15.116:0/137338578 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd808034cd0 con 0x7fd8101097c0 2024-03-04T07:52:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.931+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8101a5d00 con 0x7fd8101097c0 2024-03-04T07:52:18.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.931+0000 7fd807400640 1 -- 172.21.15.116:0/137338578 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808039040 con 0x7fd8101097c0 2024-03-04T07:52:18.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.931+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8101a61e0 con 0x7fd8101097c0 2024-03-04T07:52:18.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.932+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81010ef30 con 0x7fd8101097c0 2024-03-04T07:52:18.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.934+0000 7fd807400640 1 -- 172.21.15.116:0/137338578 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd808044020 con 0x7fd8101097c0 2024-03-04T07:52:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.935+0000 7fd807400640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd7f40a1e00 0x7fd7f40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.935+0000 7fd807400640 1 -- 172.21.15.116:0/137338578 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd8080efb50 con 0x7fd8101097c0 2024-03-04T07:52:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.935+0000 7fd80de00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd7f40a1e00 0x7fd7f40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.936+0000 7fd80de00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd7f40a1e00 0x7fd7f40a42c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd8000095f0 tx=0x7fd800005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:18.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:18.939+0000 7fd807400640 1 -- 172.21.15.116:0/137338578 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd80804b020 con 0x7fd8101097c0 2024-03-04T07:52:19.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.182+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd81002cff0 con 0x7fd8101097c0 2024-03-04T07:52:19.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.183+0000 7fd807400640 1 -- 172.21.15.116:0/137338578 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fd8080bcfd0 con 0x7fd8101097c0 2024-03-04T07:52:19.183 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:52:19.183 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:52:19.183 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:52:19.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd7f40a1e00 msgr2=0x7fd7f40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd7f40a1e00 0x7fd7f40a42c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd8000095f0 tx=0x7fd800005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8101097c0 msgr2=0x7fd8101a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8101097c0 0x7fd8101a5400 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd80802c630 tx=0x7fd808032ea0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 shutdown_connections 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd8101097c0 0x7fd8101a5400 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd810108e00 0x7fd81019e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.185+0000 7fd814a00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd7f40a1e00 0x7fd7f40a42c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.186+0000 7fd814a00640 1 --2- 172.21.15.116:0/137338578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd810107c10 0x7fd81019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.186+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 >> 172.21.15.116:0/137338578 conn(0x7fd810076be0 msgr2=0x7fd81010c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.186+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 shutdown_connections 2024-03-04T07:52:19.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.186+0000 7fd814a00640 1 -- 172.21.15.116:0/137338578 wait complete. 2024-03-04T07:52:19.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T07:52:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:18 smithi116 ceph-mon[32150]: pgmap v747: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-04T07:52:19.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:18 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3589836881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T07:52:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:18 smithi152 ceph-mon[37154]: pgmap v747: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-04T07:52:19.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:18 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3589836881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T07:52:19.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:18 smithi133 ceph-mon[36505]: pgmap v747: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-04T07:52:19.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:18 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3589836881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.410+0000 7fb180a00640 1 -- 172.21.15.116:0/14625181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 msgr2=0x7fb17c104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.410+0000 7fb180a00640 1 --2- 172.21.15.116:0/14625181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c104c90 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009900 tx=0x7fb16c02bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.410+0000 7fb180a00640 1 -- 172.21.15.116:0/14625181 shutdown_connections 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.410+0000 7fb180a00640 1 --2- 172.21.15.116:0/14625181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb17c1051d0 0x7fb17c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.410+0000 7fb180a00640 1 --2- 172.21.15.116:0/14625181 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c104c90 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.410+0000 7fb180a00640 1 --2- 172.21.15.116:0/14625181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb17c103620 0x7fb17c103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.410+0000 7fb180a00640 1 -- 172.21.15.116:0/14625181 >> 172.21.15.116:0/14625181 conn(0x7fb17c0febb0 msgr2=0x7fb17c100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.411+0000 7fb180a00640 1 -- 172.21.15.116:0/14625181 shutdown_connections 2024-03-04T07:52:19.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.411+0000 7fb180a00640 1 -- 172.21.15.116:0/14625181 wait complete. 2024-03-04T07:52:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 Processor -- start 2024-03-04T07:52:19.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 -- start start 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb17c103620 0x7fb17c1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb17c1051d0 0x7fb17c1a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c10f290 con 0x7fb17c1051d0 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c10f110 con 0x7fb17c104810 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb180a00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c10f410 con 0x7fb17c103620 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb173e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb173e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47146/0 (socket says 172.21.15.116:47146) 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb173e00640 1 -- 172.21.15.116:0/2293585004 learned_addr learned my addr 172.21.15.116:0/2293585004 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.412+0000 7fb17b200640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb17c1051d0 0x7fb17c1a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb17a800640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb17c103620 0x7fb17c1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173e00640 1 -- 172.21.15.116:0/2293585004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb17c103620 msgr2=0x7fb17c1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173e00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb17c103620 0x7fb17c1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173e00640 1 -- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb17c1051d0 msgr2=0x7fb17c1a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173e00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb17c1051d0 0x7fb17c1a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173e00640 1 -- 172.21.15.116:0/2293585004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c1a1500 con 0x7fb17c104810 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173e00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c199e20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009230 tx=0x7fb16c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173400640 1 -- 172.21.15.116:0/2293585004 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c041070 con 0x7fb17c104810 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173400640 1 -- 172.21.15.116:0/2293585004 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb16c0044d0 con 0x7fb17c104810 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb173400640 1 -- 172.21.15.116:0/2293585004 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c034440 con 0x7fb17c104810 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.413+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17c1a1790 con 0x7fb17c104810 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.414+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c1a1cd0 con 0x7fb17c104810 2024-03-04T07:52:19.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.415+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17c10a940 con 0x7fb17c104810 2024-03-04T07:52:19.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.416+0000 7fb173400640 1 -- 172.21.15.116:0/2293585004 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fb16c044070 con 0x7fb17c104810 2024-03-04T07:52:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.417+0000 7fb173400640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb15c0a1e00 0x7fb15c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.418+0000 7fb173400640 1 -- 172.21.15.116:0/2293585004 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb16c0ef930 con 0x7fb17c104810 2024-03-04T07:52:19.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.418+0000 7fb17a800640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb15c0a1e00 0x7fb15c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.418+0000 7fb17a800640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb15c0a1e00 0x7fb15c0a42c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb17c104670 tx=0x7fb164005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:19.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.421+0000 7fb173400640 1 -- 172.21.15.116:0/2293585004 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb16c0bc6d0 con 0x7fb17c104810 2024-03-04T07:52:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.660+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17c02cff0 con 0x7fb17c104810 2024-03-04T07:52:19.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.662+0000 7fb173400640 1 -- 172.21.15.116:0/2293585004 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fb16c0a70e0 con 0x7fb17c104810 2024-03-04T07:52:19.662 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:52:19.662 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/666560370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":666560370},{"type":"v1","addr":"172.21.15.116:6835","nonce":666560370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/18076009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":18076009},{"type":"v1","addr":"172.21.15.116:6841","nonce":18076009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/2796423979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":2796423979},{"type":"v1","addr":"172.21.15.152:6833","nonce":2796423979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6833/20614029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6832","nonce":20614029},{"type":"v1","addr":"172.21.15.133:6833","nonce":20614029}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24475,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3275850329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3275850329},{"type":"v1","addr":"172.21.15.116:6837","nonce":3275850329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6835/331803361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6834","nonce":331803361},{"type":"v1","addr":"172.21.15.133:6835","nonce":331803361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3438829914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3438829914},{"type":"v1","addr":"172.21.15.152:6835","nonce":3438829914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/603594525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":603594525},{"type":"v1","addr":"172.21.15.152:6837","nonce":603594525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.133:6837/336393981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6836","nonce":336393981},{"type":"v1","addr":"172.21.15.133:6837","nonce":336393981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3832975468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3832975468},{"type":"v1","addr":"172.21.15.152:6839","nonce":3832975468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T07:37:54.573948+0000","modified":"2024-03-04T07:37:58.240682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24553":{"gid":24553,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1000386690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1000386690},{"type":"v1","addr":"172.21.15.116:6839","nonce":1000386690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.133:6839/1914821860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6838","nonce":1914821860},{"type":"v1","addr":"172.21.15.133:6839","nonce":1914821860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24581]},"id":1}]} 2024-03-04T07:52:19.663 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-03-04T07:52:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb15c0a1e00 msgr2=0x7fb15c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb15c0a1e00 0x7fb15c0a42c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb17c104670 tx=0x7fb164005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 msgr2=0x7fb17c199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c199e20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009230 tx=0x7fb16c005d50 comp rx=0 tx=0).stop 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 shutdown_connections 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb17c1051d0 0x7fb17c1a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb17c104810 0x7fb17c199e20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fb15c0a1e00 0x7fb15c0a42c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 --2- 172.21.15.116:0/2293585004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb17c103620 0x7fb17c1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.664+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 >> 172.21.15.116:0/2293585004 conn(0x7fb17c0febb0 msgr2=0x7fb17c107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.665+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 shutdown_connections 2024-03-04T07:52:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.665+0000 7fb180a00640 1 -- 172.21.15.116:0/2293585004 wait complete. 2024-03-04T07:52:19.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 -- 172.21.15.116:0/462713980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc107c10 msgr2=0x7fd4cc108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 --2- 172.21.15.116:0/462713980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc107c10 0x7fd4cc108030 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009900 tx=0x7fd4bc02bc30 comp rx=0 tx=0).stop 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 -- 172.21.15.116:0/462713980 shutdown_connections 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 --2- 172.21.15.116:0/462713980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4cc1097c0 0x7fd4cc110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 --2- 172.21.15.116:0/462713980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4cc108e00 0x7fd4cc109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 --2- 172.21.15.116:0/462713980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc107c10 0x7fd4cc108030 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 -- 172.21.15.116:0/462713980 >> 172.21.15.116:0/462713980 conn(0x7fd4cc076be0 msgr2=0x7fd4cc079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 -- 172.21.15.116:0/462713980 shutdown_connections 2024-03-04T07:52:19.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.894+0000 7fd4d1200640 1 -- 172.21.15.116:0/462713980 wait complete. 2024-03-04T07:52:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.895+0000 7fd4d1200640 1 Processor -- start 2024-03-04T07:52:19.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.895+0000 7fd4d1200640 1 -- start start 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4d1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4cc107c10 0x7fd4cc19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4d1200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc108e00 0x7fd4cc19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4d1200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4cc1097c0 0x7fd4cc1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4d1200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc113890 con 0x7fd4cc107c10 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4d1200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc113710 con 0x7fd4cc108e00 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4d1200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc113a10 con 0x7fd4cc1097c0 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4cc107c10 0x7fd4cc19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc108e00 0x7fd4cc19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c9e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4cc107c10 0x7fd4cc19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42098/0 (socket says 172.21.15.116:42098) 2024-03-04T07:52:19.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c9e00640 1 -- 172.21.15.116:0/2403414260 learned_addr learned my addr 172.21.15.116:0/2403414260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c3e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc108e00 0x7fd4cc19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47162/0 (socket says 172.21.15.116:47162) 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4ca800640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4cc1097c0 0x7fd4cc1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c3e00640 1 -- 172.21.15.116:0/2403414260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4cc1097c0 msgr2=0x7fd4cc1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c3e00640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4cc1097c0 0x7fd4cc1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c3e00640 1 -- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4cc107c10 msgr2=0x7fd4cc19dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c3e00640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4cc107c10 0x7fd4cc19dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.896+0000 7fd4c3e00640 1 -- 172.21.15.116:0/2403414260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc1a5b00 con 0x7fd4cc108e00 2024-03-04T07:52:19.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.897+0000 7fd4c3e00640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc108e00 0x7fd4cc19e420 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400e9f0 tx=0x7fd4b400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.897+0000 7fd4c2a00640 1 -- 172.21.15.116:0/2403414260 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b400ccc0 con 0x7fd4cc108e00 2024-03-04T07:52:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.897+0000 7fd4c2a00640 1 -- 172.21.15.116:0/2403414260 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4b4010910 con 0x7fd4cc108e00 2024-03-04T07:52:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.897+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc1a5df0 con 0x7fd4cc108e00 2024-03-04T07:52:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.897+0000 7fd4c2a00640 1 -- 172.21.15.116:0/2403414260 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4018960 con 0x7fd4cc108e00 2024-03-04T07:52:19.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.897+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc1a62b0 con 0x7fd4cc108e00 2024-03-04T07:52:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.898+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4cc10ef30 con 0x7fd4cc108e00 2024-03-04T07:52:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.900+0000 7fd4c2a00640 1 -- 172.21.15.116:0/2403414260 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fd4b4010ac0 con 0x7fd4cc108e00 2024-03-04T07:52:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.901+0000 7fd4c2a00640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4b00a1d80 0x7fd4b00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.901+0000 7fd4c2a00640 1 -- 172.21.15.116:0/2403414260 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd4b40ce150 con 0x7fd4cc108e00 2024-03-04T07:52:19.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.901+0000 7fd4c9e00640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4b00a1d80 0x7fd4b00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:19.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.902+0000 7fd4c9e00640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4b00a1d80 0x7fd4b00a4240 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc02c8e0 tx=0x7fd4bc005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:19.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:19.905+0000 7fd4c2a00640 1 -- 172.21.15.116:0/2403414260 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd4b409aef0 con 0x7fd4cc108e00 2024-03-04T07:52:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.109+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4cc0621d0 con 0x7fd4cc108e00 2024-03-04T07:52:20.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.111+0000 7fd4c2a00640 1 -- 172.21.15.116:0/2403414260 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22778 (secure 0 0 0) 0x7fd4b409ef90 con 0x7fd4cc108e00 2024-03-04T07:52:20.111 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-04T07:52:20.111 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"af2ccd48-d9f8-11ee-95c6-87774f69a715","created":"2024-03-04T07:28:48.960816+0000","modified":"2024-03-04T07:38:58.798253+0000","last_up_change":"2024-03-04T07:36:56.772544+0000","last_in_change":"2024-03-04T07:36:36.800924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T07:32:42.025045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T07:37:52.748569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T07:37:54.082948+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"914e00c8-058b-4b8a-8432-b88bb60d516b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6803","nonce":3034556591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6805","nonce":3034556591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6809","nonce":3034556591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":3034556591},{"type":"v1","addr":"172.21.15.116:6807","nonce":3034556591}]},"public_addr":"172.21.15.116:6803/3034556591","cluster_addr":"172.21.15.116:6805/3034556591","heartbeat_back_addr":"172.21.15.116:6809/3034556591","heartbeat_front_addr":"172.21.15.116:6807/3034556591","state":["exists","up"]},{"osd":1,"uuid":"4809c734-3113-4aa9-b228-80c2e0c2dd5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6801","nonce":1626213243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6803","nonce":1626213243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6807","nonce":1626213243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":1626213243},{"type":"v1","addr":"172.21.15.133:6805","nonce":1626213243}]},"public_addr":"172.21.15.133:6801/1626213243","cluster_addr":"172.21.15.133:6803/1626213243","heartbeat_back_addr":"172.21.15.133:6807/1626213243","heartbeat_front_addr":"172.21.15.133:6805/1626213243","state":["exists","up"]},{"osd":2,"uuid":"5e50d8f2-dbf0-4066-b2cb-df0821866acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6801","nonce":4177028869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6803","nonce":4177028869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6807","nonce":4177028869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":4177028869},{"type":"v1","addr":"172.21.15.152:6805","nonce":4177028869}]},"public_addr":"172.21.15.152:6801/4177028869","cluster_addr":"172.21.15.152:6803/4177028869","heartbeat_back_addr":"172.21.15.152:6807/4177028869","heartbeat_front_addr":"172.21.15.152:6805/4177028869","state":["exists","up"]},{"osd":3,"uuid":"3acba8d9-b204-41eb-8f35-936f0acb0f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6811","nonce":2894752093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6813","nonce":2894752093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6817","nonce":2894752093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2894752093},{"type":"v1","addr":"172.21.15.116:6815","nonce":2894752093}]},"public_addr":"172.21.15.116:6811/2894752093","cluster_addr":"172.21.15.116:6813/2894752093","heartbeat_back_addr":"172.21.15.116:6817/2894752093","heartbeat_front_addr":"172.21.15.116:6815/2894752093","state":["exists","up"]},{"osd":4,"uuid":"60bcd05b-b11c-4cca-948b-7dd432c6cd2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6809","nonce":3217315189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6811","nonce":3217315189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6815","nonce":3217315189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":3217315189},{"type":"v1","addr":"172.21.15.133:6813","nonce":3217315189}]},"public_addr":"172.21.15.133:6809/3217315189","cluster_addr":"172.21.15.133:6811/3217315189","heartbeat_back_addr":"172.21.15.133:6815/3217315189","heartbeat_front_addr":"172.21.15.133:6813/3217315189","state":["exists","up"]},{"osd":5,"uuid":"c0fa4743-f949-4748-9772-3bb1292ebb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6809","nonce":2293273292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6811","nonce":2293273292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6815","nonce":2293273292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":2293273292},{"type":"v1","addr":"172.21.15.152:6813","nonce":2293273292}]},"public_addr":"172.21.15.152:6809/2293273292","cluster_addr":"172.21.15.152:6811/2293273292","heartbeat_back_addr":"172.21.15.152:6815/2293273292","heartbeat_front_addr":"172.21.15.152:6813/2293273292","state":["exists","up"]},{"osd":6,"uuid":"2442f5fc-a522-4f08-8a68-c0bc903427b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6819","nonce":836391582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6821","nonce":836391582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6825","nonce":836391582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":836391582},{"type":"v1","addr":"172.21.15.116:6823","nonce":836391582}]},"public_addr":"172.21.15.116:6819/836391582","cluster_addr":"172.21.15.116:6821/836391582","heartbeat_back_addr":"172.21.15.116:6825/836391582","heartbeat_front_addr":"172.21.15.116:6823/836391582","state":["exists","up"]},{"osd":7,"uuid":"f1dbc539-c51f-4ae3-9239-8c032ef60b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6817","nonce":3238142404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6819","nonce":3238142404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6823","nonce":3238142404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3238142404},{"type":"v1","addr":"172.21.15.133:6821","nonce":3238142404}]},"public_addr":"172.21.15.133:6817/3238142404","cluster_addr":"172.21.15.133:6819/3238142404","heartbeat_back_addr":"172.21.15.133:6823/3238142404","heartbeat_front_addr":"172.21.15.133:6821/3238142404","state":["exists","up"]},{"osd":8,"uuid":"a9bc6819-8f22-4e6d-83e1-96690fc82524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6817","nonce":1470448295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6819","nonce":1470448295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6823","nonce":1470448295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1470448295},{"type":"v1","addr":"172.21.15.152:6821","nonce":1470448295}]},"public_addr":"172.21.15.152:6817/1470448295","cluster_addr":"172.21.15.152:6819/1470448295","heartbeat_back_addr":"172.21.15.152:6823/1470448295","heartbeat_front_addr":"172.21.15.152:6821/1470448295","state":["exists","up"]},{"osd":9,"uuid":"2931b69e-19e6-4739-8f27-b1d0b83f96d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6827","nonce":3236031777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6829","nonce":3236031777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6833","nonce":3236031777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":3236031777},{"type":"v1","addr":"172.21.15.116:6831","nonce":3236031777}]},"public_addr":"172.21.15.116:6827/3236031777","cluster_addr":"172.21.15.116:6829/3236031777","heartbeat_back_addr":"172.21.15.116:6833/3236031777","heartbeat_front_addr":"172.21.15.116:6831/3236031777","state":["exists","up"]},{"osd":10,"uuid":"4111372f-edde-4015-beb6-fb5895afe8d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6825","nonce":2680230905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6827","nonce":2680230905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6830","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6831","nonce":2680230905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":2680230905},{"type":"v1","addr":"172.21.15.133:6829","nonce":2680230905}]},"public_addr":"172.21.15.133:6825/2680230905","cluster_addr":"172.21.15.133:6827/2680230905","heartbeat_back_addr":"172.21.15.133:6831/2680230905","heartbeat_front_addr":"172.21.15.133:6829/2680230905","state":["exists","up"]},{"osd":11,"uuid":"9c4a6c93-8d7c-448c-99cb-a0ede94976a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6825","nonce":1192411911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6827","nonce":1192411911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6831","nonce":1192411911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":1192411911},{"type":"v1","addr":"172.21.15.152:6829","nonce":1192411911}]},"public_addr":"172.21.15.152:6825/1192411911","cluster_addr":"172.21.15.152:6827/1192411911","heartbeat_back_addr":"172.21.15.152:6831/1192411911","heartbeat_front_addr":"172.21.15.152:6829/1192411911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:31:49.710775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:15.284905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:32:38.957866+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:11.627903+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:33:38.388267+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:02.944352+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:34:36.767053+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:03.600275+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:35:28.804371+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:02.583370+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:29.356555+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T07:36:54.762227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.2","mappings":[{"from":5,"to":7}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/1782984174":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3439724466":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2846507920":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/3742476358":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6801/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:0/2843178631":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/3218338096":"2024-03-05T07:29:57.331810+0000","172.21.15.116:6801/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:6800/1181502267":"2024-03-05T07:29:22.911402+0000","172.21.15.116:0/740422381":"2024-03-05T07:29:22.911402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T07:52:20.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.113+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4b00a1d80 msgr2=0x7fd4b00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.113+0000 7fd4d1200640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4b00a1d80 0x7fd4b00a4240 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc02c8e0 tx=0x7fd4bc005b50 comp rx=0 tx=0).stop 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.113+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc108e00 msgr2=0x7fd4cc19e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.113+0000 7fd4d1200640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc108e00 0x7fd4cc19e420 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400e9f0 tx=0x7fd4b400eeb0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 shutdown_connections 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fd4b00a1d80 0x7fd4b00a4240 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4cc1097c0 0x7fd4cc1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd4cc108e00 0x7fd4cc19e420 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 --2- 172.21.15.116:0/2403414260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4cc107c10 0x7fd4cc19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 >> 172.21.15.116:0/2403414260 conn(0x7fd4cc076be0 msgr2=0x7fd4cc10c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:20.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 shutdown_connections 2024-03-04T07:52:20.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.114+0000 7fd4d1200640 1 -- 172.21.15.116:0/2403414260 wait complete. 2024-03-04T07:52:20.133 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T07:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:19 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/137338578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:19 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2293585004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:20.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:19 smithi116 ceph-mon[32150]: pgmap v748: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-04T07:52:20.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:19 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/137338578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:20.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:19 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2293585004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:20.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:19 smithi152 ceph-mon[37154]: pgmap v748: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-04T07:52:20.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:19 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/137338578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:20.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:19 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2293585004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T07:52:20.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:19 smithi133 ceph-mon[36505]: pgmap v748: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-04T07:52:20.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 -- 172.21.15.116:0/883592753 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41049e0 msgr2=0x7ffbf4104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 --2- 172.21.15.116:0/883592753 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41049e0 0x7ffbf4104e60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009900 tx=0x7ffbe402bc40 comp rx=0 tx=0).stop 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 -- 172.21.15.116:0/883592753 shutdown_connections 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 --2- 172.21.15.116:0/883592753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41053a0 0x7ffbf410bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 --2- 172.21.15.116:0/883592753 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41049e0 0x7ffbf4104e60 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 --2- 172.21.15.116:0/883592753 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41037f0 0x7ffbf4103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 -- 172.21.15.116:0/883592753 >> 172.21.15.116:0/883592753 conn(0x7ffbf40fed80 msgr2=0x7ffbf41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.347+0000 7ffbfb600640 1 -- 172.21.15.116:0/883592753 shutdown_connections 2024-03-04T07:52:20.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.348+0000 7ffbfb600640 1 -- 172.21.15.116:0/883592753 wait complete. 2024-03-04T07:52:20.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 Processor -- start 2024-03-04T07:52:20.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 -- start start 2024-03-04T07:52:20.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41037f0 0x7ffbf4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41049e0 0x7ffbf419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41053a0 0x7ffbf41a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf410f450 con 0x7ffbf41049e0 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf410f2d0 con 0x7ffbf41037f0 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbfb600640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf410f5d0 con 0x7ffbf41053a0 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbf8c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41037f0 0x7ffbf4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbf8c00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41037f0 0x7ffbf4199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47182/0 (socket says 172.21.15.116:47182) 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.349+0000 7ffbf8c00640 1 -- 172.21.15.116:0/1040112391 learned_addr learned my addr 172.21.15.116:0/1040112391 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf9600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41053a0 0x7ffbf41a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf3e00640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41049e0 0x7ffbf419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf8c00640 1 -- 172.21.15.116:0/1040112391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41053a0 msgr2=0x7ffbf41a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf8c00640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41053a0 0x7ffbf41a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf8c00640 1 -- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41049e0 msgr2=0x7ffbf419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf8c00640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41049e0 0x7ffbf419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf8c00640 1 -- 172.21.15.116:0/1040112391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf41a1770 con 0x7ffbf41037f0 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf8c00640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41037f0 0x7ffbf4199bf0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800e9c0 tx=0x7ffbe800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf1600640 1 -- 172.21.15.116:0/1040112391 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe800ccc0 con 0x7ffbf41037f0 2024-03-04T07:52:20.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf1600640 1 -- 172.21.15.116:0/1040112391 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbe80035f0 con 0x7ffbf41037f0 2024-03-04T07:52:20.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbf1600640 1 -- 172.21.15.116:0/1040112391 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe800ce60 con 0x7ffbf41037f0 2024-03-04T07:52:20.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.350+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf41a1a60 con 0x7ffbf41037f0 2024-03-04T07:52:20.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.351+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf41a1f30 con 0x7ffbf41037f0 2024-03-04T07:52:20.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.352+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf410ab30 con 0x7ffbf41037f0 2024-03-04T07:52:20.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.353+0000 7ffbf1600640 1 -- 172.21.15.116:0/1040112391 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7ffbe80037a0 con 0x7ffbf41037f0 2024-03-04T07:52:20.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.355+0000 7ffbf1600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffbdc0a1e00 0x7ffbdc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.355+0000 7ffbf1600640 1 -- 172.21.15.116:0/1040112391 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ffbe80ce070 con 0x7ffbf41037f0 2024-03-04T07:52:20.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.355+0000 7ffbf3e00640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffbdc0a1e00 0x7ffbdc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.356+0000 7ffbf3e00640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffbdc0a1e00 0x7ffbdc0a42c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffbe402c7c0 tx=0x7ffbe4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:20.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.359+0000 7ffbf1600640 1 -- 172.21.15.116:0/1040112391 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffbe809b4a0 con 0x7ffbf41037f0 2024-03-04T07:52:20.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.595+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffbf40621d0 con 0x7ffbf41037f0 2024-03-04T07:52:20.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.605+0000 7ffbf1600640 1 -- 172.21.15.116:0/1040112391 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffbe809ada0 con 0x7ffbf41037f0 2024-03-04T07:52:20.606 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T07:52:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffbdc0a1e00 msgr2=0x7ffbdc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffbdc0a1e00 0x7ffbdc0a42c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffbe402c7c0 tx=0x7ffbe4005fd0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41037f0 msgr2=0x7ffbf4199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41037f0 0x7ffbf4199bf0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800e9c0 tx=0x7ffbe800edc0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 shutdown_connections 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7ffbdc0a1e00 0x7ffbdc0a42c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffbf41053a0 0x7ffbf41a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41049e0 0x7ffbf419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 --2- 172.21.15.116:0/1040112391 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ffbf41037f0 0x7ffbf4199bf0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 >> 172.21.15.116:0/1040112391 conn(0x7ffbf40fed80 msgr2=0x7ffbf4107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 shutdown_connections 2024-03-04T07:52:20.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.608+0000 7ffbfb600640 1 -- 172.21.15.116:0/1040112391 wait complete. 2024-03-04T07:52:20.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T07:52:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 -- 172.21.15.116:0/640887100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741037f0 msgr2=0x7fa974103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 --2- 172.21.15.116:0/640887100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741037f0 0x7fa974103c10 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa964009860 tx=0x7fa96402bb70 comp rx=0 tx=0).stop 2024-03-04T07:52:20.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 -- 172.21.15.116:0/640887100 shutdown_connections 2024-03-04T07:52:20.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 --2- 172.21.15.116:0/640887100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9741053a0 0x7fa97410bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 --2- 172.21.15.116:0/640887100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 0x7fa974104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 --2- 172.21.15.116:0/640887100 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741037f0 0x7fa974103c10 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 -- 172.21.15.116:0/640887100 >> 172.21.15.116:0/640887100 conn(0x7fa9740fed80 msgr2=0x7fa9741011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:20.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 -- 172.21.15.116:0/640887100 shutdown_connections 2024-03-04T07:52:20.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.836+0000 7fa97a200640 1 -- 172.21.15.116:0/640887100 wait complete. 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 Processor -- start 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 -- start start 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9741037f0 0x7fa974199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 0x7fa97419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741053a0 0x7fa9741a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97410f440 con 0x7fa9741049e0 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97410f2c0 con 0x7fa9741053a0 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.837+0000 7fa97a200640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97410f5c0 con 0x7fa9741037f0 2024-03-04T07:52:20.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 0x7fa97419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa973e00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741053a0 0x7fa9741a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 0x7fa97419a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42150/0 (socket says 172.21.15.116:42150) 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 -- 172.21.15.116:0/2903528135 learned_addr learned my addr 172.21.15.116:0/2903528135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa973400640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9741037f0 0x7fa974199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 -- 172.21.15.116:0/2903528135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9741037f0 msgr2=0x7fa974199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9741037f0 0x7fa974199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 -- 172.21.15.116:0/2903528135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741053a0 msgr2=0x7fa9741a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741053a0 0x7fa9741a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 -- 172.21.15.116:0/2903528135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9741a1700 con 0x7fa9741049e0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.838+0000 7fa96aa00640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 0x7fa97419a020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00e9c0 tx=0x7fa95c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.839+0000 7fa970c00640 1 -- 172.21.15.116:0/2903528135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00ccc0 con 0x7fa9741049e0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.839+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9741a19f0 con 0x7fa9741049e0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.839+0000 7fa970c00640 1 -- 172.21.15.116:0/2903528135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa95c005500 con 0x7fa9741049e0 2024-03-04T07:52:20.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.839+0000 7fa970c00640 1 -- 172.21.15.116:0/2903528135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c010430 con 0x7fa9741049e0 2024-03-04T07:52:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.839+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9741a1f90 con 0x7fa9741049e0 2024-03-04T07:52:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.840+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97410ab90 con 0x7fa9741049e0 2024-03-04T07:52:20.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.842+0000 7fa970c00640 1 -- 172.21.15.116:0/2903528135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7fa95c0046c0 con 0x7fa9741049e0 2024-03-04T07:52:20.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.843+0000 7fa970c00640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9580a1e00 0x7fa9580a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:20.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.843+0000 7fa970c00640 1 -- 172.21.15.116:0/2903528135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8326+0+0 (secure 0 0 0) 0x7fa95c0ce0f0 con 0x7fa9741049e0 2024-03-04T07:52:20.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.843+0000 7fa973400640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9580a1e00 0x7fa9580a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:20.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.844+0000 7fa973400640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9580a1e00 0x7fa9580a42c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa9640349e0 tx=0x7fa964034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:20.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:20.848+0000 7fa970c00640 1 -- 172.21.15.116:0/2903528135 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa95c09b460 con 0x7fa9741049e0 2024-03-04T07:52:20.992 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b[67089]: ignoring --setuser ceph since I am not root 2024-03-04T07:52:20.992 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b[67089]: ignoring --setgroup ceph since I am not root 2024-03-04T07:52:20.992 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mds[67093]: -- 172.21.15.133:0/32673074 <== mon.2 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c9f615ba40 con 0x55c9f612ec00 2024-03-04T07:52:20.992 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b[67089]: starting mds.b at 2024-03-04T07:52:20.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a[32146]: 2024-03-04T07:52:20.598+0000 7ff89bc00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T07:52:20.993 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g[71233]: ignoring --setuser ceph since I am not root 2024-03-04T07:52:20.993 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g[71233]: ignoring --setgroup ceph since I am not root 2024-03-04T07:52:20.993 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mds[71237]: -- 172.21.15.116:0/3647646822 <== mon.2 v2:172.21.15.133:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562c4e071a40 con 0x562c4e045000 2024-03-04T07:52:20.993 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g[71233]: starting mds.g at 2024-03-04T07:52:21.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.074+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa9740621d0 con 0x7fa9741049e0 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2403414260' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1040112391' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: osdmap e88: 12 total, 12 up, 12 in 2024-03-04T07:52:21.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:20 smithi116 ceph-mon[32150]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-04T07:52:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2403414260' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1040112391' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T07:52:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T07:52:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T07:52:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T07:52:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-04T07:52:21.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: osdmap e88: 12 total, 12 up, 12 in 2024-03-04T07:52:21.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:20 smithi152 ceph-mon[37154]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-04T07:52:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2403414260' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T07:52:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1040112391' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T07:52:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T07:52:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T07:52:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T07:52:21.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-04T07:52:21.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: osdmap e88: 12 total, 12 up, 12 in 2024-03-04T07:52:21.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:20 smithi133 ceph-mon[36505]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-04T07:52:21.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.604+0000 7fa970c00640 1 -- 172.21.15.116:0/2903528135 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa95c09ad60 con 0x7fa9741049e0 2024-03-04T07:52:21.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.608+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9580a1e00 msgr2=0x7fa9580a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.608+0000 7fa97a200640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9580a1e00 0x7fa9580a42c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa9640349e0 tx=0x7fa964034ab0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.608+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 msgr2=0x7fa97419a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.608+0000 7fa97a200640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 0x7fa97419a020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00e9c0 tx=0x7fa95c00ee80 comp rx=0 tx=0).stop 2024-03-04T07:52:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 shutdown_connections 2024-03-04T07:52:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9741053a0 0x7fa9741a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9741049e0 0x7fa97419a020 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7fa9580a1e00 0x7fa9580a42c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 --2- 172.21.15.116:0/2903528135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9741037f0 0x7fa974199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 >> 172.21.15.116:0/2903528135 conn(0x7fa9740fed80 msgr2=0x7fa974107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 shutdown_connections 2024-03-04T07:52:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.609+0000 7fa97a200640 1 -- 172.21.15.116:0/2903528135 wait complete. 2024-03-04T07:52:21.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T07:52:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 -- 172.21.15.116:0/2718716708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081027c0 msgr2=0x7f2708102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 --2- 172.21.15.116:0/2718716708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081027c0 0x7f2708102c40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009900 tx=0x7f26f002bc40 comp rx=0 tx=0).stop 2024-03-04T07:52:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 -- 172.21.15.116:0/2718716708 shutdown_connections 2024-03-04T07:52:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 --2- 172.21.15.116:0/2718716708 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2708103180 0x7f2708109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 --2- 172.21.15.116:0/2718716708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081027c0 0x7f2708102c40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 --2- 172.21.15.116:0/2718716708 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27081015d0 0x7f27081019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 -- 172.21.15.116:0/2718716708 >> 172.21.15.116:0/2718716708 conn(0x7f27080fcb40 msgr2=0x7f27080fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 -- 172.21.15.116:0/2718716708 shutdown_connections 2024-03-04T07:52:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.846+0000 7f270cc00640 1 -- 172.21.15.116:0/2718716708 wait complete. 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 Processor -- start 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 -- start start 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081015d0 0x7f2708075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27081027c0 0x7f27080739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2708103180 0x7f2708073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270810d290 con 0x7f27081015d0 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270810d110 con 0x7f27081027c0 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f270cc00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270810d410 con 0x7f2708103180 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f2705e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081015d0 0x7f2708075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f2705e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081015d0 0x7f2708075380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42166/0 (socket says 172.21.15.116:42166) 2024-03-04T07:52:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f2705e00640 1 -- 172.21.15.116:0/1656311698 learned_addr learned my addr 172.21.15.116:0/1656311698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f2705400640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27081027c0 0x7f27080739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.847+0000 7f2706800640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2708103180 0x7f2708073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f2705e00640 1 -- 172.21.15.116:0/1656311698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2708103180 msgr2=0x7f2708073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f2705e00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2708103180 0x7f2708073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f2705e00640 1 -- 172.21.15.116:0/1656311698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27081027c0 msgr2=0x7f27080739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f2705e00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27081027c0 0x7f27080739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f2705e00640 1 -- 172.21.15.116:0/1656311698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2708074790 con 0x7f27081015d0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f2705e00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081015d0 0x7f2708075380 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f26f800e9c0 tx=0x7f26f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f26fea00640 1 -- 172.21.15.116:0/1656311698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f800cd80 con 0x7f27081015d0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f26fea00640 1 -- 172.21.15.116:0/1656311698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26f80035f0 con 0x7f27081015d0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f26fea00640 1 -- 172.21.15.116:0/1656311698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8010690 con 0x7f27081015d0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27081a19f0 con 0x7f27081015d0 2024-03-04T07:52:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.848+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27081a1ec0 con 0x7f27081015d0 2024-03-04T07:52:21.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.850+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2708068810 con 0x7f27081015d0 2024-03-04T07:52:21.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.851+0000 7f26fea00640 1 -- 172.21.15.116:0/1656311698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f26f80037a0 con 0x7f27081015d0 2024-03-04T07:52:21.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.853+0000 7f26fea00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f26e80a1e00 0x7f26e80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:21.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.853+0000 7f26fea00640 1 -- 172.21.15.116:0/1656311698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8326+0+0 (secure 0 0 0) 0x7f26f80ce220 con 0x7f27081015d0 2024-03-04T07:52:21.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.853+0000 7f2705400640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f26e80a1e00 0x7f26e80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:21.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.854+0000 7f2705400640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f26e80a1e00 0x7f26e80a42c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f26f002c250 tx=0x7f26f0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:21.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:21.857+0000 7f26fea00640 1 -- 172.21.15.116:0/1656311698 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f26f809b5f0 con 0x7f27081015d0 2024-03-04T07:52:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:22.057+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f27081088f0 con 0x7f27081015d0 2024-03-04T07:52:22.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2903528135' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:52:22.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T07:52:22.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T07:52:22.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: Cluster is now healthy 2024-03-04T07:52:22.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/2903528135' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-04T07:52:22.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.133:6838/109132850,v1:172.21.15.133:6839/109132850] up:boot 2024-03-04T07:52:22.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: mds.? [v2:172.21.15.116:6838/2413202862,v1:172.21.15.116:6839/2413202862] up:boot 2024-03-04T07:52:22.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: fsmap 12 up:standby 2024-03-04T07:52:22.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T07:52:22.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:52:22.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:21 smithi116 ceph-mon[32150]: pgmap v750: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 3 op/s 2024-03-04T07:52:22.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2903528135' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: Cluster is now healthy 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/2903528135' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.133:6838/109132850,v1:172.21.15.133:6839/109132850] up:boot 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: mds.? [v2:172.21.15.116:6838/2413202862,v1:172.21.15.116:6839/2413202862] up:boot 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: fsmap 12 up:standby 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:52:22.272 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:21 smithi152 ceph-mon[37154]: pgmap v750: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 3 op/s 2024-03-04T07:52:22.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2903528135' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T07:52:22.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: Cluster is now healthy 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/2903528135' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.133:6838/109132850,v1:172.21.15.133:6839/109132850] up:boot 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: mds.? [v2:172.21.15.116:6838/2413202862,v1:172.21.15.116:6839/2413202862] up:boot 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: fsmap 12 up:standby 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T07:52:22.338 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: from='mgr.14152 172.21.15.116:0/2795203544' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T07:52:22.339 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:21 smithi133 ceph-mon[36505]: pgmap v750: 97 pgs: 97 active+clean; 4.6 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 3 op/s 2024-03-04T07:52:22.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:22.998+0000 7f26fea00640 1 -- 172.21.15.116:0/1656311698 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) v1 ==== 181+0+0 (secure 0 0 0) 0x7f26f809aef0 con 0x7f27081015d0 2024-03-04T07:52:22.998 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2024-03-04T07:52:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.001+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f26e80a1e00 msgr2=0x7f26e80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.001+0000 7f270cc00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f26e80a1e00 0x7f26e80a42c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f26f002c250 tx=0x7f26f0005fd0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.001+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081015d0 msgr2=0x7f2708075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.001+0000 7f270cc00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081015d0 0x7f2708075380 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f26f800e9c0 tx=0x7f26f800ee80 comp rx=0 tx=0).stop 2024-03-04T07:52:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.002+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 shutdown_connections 2024-03-04T07:52:23.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.002+0000 7f270cc00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f26e80a1e00 0x7f26e80a42c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.002+0000 7f270cc00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2708103180 0x7f2708073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.002+0000 7f270cc00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f27081027c0 0x7f27080739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.002+0000 7f270cc00640 1 --2- 172.21.15.116:0/1656311698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27081015d0 0x7f2708075380 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.002+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 >> 172.21.15.116:0/1656311698 conn(0x7f27080fcb40 msgr2=0x7f2708105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:23.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.003+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 shutdown_connections 2024-03-04T07:52:23.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.003+0000 7f270cc00640 1 -- 172.21.15.116:0/1656311698 wait complete. 2024-03-04T07:52:23.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.236+0000 7f989a000640 1 -- 172.21.15.116:0/4063184722 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894077b70 msgr2=0x7f9894078410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.236+0000 7f989a000640 1 --2- 172.21.15.116:0/4063184722 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894077b70 0x7f9894078410 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f988c009be0 tx=0x7f988c02bef0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.237+0000 7f989a000640 1 -- 172.21.15.116:0/4063184722 shutdown_connections 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.237+0000 7f989a000640 1 --2- 172.21.15.116:0/4063184722 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894077b70 0x7f9894078410 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.237+0000 7f989a000640 1 --2- 172.21.15.116:0/4063184722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98940771d0 0x7f9894077630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.237+0000 7f989a000640 1 --2- 172.21.15.116:0/4063184722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894075f90 0x7f98940763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.237+0000 7f989a000640 1 -- 172.21.15.116:0/4063184722 >> 172.21.15.116:0/4063184722 conn(0x7f98940fed80 msgr2=0x7f98941011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.237+0000 7f989a000640 1 -- 172.21.15.116:0/4063184722 shutdown_connections 2024-03-04T07:52:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.237+0000 7f989a000640 1 -- 172.21.15.116:0/4063184722 wait complete. 2024-03-04T07:52:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 Processor -- start 2024-03-04T07:52:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 -- start start 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894075f90 0x7f989419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98940771d0 0x7f989419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894077b70 0x7f98941a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98941138a0 con 0x7f9894077b70 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9894113720 con 0x7f9894075f90 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f989a000640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9894113a20 con 0x7f98940771d0 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f9893e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894077b70 0x7f98941a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f9893400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894075f90 0x7f989419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f9893400640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894075f90 0x7f989419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.116:47244/0 (socket says 172.21.15.116:47244) 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f9893e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894077b70 0x7f98941a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42178/0 (socket says 172.21.15.116:42178) 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.238+0000 7f9893400640 1 -- 172.21.15.116:0/3587729383 learned_addr learned my addr 172.21.15.116:0/3587729383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-04T07:52:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f988be00640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98940771d0 0x7f989419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9893400640 1 -- 172.21.15.116:0/3587729383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98940771d0 msgr2=0x7f989419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9893400640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98940771d0 0x7f989419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9893400640 1 -- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894077b70 msgr2=0x7f98941a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9893400640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894077b70 0x7f98941a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9893400640 1 -- 172.21.15.116:0/3587729383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98941a5ac0 con 0x7f9894075f90 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f988be00640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98940771d0 0x7f989419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9893e00640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894077b70 0x7f98941a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9893400640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894075f90 0x7f989419dea0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f988400e9c0 tx=0x7f988400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f9890c00640 1 -- 172.21.15.116:0/3587729383 <== mon.2 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988400cde0 con 0x7f9894075f90 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.239+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98941a5db0 con 0x7f9894075f90 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.240+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98941a6270 con 0x7f9894075f90 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.240+0000 7f9890c00640 1 -- 172.21.15.116:0/3587729383 <== mon.2 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98840035f0 con 0x7f9894075f90 2024-03-04T07:52:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.240+0000 7f9890c00640 1 -- 172.21.15.116:0/3587729383 <== mon.2 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884010690 con 0x7f9894075f90 2024-03-04T07:52:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.241+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98941102d0 con 0x7f9894075f90 2024-03-04T07:52:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.243+0000 7f9890c00640 1 -- 172.21.15.116:0/3587729383 <== mon.2 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 136757+0+0 (secure 0 0 0) 0x7f98840037a0 con 0x7f9894075f90 2024-03-04T07:52:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.244+0000 7f9890c00640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98780a1d80 0x7f98780a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T07:52:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.244+0000 7f9890c00640 1 -- 172.21.15.116:0/3587729383 <== mon.2 v2:172.21.15.133:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 7801+0+0 (secure 0 0 0) 0x7f98840cdfb0 con 0x7f9894075f90 2024-03-04T07:52:23.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.244+0000 7f988be00640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98780a1d80 0x7f98780a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T07:52:23.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.245+0000 7f988be00640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98780a1d80 0x7f98780a4240 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f989419f3b0 tx=0x7f987c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T07:52:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.246+0000 7f9890c00640 1 -- 172.21.15.116:0/3587729383 <== mon.2 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f988409b540 con 0x7f9894075f90 2024-03-04T07:52:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:23 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1656311698' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:23.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:23 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1656311698' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:23.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:23 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1656311698' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:23.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:23.455+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f989402cff0 con 0x7f9894075f90 2024-03-04T07:52:24.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.028+0000 7f9890c00640 1 -- 172.21.15.116:0/3587729383 <== mon.2 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7f988409ae40 con 0x7f9894075f90 2024-03-04T07:52:24.029 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2024-03-04T07:52:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98780a1d80 msgr2=0x7f98780a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98780a1d80 0x7f98780a4240 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f989419f3b0 tx=0x7f987c005f90 comp rx=0 tx=0).stop 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894075f90 msgr2=0x7f989419dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894075f90 0x7f989419dea0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f988400e9c0 tx=0x7f988400ee80 comp rx=0 tx=0).stop 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 shutdown_connections 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9894077b70 0x7f98941a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.116:6800/142890918,v1:172.21.15.116:6801/142890918] conn(0x7f98780a1d80 0x7f98780a4240 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f98940771d0 0x7f989419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 --2- 172.21.15.116:0/3587729383 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9894075f90 0x7f989419dea0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 >> 172.21.15.116:0/3587729383 conn(0x7f98940fed80 msgr2=0x7f9894100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.033+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 shutdown_connections 2024-03-04T07:52:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-04T07:52:24.034+0000 7f989a000640 1 -- 172.21.15.116:0/3587729383 wait complete. 2024-03-04T07:52:24.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:52:24.137 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:52:24.173 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:52:24.208 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-04T07:52:24.208 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T07:52:24.267 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T07:52:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:24 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/1656311698' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T07:52:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:24 smithi152 ceph-mon[37154]: osdmap e89: 12 total, 12 up, 12 in 2024-03-04T07:52:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:24 smithi152 ceph-mon[37154]: from='client.? 172.21.15.116:0/3587729383' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:24 smithi152 ceph-mon[37154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:24.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:24 smithi152 ceph-mon[37154]: pgmap v752: 65 pgs: 65 active+clean; 577 KiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:52:24.284 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T07:52:24.306 INFO:tasks.cephadm:Stopping all daemons... 2024-03-04T07:52:24.306 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-04T07:52:24.307 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a 2024-03-04T07:52:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:24 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/1656311698' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T07:52:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:24 smithi133 ceph-mon[36505]: osdmap e89: 12 total, 12 up, 12 in 2024-03-04T07:52:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:24 smithi133 ceph-mon[36505]: from='client.? 172.21.15.116:0/3587729383' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:24 smithi133 ceph-mon[36505]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:24.337 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:24 smithi133 ceph-mon[36505]: pgmap v752: 65 pgs: 65 active+clean; 577 KiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:52:24.380 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/1656311698' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T07:52:24.380 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 ceph-mon[32150]: osdmap e89: 12 total, 12 up, 12 in 2024-03-04T07:52:24.380 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 ceph-mon[32150]: from='client.? 172.21.15.116:0/3587729383' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:24.380 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 ceph-mon[32150]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T07:52:24.380 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 ceph-mon[32150]: pgmap v752: 65 pgs: 65 active+clean; 577 KiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T07:52:24.758 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 systemd[1]: Stopping Ceph mon.a for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:25.240 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a[32146]: 2024-03-04T07:52:24.862+0000 7ff8a2a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:52:25.240 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a[32146]: 2024-03-04T07:52:24.862+0000 7ff8a2a00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-04T07:52:25.240 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:24 smithi116 podman[96872]: 2024-03-04 07:52:24.903787403 +0000 UTC m=+0.181783318 container died e0bdece57fe415014a210d8f463ccf707b43a325b45140edf8f73462217462e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T07:52:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:25 smithi116 podman[96872]: 2024-03-04 07:52:25.510926672 +0000 UTC m=+0.788922626 container cleanup e0bdece57fe415014a210d8f463ccf707b43a325b45140edf8f73462217462e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T07:52:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:25 smithi116 bash[96872]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a 2024-03-04T07:52:26.009 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 04 07:52:25 smithi116 podman[96886]: 2024-03-04 07:52:25.750149123 +0000 UTC m=+0.833563366 container remove e0bdece57fe415014a210d8f463ccf707b43a325b45140edf8f73462217462e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-04T07:52:26.201 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.a.service' 2024-03-04T07:52:26.358 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:26.359 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-04T07:52:26.359 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-04T07:52:26.359 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.b 2024-03-04T07:52:26.758 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:26 smithi133 systemd[1]: Stopping Ceph mon.b for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-b[36501]: 2024-03-04T07:52:26.757+0000 7fb739eb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:52:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-b[36501]: 2024-03-04T07:52:26.757+0000 7fb739eb6700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-04T07:52:27.087 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:26 smithi133 podman[70950]: 2024-03-04 07:52:26.992715227 +0000 UTC m=+0.433076355 container died 55bc537d1fecc49ce86fce4d686b7e329c0c06a527f968c354efd561ed538fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-b, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T07:52:27.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:27 smithi133 podman[70950]: 2024-03-04 07:52:27.475569344 +0000 UTC m=+0.915930467 container cleanup 55bc537d1fecc49ce86fce4d686b7e329c0c06a527f968c354efd561ed538fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2) 2024-03-04T07:52:27.587 INFO:journalctl@ceph.mon.b.smithi133.stdout:Mar 04 07:52:27 smithi133 bash[70950]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-b 2024-03-04T07:52:27.945 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.b.service' 2024-03-04T07:52:27.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:27.991 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-04T07:52:27.991 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-04T07:52:27.991 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.c 2024-03-04T07:52:28.271 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:28 smithi152 systemd[1]: Stopping Ceph mon.c for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-c[37149]: 2024-03-04T07:52:28.310+0000 7f90274b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:52:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-c[37149]: 2024-03-04T07:52:28.310+0000 7f90274b9700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-04T07:52:28.771 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:28 smithi152 podman[71763]: 2024-03-04 07:52:28.505255615 +0000 UTC m=+0.312312944 container died d5d2c117cb3670670d739f07f1247e2dae782e87f8cacf41482c1e52b6b070fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-c, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T07:52:29.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:28 smithi152 podman[71763]: 2024-03-04 07:52:28.978803019 +0000 UTC m=+0.785860329 container cleanup d5d2c117cb3670670d739f07f1247e2dae782e87f8cacf41482c1e52b6b070fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-04T07:52:29.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 04 07:52:28 smithi152 bash[71763]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mon-c 2024-03-04T07:52:29.451 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mon.c.service' 2024-03-04T07:52:29.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:29.499 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-04T07:52:29.499 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-04T07:52:29.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.x 2024-03-04T07:52:30.008 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:52:29 smithi116 systemd[1]: Stopping Ceph mgr.x for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:30.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:52:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:52:30.117+0000 7f5279400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:52:30.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:52:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x[32389]: 2024-03-04T07:52:30.117+0000 7f5279400700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-04T07:52:30.509 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:52:30 smithi116 podman[96992]: 2024-03-04 07:52:30.211993795 +0000 UTC m=+0.279360642 container died dff4e95798d3f499296d102432bfefcc544f5245087b381aa964d824b47b810b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T07:52:30.931 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:52:30 smithi116 podman[96992]: 2024-03-04 07:52:30.696035176 +0000 UTC m=+0.763402039 container cleanup dff4e95798d3f499296d102432bfefcc544f5245087b381aa964d824b47b810b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:52:30.932 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:52:30 smithi116 bash[96992]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x 2024-03-04T07:52:31.258 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 04 07:52:30 smithi116 podman[97005]: 2024-03-04 07:52:30.930941423 +0000 UTC m=+0.707018918 container remove dff4e95798d3f499296d102432bfefcc544f5245087b381aa964d824b47b810b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:52:31.447 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-04T07:52:31.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:31.559 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-04T07:52:31.559 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-04T07:52:31.559 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.y 2024-03-04T07:52:31.839 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:52:31 smithi133 systemd[1]: Stopping Ceph mgr.y for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:32.189 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:52:31 smithi133 podman[71076]: 2024-03-04 07:52:31.89252727 +0000 UTC m=+0.146511032 container died e84d6d702eadf4b2079c7e11e913930ff2ff0d9751ee76fec48fcd48b57f6507 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T07:52:32.507 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:52:32 smithi133 podman[71076]: 2024-03-04 07:52:32.297415642 +0000 UTC m=+0.551399413 container cleanup e84d6d702eadf4b2079c7e11e913930ff2ff0d9751ee76fec48fcd48b57f6507 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux ) 2024-03-04T07:52:32.507 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Mar 04 07:52:32 smithi133 bash[71076]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-y 2024-03-04T07:52:32.722 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-04T07:52:32.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:32.761 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-04T07:52:32.762 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-04T07:52:32.762 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.z 2024-03-04T07:52:33.096 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:52:32 smithi152 systemd[1]: Stopping Ceph mgr.z for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:33.351 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:52:33 smithi152 podman[71889]: 2024-03-04 07:52:33.09690702 +0000 UTC m=+0.139108978 container died b76a315aee7387c9a6de4a9d838414854cbe773ea8b116a94bc6acc7cba0af1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:52:33.744 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:52:33 smithi152 podman[71889]: 2024-03-04 07:52:33.471260617 +0000 UTC m=+0.513462560 container cleanup b76a315aee7387c9a6de4a9d838414854cbe773ea8b116a94bc6acc7cba0af1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-04T07:52:33.744 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 04 07:52:33 smithi152 bash[71889]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mgr-z 2024-03-04T07:52:33.889 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mgr.z.service' 2024-03-04T07:52:33.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:33.928 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-04T07:52:33.928 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-04T07:52:33.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.0 2024-03-04T07:52:34.482 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:34 smithi116 systemd[1]: Stopping Ceph osd.0 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:34.758 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:34 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0[44688]: 2024-03-04T07:52:34.481+0000 7fb30f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:52:34.759 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:34 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0[44688]: 2024-03-04T07:52:34.481+0000 7fb30f200700 -1 osd.0 90 *** Got signal Terminated *** 2024-03-04T07:52:34.759 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:34 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0[44688]: 2024-03-04T07:52:34.481+0000 7fb30f200700 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:52:40.258 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:39 smithi116 podman[97111]: 2024-03-04 07:52:39.996673585 +0000 UTC m=+5.671006289 container died 05fe9353c209a9222cd13ab94d0129575cac41d5ce63f6ac325da45d488c479e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-04T07:52:40.650 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:40 smithi116 podman[97111]: 2024-03-04 07:52:40.484381266 +0000 UTC m=+6.158713971 container cleanup 05fe9353c209a9222cd13ab94d0129575cac41d5ce63f6ac325da45d488c479e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:52:40.650 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:40 smithi116 bash[97111]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0 2024-03-04T07:52:40.965 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:40 smithi116 podman[97131]: 2024-03-04 07:52:40.734673247 +0000 UTC m=+0.724793891 container remove 05fe9353c209a9222cd13ab94d0129575cac41d5ce63f6ac325da45d488c479e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-04T07:52:41.850 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:41 smithi116 podman[97232]: 2024-03-04 07:52:41.419650385 +0000 UTC m=+0.032087316 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:52:41.850 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:41 smithi116 podman[97232]: 2024-03-04 07:52:41.545696908 +0000 UTC m=+0.158133825 container create 2693a60a672b58e2db15487d95b82e24790dd0408300a240753aa9ee9e8c811b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-04T07:52:42.259 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:41 smithi116 podman[97232]: 2024-03-04 07:52:41.850025509 +0000 UTC m=+0.462462521 container init 2693a60a672b58e2db15487d95b82e24790dd0408300a240753aa9ee9e8c811b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-04T07:52:42.259 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:41 smithi116 podman[97232]: 2024-03-04 07:52:41.861231136 +0000 UTC m=+0.473668077 container start 2693a60a672b58e2db15487d95b82e24790dd0408300a240753aa9ee9e8c811b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-04T07:52:42.259 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:41 smithi116 podman[97232]: 2024-03-04 07:52:41.974155623 +0000 UTC m=+0.586592631 container attach 2693a60a672b58e2db15487d95b82e24790dd0408300a240753aa9ee9e8c811b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-04T07:52:42.623 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 04 07:52:42 smithi116 podman[97232]: 2024-03-04 07:52:42.281725142 +0000 UTC m=+0.894162070 container died 2693a60a672b58e2db15487d95b82e24790dd0408300a240753aa9ee9e8c811b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:52:43.017 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.0.service' 2024-03-04T07:52:43.135 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:43.136 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-04T07:52:43.136 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-04T07:52:43.136 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.1 2024-03-04T07:52:43.452 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:43 smithi133 systemd[1]: Stopping Ceph osd.1 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:43.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1[41131]: 2024-03-04T07:52:43.451+0000 7f7d8c79a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:52:43.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1[41131]: 2024-03-04T07:52:43.451+0000 7f7d8c79a700 -1 osd.1 90 *** Got signal Terminated *** 2024-03-04T07:52:43.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1[41131]: 2024-03-04T07:52:43.451+0000 7f7d8c79a700 -1 osd.1 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:52:49.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:48 smithi133 podman[71210]: 2024-03-04 07:52:48.803450344 +0000 UTC m=+5.473352528 container died 648aba92109681afde38d60f10d592d459347ef196715b7d4ec7369d69880836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-04T07:52:49.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:49 smithi133 podman[71210]: 2024-03-04 07:52:49.209709094 +0000 UTC m=+5.879611269 container cleanup 648aba92109681afde38d60f10d592d459347ef196715b7d4ec7369d69880836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-04T07:52:49.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:49 smithi133 bash[71210]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1 2024-03-04T07:52:49.768 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:49 smithi133 podman[71225]: 2024-03-04 07:52:49.461257762 +0000 UTC m=+0.653334247 container remove 648aba92109681afde38d60f10d592d459347ef196715b7d4ec7369d69880836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T07:52:50.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:49 smithi133 podman[71322]: 2024-03-04 07:52:49.671516837 +0000 UTC m=+0.017404273 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:52:50.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:49 smithi133 podman[71322]: 2024-03-04 07:52:49.812362115 +0000 UTC m=+0.158249575 container create 2aee7b9afff2db5049b62caf90046ebeabd21b8a2dd5dcda41f92f0ed2a81459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1-deactivate, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-04T07:52:50.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:50 smithi133 podman[71322]: 2024-03-04 07:52:50.048607308 +0000 UTC m=+0.394494728 container init 2aee7b9afff2db5049b62caf90046ebeabd21b8a2dd5dcda41f92f0ed2a81459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-04T07:52:50.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:50 smithi133 podman[71322]: 2024-03-04 07:52:50.052113721 +0000 UTC m=+0.398001136 container start 2aee7b9afff2db5049b62caf90046ebeabd21b8a2dd5dcda41f92f0ed2a81459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-04T07:52:50.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:50 smithi133 podman[71322]: 2024-03-04 07:52:50.148509677 +0000 UTC m=+0.494397126 container attach 2aee7b9afff2db5049b62caf90046ebeabd21b8a2dd5dcda41f92f0ed2a81459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) 2024-03-04T07:52:50.718 INFO:journalctl@ceph.osd.1.smithi133.stdout:Mar 04 07:52:50 smithi133 podman[71322]: 2024-03-04 07:52:50.342678516 +0000 UTC m=+0.688565947 container died 2aee7b9afff2db5049b62caf90046ebeabd21b8a2dd5dcda41f92f0ed2a81459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-04T07:52:51.043 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.1.service' 2024-03-04T07:52:51.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:51.086 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-04T07:52:51.086 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-04T07:52:51.086 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.2 2024-03-04T07:52:51.415 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:51 smithi152 systemd[1]: Stopping Ceph osd.2 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:52:51.771 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2[42291]: 2024-03-04T07:52:51.414+0000 7f02db925700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:52:51.771 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2[42291]: 2024-03-04T07:52:51.414+0000 7f02db925700 -1 osd.2 90 *** Got signal Terminated *** 2024-03-04T07:52:51.771 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2[42291]: 2024-03-04T07:52:51.414+0000 7f02db925700 -1 osd.2 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:52:57.083 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:56 smithi152 podman[72013]: 2024-03-04 07:52:56.798586039 +0000 UTC m=+5.514236582 container died 1c35311fb1ee504bf4495c97d4e42e9a38a56137a8d0939815cbddbd98dfa636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-04T07:52:57.452 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:57 smithi152 podman[72013]: 2024-03-04 07:52:57.164941257 +0000 UTC m=+5.880591792 container cleanup 1c35311fb1ee504bf4495c97d4e42e9a38a56137a8d0939815cbddbd98dfa636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:52:57.452 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:57 smithi152 bash[72013]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2 2024-03-04T07:52:57.755 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:57 smithi152 podman[72043]: 2024-03-04 07:52:57.449939176 +0000 UTC m=+0.642173815 container remove 1c35311fb1ee504bf4495c97d4e42e9a38a56137a8d0939815cbddbd98dfa636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T07:52:58.020 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:57 smithi152 podman[72133]: 2024-03-04 07:52:57.658195506 +0000 UTC m=+0.017370248 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:52:58.020 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:57 smithi152 podman[72133]: 2024-03-04 07:52:57.7760817 +0000 UTC m=+0.135256425 container create 67f94af8e9c6799f8f8505a423c1a48ca78eb8a411778212067724e73466ff30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T07:52:58.020 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:58 smithi152 podman[72133]: 2024-03-04 07:52:58.020732981 +0000 UTC m=+0.379907723 container init 67f94af8e9c6799f8f8505a423c1a48ca78eb8a411778212067724e73466ff30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-04T07:52:58.271 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:58 smithi152 podman[72133]: 2024-03-04 07:52:58.023920389 +0000 UTC m=+0.383095121 container start 67f94af8e9c6799f8f8505a423c1a48ca78eb8a411778212067724e73466ff30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T07:52:58.271 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:58 smithi152 podman[72133]: 2024-03-04 07:52:58.095602369 +0000 UTC m=+0.454777117 container attach 67f94af8e9c6799f8f8505a423c1a48ca78eb8a411778212067724e73466ff30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2-deactivate, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T07:52:58.598 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 04 07:52:58 smithi152 podman[72133]: 2024-03-04 07:52:58.321306984 +0000 UTC m=+0.680481729 container died 67f94af8e9c6799f8f8505a423c1a48ca78eb8a411778212067724e73466ff30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, ceph=True) 2024-03-04T07:52:58.929 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.2.service' 2024-03-04T07:52:58.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:52:58.974 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-04T07:52:58.974 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-04T07:52:58.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.3 2024-03-04T07:52:59.509 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:52:59 smithi116 systemd[1]: Stopping Ceph osd.3 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:00.009 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:52:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3[51662]: 2024-03-04T07:52:59.596+0000 7fd8c5000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:00.009 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:52:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3[51662]: 2024-03-04T07:52:59.596+0000 7fd8c5000700 -1 osd.3 90 *** Got signal Terminated *** 2024-03-04T07:53:00.009 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:52:59 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3[51662]: 2024-03-04T07:52:59.596+0000 7fd8c5000700 -1 osd.3 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:53:00.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:00.156+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:01.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:00 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:00.994+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:01.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:01.187+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:01.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:01 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:01.393+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:02.409 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:02 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:02.024+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:02.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:02.143+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:02.767 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:02 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:02.408+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:03.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:03 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:03.007+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:03.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:03.176+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:03.758 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:03 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3[51662]: 2024-03-04T07:53:03.287+0000 7fd8c0800700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:37.376527+0000 front 2024-03-04T07:52:37.376586+0000 (oldest deadline 2024-03-04T07:53:03.276656+0000) 2024-03-04T07:53:03.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:03 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:03.435+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:04.337 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:04 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:04.036+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:04.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:04 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:04.148+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:04.758 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:04 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3[51662]: 2024-03-04T07:53:04.337+0000 7fd8c0800700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:37.376527+0000 front 2024-03-04T07:52:37.376586+0000 (oldest deadline 2024-03-04T07:53:03.276656+0000) 2024-03-04T07:53:04.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:04 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:04.424+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:05.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:04.854+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:05.384 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:05 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:05.010+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:05.384 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:05 smithi116 podman[97337]: 2024-03-04 07:53:05.10218175 +0000 UTC m=+5.706841236 container died 9dc47f04885484cc733d81fb6feb246368963bc14db754e7a101d1d0e48e3d27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-03-04T07:53:05.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:05 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:05.116+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:05.758 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:05 smithi116 podman[97337]: 2024-03-04 07:53:05.589014752 +0000 UTC m=+6.193674267 container cleanup 9dc47f04885484cc733d81fb6feb246368963bc14db754e7a101d1d0e48e3d27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) 2024-03-04T07:53:05.759 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:05 smithi116 bash[97337]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3 2024-03-04T07:53:05.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:05 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:05.383+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:06.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:05.806+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:06.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:05.721+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:06.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:06 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:05.999+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:06.258 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:05 smithi116 podman[97355]: 2024-03-04 07:53:05.907138054 +0000 UTC m=+0.791058799 container remove 9dc47f04885484cc733d81fb6feb246368963bc14db754e7a101d1d0e48e3d27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-04T07:53:06.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:06 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:06.141+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:06.742 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:06 smithi116 podman[97446]: 2024-03-04 07:53:06.428036516 +0000 UTC m=+0.034440677 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:53:06.742 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:06 smithi116 podman[97446]: 2024-03-04 07:53:06.586587761 +0000 UTC m=+0.192991949 container create 2dad687b886ef3407fd2f1b95a32ad878ad8fdd3eb0fd29fa8ce9cb76c04ffcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:53:06.742 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:06 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:06.429+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:07.009 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:06 smithi116 podman[97446]: 2024-03-04 07:53:06.870334923 +0000 UTC m=+0.476739117 container init 2dad687b886ef3407fd2f1b95a32ad878ad8fdd3eb0fd29fa8ce9cb76c04ffcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-04T07:53:07.009 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:06 smithi116 podman[97446]: 2024-03-04 07:53:06.881159648 +0000 UTC m=+0.487563842 container start 2dad687b886ef3407fd2f1b95a32ad878ad8fdd3eb0fd29fa8ce9cb76c04ffcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T07:53:07.009 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:06 smithi116 podman[97446]: 2024-03-04 07:53:06.985831491 +0000 UTC m=+0.592235745 container attach 2dad687b886ef3407fd2f1b95a32ad878ad8fdd3eb0fd29fa8ce9cb76c04ffcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:53:07.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:06.705+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:07.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:06.763+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:07.310 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:07 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:07.009+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:07.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:07 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:07.125+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:07.675 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:07 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:07.418+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:07.676 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:07 smithi116 podman[97472]: 2024-03-04 07:53:07.360186467 +0000 UTC m=+0.031837435 container died 2dad687b886ef3407fd2f1b95a32ad878ad8fdd3eb0fd29fa8ce9cb76c04ffcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-03-04T07:53:08.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:07 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:07.703+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:08.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:07 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:07.810+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:08.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.3.service' 2024-03-04T07:53:08.256 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:07 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:07.978+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:08.256 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:08 smithi116 podman[97472]: 2024-03-04 07:53:08.1400753 +0000 UTC m=+0.811726304 container remove 2dad687b886ef3407fd2f1b95a32ad878ad8fdd3eb0fd29fa8ce9cb76c04ffcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-03-04T07:53:08.256 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:08 smithi116 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-04T07:53:08.256 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:08 smithi116 systemd[1]: Stopped Ceph osd.3 for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:53:08.256 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 04 07:53:08 smithi116 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.3.service: Consumed 21.070s CPU time. 2024-03-04T07:53:08.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:53:08.289 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-04T07:53:08.290 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-04T07:53:08.290 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.4 2024-03-04T07:53:08.508 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:08 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:08.457+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:08.587 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:08 smithi133 systemd[1]: Stopping Ceph osd.4 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:08.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:08 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:08.094+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:09.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:08.666+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:09.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:08.796+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:09.087 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:08 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:53:08.605+0000 7fb331a50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:09.087 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:08 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:53:08.605+0000 7fb331a50700 -1 osd.4 90 *** Got signal Terminated *** 2024-03-04T07:53:09.087 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:08 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:53:08.605+0000 7fb331a50700 -1 osd.4 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:53:09.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:08 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:08.972+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:09.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:09 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:09.128+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:09.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:09 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:09.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:10.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:09.773+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:10.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:09.679+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:10.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:09 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:09.995+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:10.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:10 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:10.133+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:10.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:10 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:10.133+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:10.760 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:10 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:10.492+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:11.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:10.743+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:11.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:10.697+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:11.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:10.697+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:11.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:10 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:10.970+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:11.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:10 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:10.970+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:11.374 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:11 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:11.099+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:11.374 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:11 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:11.099+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:11.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:11 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:11.474+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:11.837 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:11 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:53:11.373+0000 7fb32a03c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:48.085333+0000 front 2024-03-04T07:52:48.085362+0000 (oldest deadline 2024-03-04T07:53:10.385443+0000) 2024-03-04T07:53:12.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:11.681+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:12.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:11.681+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:12.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:11.852+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:12.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:11.698+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:12.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:11 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:11.980+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:12.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:11 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:11.980+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:12.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:12 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:12.082+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:12.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:12 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:12.082+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:12.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:12 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:12.458+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:12.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:12 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:12.458+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:12.837 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:12 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:53:12.420+0000 7fb32a03c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:48.085333+0000 front 2024-03-04T07:52:48.085362+0000 (oldest deadline 2024-03-04T07:53:10.385443+0000) 2024-03-04T07:53:13.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:12 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:12.849+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:13.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:12 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:12.704+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:13.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:12 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:12.704+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:13.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:12 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:12.747+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:13.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:12 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:12.747+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:13.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:12 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:12.931+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:13.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:12 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:12.931+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:13.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:12 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:12.980+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:13.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:13 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:13.046+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:13.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:13 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:13.046+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:13.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:13 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:13.469+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:13.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:13 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:13.469+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:13.837 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:13 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4[47842]: 2024-03-04T07:53:13.383+0000 7fb32a03c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:48.085333+0000 front 2024-03-04T07:52:48.085362+0000 (oldest deadline 2024-03-04T07:53:10.385443+0000) 2024-03-04T07:53:14.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:13 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:13.671+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:14.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:13 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:13.671+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:14.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:13 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:13.827+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:14.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:13 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:13.765+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:14.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:13 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:13.765+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:14.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:13 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:13.967+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:14.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:13 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:13.967+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:14.331 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:14 smithi133 podman[71440]: 2024-03-04 07:53:14.039117869 +0000 UTC m=+5.558123742 container died c49399a54bcbe37d6a8988c6ba9163a18d4049224e6cb234af9ef0e83116cc7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5) 2024-03-04T07:53:14.331 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:14 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:14.021+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:14.332 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:14 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:14.028+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:14.332 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:14 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:14.028+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:14.587 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:14 smithi133 podman[71440]: 2024-03-04 07:53:14.455113869 +0000 UTC m=+5.974119733 container cleanup c49399a54bcbe37d6a8988c6ba9163a18d4049224e6cb234af9ef0e83116cc7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, ceph=True) 2024-03-04T07:53:14.587 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:14 smithi133 bash[71440]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4 2024-03-04T07:53:14.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:14.472+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:14.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:14.472+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:15.008 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:14 smithi133 podman[71457]: 2024-03-04 07:53:14.681814722 +0000 UTC m=+0.639332778 container remove c49399a54bcbe37d6a8988c6ba9163a18d4049224e6cb234af9ef0e83116cc7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:53:15.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:14 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:14.838+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:15.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:14 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:14.698+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:15.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:14 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:14.698+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:15.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:14 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:14.764+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:15.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:14 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:14.764+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:15.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:14.955+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:15.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:14.955+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:15.263 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:15 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:15.007+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:15.263 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:15 smithi133 podman[71549]: 2024-03-04 07:53:14.978875361 +0000 UTC m=+0.017752453 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:53:15.263 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:15 smithi133 podman[71549]: 2024-03-04 07:53:15.116069337 +0000 UTC m=+0.154946466 container create 05243ba59292c8322e9cf4131c3598d1b0398a46fe7453a61a81a64d681e81e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-04T07:53:15.263 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:15 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:15.013+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:15.263 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:15 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:15.013+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:15.674 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:15 smithi133 podman[71549]: 2024-03-04 07:53:15.385538407 +0000 UTC m=+0.424415494 container init 05243ba59292c8322e9cf4131c3598d1b0398a46fe7453a61a81a64d681e81e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-04T07:53:15.674 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:15 smithi133 podman[71549]: 2024-03-04 07:53:15.388743701 +0000 UTC m=+0.427620783 container start 05243ba59292c8322e9cf4131c3598d1b0398a46fe7453a61a81a64d681e81e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222) 2024-03-04T07:53:15.674 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:15 smithi133 podman[71549]: 2024-03-04 07:53:15.485393762 +0000 UTC m=+0.524270866 container attach 05243ba59292c8322e9cf4131c3598d1b0398a46fe7453a61a81a64d681e81e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-04T07:53:15.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:15 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:15.477+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:15.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:15 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:15.477+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:16.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:15 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:15.855+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:16.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:15 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:15.721+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:16.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:15 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:15.721+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:16.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:15 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:15.770+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:16.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:15 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:15.770+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:16.022 INFO:journalctl@ceph.osd.4.smithi133.stdout:Mar 04 07:53:15 smithi133 podman[71549]: 2024-03-04 07:53:15.674552022 +0000 UTC m=+0.713429123 container died 05243ba59292c8322e9cf4131c3598d1b0398a46fe7453a61a81a64d681e81e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-04T07:53:16.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:15 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:15.981+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:16.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:15 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:15.981+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:16.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:16 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:16.049+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:16.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:16 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:16.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:16.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:16 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:16.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:16.388 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.4.service' 2024-03-04T07:53:16.432 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:53:16.432 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-04T07:53:16.432 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-04T07:53:16.432 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.5 2024-03-04T07:53:16.737 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:16 smithi152 systemd[1]: Stopping Ceph osd.5 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:16.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:16.467+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:16.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:16.467+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:17.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:16.894+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:17.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:16.745+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:17.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:16.745+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:17.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:16.736+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:17.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:16.736+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:17.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:16.743+0000 7f3433891700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:17.023 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:16.743+0000 7f3433891700 -1 osd.5 90 *** Got signal Terminated *** 2024-03-04T07:53:17.023 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:16 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:16.743+0000 7f3433891700 -1 osd.5 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:53:17.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:16.947+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:17.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:16 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:16.947+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:17.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:17 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:17.085+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:17.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:17 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:17.012+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:17.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:17 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:17.012+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:17.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:17 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:17.457+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:17.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:17 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:17.457+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:18.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:17 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:17.770+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:18.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:17 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:17.770+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:18.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:17 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:17.770+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.199220+0000 front 2024-03-04T07:52:51.199329+0000 (oldest deadline 2024-03-04T07:53:17.099351+0000) 2024-03-04T07:53:18.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:17 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:17.852+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:18.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:17 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:17.852+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:18.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:17 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:17.713+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:18.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:17 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:17.713+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:18.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:17 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:17.950+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:18.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:17 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:17.950+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:18.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:18 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:18.115+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:18.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:18 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:18.013+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:18.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:18 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:18.013+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:18.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:18.459+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:18.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:18.459+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:19.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:18.770+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:19.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:18.770+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:19.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:18.770+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.199220+0000 front 2024-03-04T07:52:51.199329+0000 (oldest deadline 2024-03-04T07:53:17.099351+0000) 2024-03-04T07:53:19.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:18.826+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:19.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:18.826+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:19.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:18.736+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:19.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:18.736+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:19.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:18 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:18.736+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:19.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:18.904+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:19.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:18.904+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:19.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:19 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:19.122+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:19.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:19 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:19.122+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:19.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:19 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:19.050+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:19.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:19 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:19.050+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:19.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:19 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:19.050+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:19.952 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:19.510+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:19.952 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:19.510+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:20.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:19.753+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:20.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:19.753+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:20.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:19.753+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.199220+0000 front 2024-03-04T07:52:51.199329+0000 (oldest deadline 2024-03-04T07:53:17.099351+0000) 2024-03-04T07:53:20.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:19.843+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:20.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:19.843+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:20.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:19.766+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:20.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:19.766+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:20.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:19 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:19.766+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:20.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:19.951+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:20.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:19 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:19.951+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:20.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:20.097+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:20.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:20.097+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:20.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:20.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:20.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:20.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:20.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:20 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:20.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:20.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:20.475+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:20.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:20.475+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:21.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:20.753+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:21.021 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:20.753+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:21.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:20.753+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.199220+0000 front 2024-03-04T07:52:51.199329+0000 (oldest deadline 2024-03-04T07:53:17.099351+0000) 2024-03-04T07:53:21.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:20.834+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:21.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:20.834+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:21.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:20.718+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:21.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:20.718+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:21.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:20 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:20.718+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:21.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:20.908+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:21.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:20 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:20.908+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:21.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:21.141+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:21.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:21.141+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:21.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:21.030+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:21.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:21.030+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:21.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:21 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:21.030+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:21.896 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:21 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:21.525+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:21.897 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:21 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:21.525+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:22.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:21.787+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:22.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:21.787+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:22.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:21.693+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:22.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:21.693+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:22.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:21.693+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:22.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:21.716+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:38.897415+0000 front 2024-03-04T07:52:38.897657+0000 (oldest deadline 2024-03-04T07:53:04.797197+0000) 2024-03-04T07:53:22.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:21.716+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:44.797968+0000 front 2024-03-04T07:52:44.797775+0000 (oldest deadline 2024-03-04T07:53:10.697795+0000) 2024-03-04T07:53:22.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:21 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5[48842]: 2024-03-04T07:53:21.716+0000 7f342fe85700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.199220+0000 front 2024-03-04T07:52:51.199329+0000 (oldest deadline 2024-03-04T07:53:17.099351+0000) 2024-03-04T07:53:22.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:21 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:21.896+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:22.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:21 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:21.896+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:22.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:21 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:21.896+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:22.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:22.178+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:22.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:22.178+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:22.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:22.008+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:22.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:22.008+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:22.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:22.008+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:22.478 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:22 smithi152 podman[72236]: 2024-03-04 07:53:22.159534929 +0000 UTC m=+5.525952885 container died 4db1d6eaf80b409db796306266d8d91cefe68f0916fb4a82376a5281743f61b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-04T07:53:22.771 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:22.685+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:22.771 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:22.685+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:22.771 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:22.685+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:22.772 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:22 smithi152 podman[72236]: 2024-03-04 07:53:22.586133716 +0000 UTC m=+5.952551661 container cleanup 4db1d6eaf80b409db796306266d8d91cefe68f0916fb4a82376a5281743f61b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True) 2024-03-04T07:53:22.772 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:22 smithi152 bash[72236]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5 2024-03-04T07:53:22.936 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:22.521+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:22.936 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:22.521+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:22.937 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:22.521+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:23.190 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:22.781+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:23.190 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:22 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:22.781+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:23.191 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:22 smithi152 podman[72255]: 2024-03-04 07:53:22.787451715 +0000 UTC m=+0.632166943 container remove 4db1d6eaf80b409db796306266d8d91cefe68f0916fb4a82376a5281743f61b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:53:23.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:22.936+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:23.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:22.936+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:23.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:22 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:22.936+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:23.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:23 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:23.182+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:23.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:23 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:23.182+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:23.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:22.971+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:23.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:22.971+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:23.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:22 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:22.971+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:23.508 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:23 smithi152 podman[72353]: 2024-03-04 07:53:23.124196314 +0000 UTC m=+0.017988922 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:53:23.508 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:23 smithi152 podman[72353]: 2024-03-04 07:53:23.247275979 +0000 UTC m=+0.141068573 container create 11f8dcb5e95bbf6bfb4c9b779c962d359bb01ea7641d49d0379856ff0d298e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T07:53:23.509 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:23 smithi152 podman[72353]: 2024-03-04 07:53:23.508688417 +0000 UTC m=+0.402481017 container init 11f8dcb5e95bbf6bfb4c9b779c962d359bb01ea7641d49d0379856ff0d298e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-04T07:53:23.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:23 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:23.487+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:23.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:23 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:23.487+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:23.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:23 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:23.487+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:23.766 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:23 smithi152 podman[72353]: 2024-03-04 07:53:23.51228121 +0000 UTC m=+0.406073805 container start 11f8dcb5e95bbf6bfb4c9b779c962d359bb01ea7641d49d0379856ff0d298e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:53:23.766 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:23 smithi152 podman[72353]: 2024-03-04 07:53:23.600016131 +0000 UTC m=+0.493808727 container attach 11f8dcb5e95bbf6bfb4c9b779c962d359bb01ea7641d49d0379856ff0d298e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS) 2024-03-04T07:53:23.766 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:23 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:23.690+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:23.766 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:23 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:23.690+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:23.766 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:23 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:23.690+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:24.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:23 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:23.764+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:24.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:23 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:23.764+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:24.022 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 04 07:53:23 smithi152 podman[72353]: 2024-03-04 07:53:23.809177665 +0000 UTC m=+0.702970269 container died 11f8dcb5e95bbf6bfb4c9b779c962d359bb01ea7641d49d0379856ff0d298e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux ) 2024-03-04T07:53:24.207 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:23 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:23.940+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:24.207 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:23 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:23.940+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:24.208 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:23 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:23.940+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:24.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:23 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:23.888+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:24.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:23 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:23.888+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:24.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:23 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:23.888+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:24.445 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.5.service' 2024-03-04T07:53:24.487 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:53:24.487 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-04T07:53:24.487 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-04T07:53:24.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.6 2024-03-04T07:53:24.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:24.207+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:24.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:24.207+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:24.758 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:24 smithi116 systemd[1]: Stopping Ceph osd.6 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:24.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:24.499+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:24.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:24.499+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:24.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:24.499+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:25.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:24.766+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:25.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:24.766+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:25.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:24.721+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:25.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:24.721+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:25.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:24 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:24.721+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:25.192 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:24.909+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:25.192 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:24.909+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:25.192 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:24 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:24.909+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:25.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:24.857+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:25.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:24.857+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:25.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:24.857+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:25.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:25.065+0000 7f7732800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:25.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:25.065+0000 7f7732800700 -1 osd.6 90 *** Got signal Terminated *** 2024-03-04T07:53:25.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:25.065+0000 7f7732800700 -1 osd.6 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:53:25.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:25.191+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:25.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:25.191+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:25.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:25.548+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:25.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:25.548+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:25.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:25.548+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:26.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:25 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:25.768+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:26.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:25 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:25.768+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:26.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:25 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:25.752+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:26.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:25 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:25.752+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:26.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:25 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:25.752+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:26.230 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:25.887+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:26.230 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:25.887+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:26.230 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:25 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:25.887+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:26.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:25.905+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:26.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:25.905+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:26.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:25 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:25.905+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:26.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:26.229+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:26.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:26.229+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:26.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:26.229+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:26.947 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:26.515+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:26.947 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:26.515+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:26.947 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:26.515+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:26.947 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:26.515+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:27.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:26.945+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:27.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:26.946+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:27.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:26.946+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:27.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:26 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:26.946+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.373196+0000 front 2024-03-04T07:53:01.373333+0000 (oldest deadline 2024-03-04T07:53:26.073530+0000) 2024-03-04T07:53:27.271 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:26 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:26.779+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:27.271 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:26 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:26.779+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:27.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:26 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:26.787+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:27.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:26 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:26.787+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:27.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:26 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:26.787+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:27.275 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:26.918+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:27.275 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:26.918+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:27.275 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:26 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:26.918+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:27.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:27.275+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:27.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:27.275+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:27.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:27.275+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:27.917 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:27.543+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:27.917 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:27.543+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:27.917 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:27.543+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:27.917 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:27.543+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:28.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:27 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:27.753+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:28.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:27 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:27.753+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:28.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:27 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:27.818+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:28.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:27 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:27.818+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:28.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:27 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:27.818+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:28.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:27.914+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:28.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:27.914+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:28.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:27.914+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:28.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:27 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:27.914+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.373196+0000 front 2024-03-04T07:53:01.373333+0000 (oldest deadline 2024-03-04T07:53:26.073530+0000) 2024-03-04T07:53:28.323 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:27.903+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:28.323 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:27.903+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:28.323 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:27 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:27.903+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:28.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:28 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:28.322+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:28.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:28 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:28.322+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:28.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:28 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:28.322+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:28.942 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:28.529+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:28.942 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:28.529+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:28.942 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:28.529+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:28.942 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:28.529+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:29.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:28.741+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:29.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:28.741+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:29.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:28.741+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:29.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:28.859+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:29.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:28.859+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:29.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:28 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:28.859+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:29.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:28.940+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:29.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:28.940+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:29.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:28.940+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:29.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:28 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:28.940+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.373196+0000 front 2024-03-04T07:53:01.373333+0000 (oldest deadline 2024-03-04T07:53:26.073530+0000) 2024-03-04T07:53:29.328 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:28 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:28.935+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:29.328 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:28 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:28.935+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:29.328 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:28 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:28.935+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:29.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:29 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:29.328+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:29.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:29 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:29.328+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:29.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:29 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:29.328+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:29.963 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:29.525+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:29.963 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:29.525+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:29.963 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:29.525+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:29.963 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:29.525+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:30.258 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:29.961+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.365517+0000 front 2024-03-04T07:52:36.364928+0000 (oldest deadline 2024-03-04T07:53:00.464454+0000) 2024-03-04T07:53:30.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:29.961+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.470127+0000 front 2024-03-04T07:52:45.469996+0000 (oldest deadline 2024-03-04T07:53:10.770091+0000) 2024-03-04T07:53:30.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:29.961+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.072578+0000 front 2024-03-04T07:52:56.072644+0000 (oldest deadline 2024-03-04T07:53:21.372208+0000) 2024-03-04T07:53:30.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:29 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6[58194]: 2024-03-04T07:53:29.961+0000 7f7729000700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.373196+0000 front 2024-03-04T07:53:01.373333+0000 (oldest deadline 2024-03-04T07:53:26.073530+0000) 2024-03-04T07:53:30.271 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:29.780+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:30.271 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:29.780+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:30.271 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:29.780+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:30.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:29.850+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:30.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:29.850+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:30.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:29 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:29.850+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:30.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:29 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:29.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:30.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:29 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:29.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:30.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:29 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:29.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:30.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:29 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:29.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:30.608 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:30.354+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:30.608 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:30.354+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:30.608 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:30.354+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:30.890 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:30 smithi116 podman[97563]: 2024-03-04 07:53:30.570682922 +0000 UTC m=+5.637127681 container died c74a69216f072eabc423fc865f045a949c2666e96595269064113f0ee162cc4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:53:30.890 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:30.557+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:30.891 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:30.557+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:30.891 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:30.557+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:30.891 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:30.557+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:31.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:30 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:30.738+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:31.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:30 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:30.738+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:31.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:30 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:30.738+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:31.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:30 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:30.878+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:31.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:30 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:30.878+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:31.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:30 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:30.878+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:31.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:31 smithi116 podman[97563]: 2024-03-04 07:53:31.009094126 +0000 UTC m=+6.075538922 container cleanup c74a69216f072eabc423fc865f045a949c2666e96595269064113f0ee162cc4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True) 2024-03-04T07:53:31.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:31 smithi116 bash[97563]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6 2024-03-04T07:53:31.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:30.932+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:31.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:30.932+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:31.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:30.932+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:31.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:30 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:30.932+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:31.513 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:31 smithi116 podman[97583]: 2024-03-04 07:53:31.260231633 +0000 UTC m=+0.678777761 container remove c74a69216f072eabc423fc865f045a949c2666e96595269064113f0ee162cc4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T07:53:31.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:31 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:31.360+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:31.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:31 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:31.360+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:31.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:31 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:31.360+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:32.008 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:31 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:31.511+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:32.008 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:31 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:31.511+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:32.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:31 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:31.511+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:32.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:31 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:31.511+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:32.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:31 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:31.692+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:32.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:31 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:31.692+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:32.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:31 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:31.692+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:32.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:31 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:31.872+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:32.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:31 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:31.872+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:32.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:31 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:31.872+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:32.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:31 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:31.950+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:32.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:31 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:31.950+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:32.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:31 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:31.950+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:32.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:31 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:31.950+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:32.466 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:32 smithi116 podman[97671]: 2024-03-04 07:53:32.062657328 +0000 UTC m=+0.040164169 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:53:32.466 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:32 smithi116 podman[97671]: 2024-03-04 07:53:32.152807406 +0000 UTC m=+0.130314267 container create 5ad3c415a04ef78090e56663a5c00fdfd55760c0f541e3da87205f777f6cbdd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:53:32.466 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:32 smithi116 podman[97671]: 2024-03-04 07:53:32.465172946 +0000 UTC m=+0.442679814 container init 5ad3c415a04ef78090e56663a5c00fdfd55760c0f541e3da87205f777f6cbdd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-04T07:53:32.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:32 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:32.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:32.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:32 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:32.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:32.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:32 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:32.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:32.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:32 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:32.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:32.760 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:32 smithi116 podman[97671]: 2024-03-04 07:53:32.476604101 +0000 UTC m=+0.454110969 container start 5ad3c415a04ef78090e56663a5c00fdfd55760c0f541e3da87205f777f6cbdd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6-deactivate, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-04T07:53:32.760 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:32 smithi116 podman[97671]: 2024-03-04 07:53:32.555258384 +0000 UTC m=+0.532765275 container attach 5ad3c415a04ef78090e56663a5c00fdfd55760c0f541e3da87205f777f6cbdd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, ceph=True) 2024-03-04T07:53:32.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:32 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:32.381+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:32.838 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:32 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:32.381+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:32.838 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:32 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:32.381+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:32.906 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:32 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:32.668+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:32.907 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:32 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:32.668+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:32.907 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:32 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:32.668+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:33.259 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:32 smithi116 podman[97671]: 2024-03-04 07:53:32.904374204 +0000 UTC m=+0.881881053 container died 5ad3c415a04ef78090e56663a5c00fdfd55760c0f541e3da87205f777f6cbdd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:53:33.261 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:32 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:32.906+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:33.262 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:32 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:32.906+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:33.262 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:32 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:32.906+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:33.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:32 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:32.948+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:33.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:32 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:32.948+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:33.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:32 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:32.948+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:33.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:32 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:32.948+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:33.721 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.6.service' 2024-03-04T07:53:33.804 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:33 smithi116 podman[97697]: 2024-03-04 07:53:33.667197257 +0000 UTC m=+0.747552318 container remove 5ad3c415a04ef78090e56663a5c00fdfd55760c0f541e3da87205f777f6cbdd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-04T07:53:33.804 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:33 smithi116 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-04T07:53:33.804 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:33 smithi116 systemd[1]: Stopped Ceph osd.6 for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:53:33.805 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 04 07:53:33 smithi116 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.6.service: Consumed 21.293s CPU time. 2024-03-04T07:53:33.805 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:33 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:33.549+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:33.805 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:33 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:33.549+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:33.805 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:33 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:33.549+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:33.805 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:33 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:33.549+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:33.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:53:33.831 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-04T07:53:33.831 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-04T07:53:33.831 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.7 2024-03-04T07:53:33.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:33.368+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:33.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:33.368+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:33.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:33.368+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:33.945 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:33 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:33.633+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:33.945 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:33 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:33.633+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:33.945 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:33 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:33.633+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:34.157 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:33 smithi133 systemd[1]: Stopping Ceph osd.7 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:34.158 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:33.945+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:34.158 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:33.945+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:34.158 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:33.945+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:34.158 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:33 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:33.945+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:34.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:33 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:33.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:34.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:33 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:33.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:34.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:33 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:33.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:34.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:34.156+0000 7fc5a17e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:34.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:34.156+0000 7fc5a17e4700 -1 osd.7 90 *** Got signal Terminated *** 2024-03-04T07:53:34.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:34.156+0000 7fc5a17e4700 -1 osd.7 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:53:34.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:34.401+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:34.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:34.401+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:34.587 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:34.401+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:34.964 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:34 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:34.588+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:34.964 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:34 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:34.588+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:34.964 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:34 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:34.588+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:35.008 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:34 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:34.555+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:35.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:34 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:34.555+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:35.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:34 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:34.555+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:35.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:34 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:34.555+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:35.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:34 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:34.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:35.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:34 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:34.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:35.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:34 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:34.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:35.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:34.917+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:35.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:34.917+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:35.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:34.917+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:35.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:34 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:34.917+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:35.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:35.445+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:35.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:35.445+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:35.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:35.445+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:35.965 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:35 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:35.549+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:35.965 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:35 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:35.549+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:35.965 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:35 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:35.549+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:36.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:35 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:35.590+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:36.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:35 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:35.590+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:36.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:35 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:35.590+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:36.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:35 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:35.590+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:36.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:35 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:35.590+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:36.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:35 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:35.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:36.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:35 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:35.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:36.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:35 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:35.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:36.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:35.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:36.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:35.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:36.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:35.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:36.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:35.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:36.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:36.449+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:36.838 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:36.449+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:36.838 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:36.449+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:36.973 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:36.557+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:36.974 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:36.557+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:36.974 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:36.557+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:36.974 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:36.557+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:37.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:36 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:36.596+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:37.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:36 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:36.596+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:37.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:36 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:36.596+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:37.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:36 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:36.596+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:37.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:36 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:36.596+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:37.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:36.972+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:37.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:36.972+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:37.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:36.972+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:37.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:36 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:36.972+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:37.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:36.978+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:37.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:36.978+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:37.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:36.978+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:37.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:36.978+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:37.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:36 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:36.978+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:37.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:37 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:37.472+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:37.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:37 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:37.472+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:37.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:37 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:37.472+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:37.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:37 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:37.472+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:11.757471+0000 front 2024-03-04T07:53:11.757504+0000 (oldest deadline 2024-03-04T07:53:36.457619+0000) 2024-03-04T07:53:37.969 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:37.567+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:37.969 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:37.567+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:37.969 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:37.567+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:37.969 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:37.567+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:38.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:37 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:37.562+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:38.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:37 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:37.562+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:38.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:37 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:37.562+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:38.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:37 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:37.562+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:38.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:37 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:37.562+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:38.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:37.968+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:38.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:37.968+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:38.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:37.968+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:38.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:37 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:37.968+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:38.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:38.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:38.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:38.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:38.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:38.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:38.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:38.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:38.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:38.007+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:38.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:38.505+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:46.753113+0000 front 2024-03-04T07:52:46.753195+0000 (oldest deadline 2024-03-04T07:53:12.053187+0000) 2024-03-04T07:53:38.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:38.505+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:53.154274+0000 front 2024-03-04T07:52:53.154160+0000 (oldest deadline 2024-03-04T07:53:19.054172+0000) 2024-03-04T07:53:38.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:38.505+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:02.555706+0000 front 2024-03-04T07:53:02.555670+0000 (oldest deadline 2024-03-04T07:53:25.455520+0000) 2024-03-04T07:53:38.837 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:38 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7[54449]: 2024-03-04T07:53:38.505+0000 7fc59ddd8700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:11.757471+0000 front 2024-03-04T07:53:11.757504+0000 (oldest deadline 2024-03-04T07:53:36.457619+0000) 2024-03-04T07:53:38.933 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:38.521+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:38.933 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:38.521+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:38.933 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:38.521+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:38.934 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:38.521+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:39.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:38 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:38.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:39.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:38 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:38.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:39.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:38 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:38.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:39.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:38 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:38.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:39.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:38 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:38.527+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:39.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:38.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:39.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:38.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:39.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:38.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:39.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:38 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:38.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:39.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.015+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:39.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.015+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:39.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.015+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:39.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.015+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:39.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.015+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:39.760 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:39 smithi133 podman[71666]: 2024-03-04 07:53:39.461469891 +0000 UTC m=+5.437630130 container died 0a60e45a29ef75d21e02289bea8e98f8a75dc160f8a8ac6a45163a5e7a67adf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-04T07:53:39.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:39.498+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:39.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:39.498+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:39.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:39.498+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:39.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:39.498+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:40.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:39 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:39.538+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:40.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:39 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:39.538+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:40.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:39 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:39.538+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:40.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:39 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:39.538+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:40.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:39 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:39.538+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:40.051 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:39 smithi133 podman[71666]: 2024-03-04 07:53:39.858941728 +0000 UTC m=+5.835101955 container cleanup 0a60e45a29ef75d21e02289bea8e98f8a75dc160f8a8ac6a45163a5e7a67adf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-04T07:53:40.052 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:39 smithi133 bash[71666]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7 2024-03-04T07:53:40.052 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.984+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:40.052 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.984+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:40.052 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.984+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:40.052 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.984+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:40.052 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:39 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:39.984+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:40.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:39.886+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:40.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:39.886+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:40.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:39.886+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:40.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:39 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:39.886+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:40.337 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:40 smithi133 podman[71683]: 2024-03-04 07:53:40.051913005 +0000 UTC m=+0.589099029 container remove 0a60e45a29ef75d21e02289bea8e98f8a75dc160f8a8ac6a45163a5e7a67adf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-04T07:53:40.705 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:40 smithi133 podman[71773]: 2024-03-04 07:53:40.349491896 +0000 UTC m=+0.017448012 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:53:40.705 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:40 smithi133 podman[71773]: 2024-03-04 07:53:40.477660936 +0000 UTC m=+0.145617043 container create 19d552f2ef886fd98484c69af8f0ef5cb56bd3d9daca2b0d6776443521b0f09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-04T07:53:40.705 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:40 smithi133 podman[71773]: 2024-03-04 07:53:40.705573073 +0000 UTC m=+0.373529188 container init 19d552f2ef886fd98484c69af8f0ef5cb56bd3d9daca2b0d6776443521b0f09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:53:40.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:40.489+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:40.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:40.489+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:40.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:40.489+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:40.772 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:40.489+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:40.911 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:40 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:40.554+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:40.911 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:40 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:40.554+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:40.911 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:40 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:40.554+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:40.911 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:40 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:40.554+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:40.911 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:40 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:40.554+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:41.002 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:40 smithi133 podman[71773]: 2024-03-04 07:53:40.708659094 +0000 UTC m=+0.376615213 container start 19d552f2ef886fd98484c69af8f0ef5cb56bd3d9daca2b0d6776443521b0f09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7-deactivate, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-04T07:53:41.002 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:40 smithi133 podman[71773]: 2024-03-04 07:53:40.805444059 +0000 UTC m=+0.473400191 container attach 19d552f2ef886fd98484c69af8f0ef5cb56bd3d9daca2b0d6776443521b0f09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-04T07:53:41.002 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:40 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:40.947+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:41.002 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:40 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:40.947+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:41.002 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:40 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:40.947+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:41.002 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:40 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:40.947+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:41.003 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:40 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:40.947+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:41.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:40.848+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:41.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:40.848+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:41.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:40.848+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:41.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:40 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:40.848+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:41.317 INFO:journalctl@ceph.osd.7.smithi133.stdout:Mar 04 07:53:41 smithi133 podman[71773]: 2024-03-04 07:53:41.002453493 +0000 UTC m=+0.670409611 container died 19d552f2ef886fd98484c69af8f0ef5cb56bd3d9daca2b0d6776443521b0f09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T07:53:41.574 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:41 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:41.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:41.574 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:41 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:41.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:41.574 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:41 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:41.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:41.574 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:41 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:41.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:41.574 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:41 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:41.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:41.662 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.7.service' 2024-03-04T07:53:41.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:53:41.743 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-04T07:53:41.743 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-04T07:53:41.743 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.8 2024-03-04T07:53:41.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:41.494+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:41.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:41.494+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:41.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:41.494+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:41.771 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:41.494+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:42.048 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:41 smithi152 systemd[1]: Stopping Ceph osd.8 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:42.048 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:41.853+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:42.049 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:41.853+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:42.049 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:41.853+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:42.049 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:41 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:41.853+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:42.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:41 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:41.901+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:42.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:41 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:41.901+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:42.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:41 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:41.901+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:42.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:41 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:41.901+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:42.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:41 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:41.901+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:42.477 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:42.048+0000 7fb69ad07700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:42.478 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:42.048+0000 7fb69ad07700 -1 osd.8 90 *** Got signal Terminated *** 2024-03-04T07:53:42.478 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:42.048+0000 7fb69ad07700 -1 osd.8 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:53:42.478 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:42.476+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:42.478 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:42.476+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:42.478 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:42.477+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:42.478 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:42.477+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:42.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:42 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:42.501+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:42.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:42 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:42.501+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:42.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:42 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:42.501+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:42.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:42 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:42.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:42.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:42 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:42.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:42.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:42 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:42.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:43.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:42.865+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:43.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:42.865+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:43.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:42.865+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:43.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:42 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:42.865+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:43.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:42 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:42.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:43.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:42 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:42.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:43.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:42 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:42.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:43.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:42 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:42.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:43.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:42 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:42.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:43.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:42 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:42.894+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:43.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:43 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:43.483+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:43.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:43 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:43.483+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:43.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:43 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:43.483+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:43.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:43 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:43.483+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:43.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:43 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:43.483+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:43.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:43 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:43.483+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:43.831 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:43.520+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:43.831 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:43.520+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:43.831 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:43.520+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:43.832 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:43.520+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:43.832 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:43.520+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:20.346948+0000 front 2024-03-04T07:53:20.346968+0000 (oldest deadline 2024-03-04T07:53:42.646970+0000) 2024-03-04T07:53:44.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:43.831+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:44.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:43.831+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:44.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:43.831+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:44.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:43 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:43.831+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:44.339 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:43.944+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:44.339 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:43.944+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:44.339 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:43.944+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:44.339 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:43.944+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:44.340 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:43.944+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:44.340 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:43 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:43.944+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:44.758 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:44 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:44.492+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:44.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:44 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:44.492+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:44.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:44 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:44.492+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:44.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:44 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:44.492+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:44.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:44 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:44.492+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:44.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:44 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:44.492+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:44.840 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:44.526+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:44.840 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:44.526+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:44.840 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:44.526+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:44.840 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:44.526+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:44.840 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:44.526+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:20.346948+0000 front 2024-03-04T07:53:20.346968+0000 (oldest deadline 2024-03-04T07:53:42.646970+0000) 2024-03-04T07:53:45.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:44.839+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:45.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:44.839+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:45.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:44.839+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:45.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:44 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:44.839+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:45.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:44 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:44.994+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:45.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:44 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:44.994+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:45.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:44 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:44.994+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:45.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:44 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:44.994+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:45.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:44 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:44.994+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:45.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:44 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:44.994+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:45.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:45 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:45.498+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:45.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:45 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:45.498+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:45.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:45 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:45.498+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:45.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:45 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:45.498+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:45.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:45 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:45.498+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:45.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:45 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:45.498+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:45.820 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:45.530+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:45.820 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:45.530+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:45.820 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:45.530+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:45.820 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:45.530+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:45.820 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:45.530+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:20.346948+0000 front 2024-03-04T07:53:20.346968+0000 (oldest deadline 2024-03-04T07:53:42.646970+0000) 2024-03-04T07:53:46.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:45.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:46.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:45.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:46.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:45.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:46.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:45.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:46.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:45 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:45.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:46.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:45 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:45.965+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:46.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:45 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:45.965+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:46.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:45 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:45.965+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:46.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:45 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:45.965+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:46.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:45 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:45.965+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:46.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:45 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:45.965+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:47.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:46 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:46.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:47.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:46 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:46.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:47.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:46 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:46.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:47.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:46 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:46.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:47.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:46 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:46.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:47.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:46 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:46.514+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:47.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:46.810+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:47.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:46.810+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:47.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:46.810+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:47.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:46.810+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:47.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:46.810+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:47.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:46.568+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.742472+0000 front 2024-03-04T07:52:45.742525+0000 (oldest deadline 2024-03-04T07:53:11.042622+0000) 2024-03-04T07:53:47.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:46.568+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.043163+0000 front 2024-03-04T07:52:51.043194+0000 (oldest deadline 2024-03-04T07:53:16.943395+0000) 2024-03-04T07:53:47.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:46.568+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:04.545128+0000 front 2024-03-04T07:53:04.545297+0000 (oldest deadline 2024-03-04T07:53:28.044870+0000) 2024-03-04T07:53:47.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:46.568+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:10.946106+0000 front 2024-03-04T07:53:10.946127+0000 (oldest deadline 2024-03-04T07:53:36.246052+0000) 2024-03-04T07:53:47.022 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:46 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8[55409]: 2024-03-04T07:53:46.568+0000 7fb6972fb700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:20.346948+0000 front 2024-03-04T07:53:20.346968+0000 (oldest deadline 2024-03-04T07:53:42.646970+0000) 2024-03-04T07:53:47.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:46.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:47.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:46.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:47.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:46.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:47.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:46.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:47.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:46.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:47.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:46.943+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:47.765 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:47 smithi152 podman[72460]: 2024-03-04 07:53:47.482494856 +0000 UTC m=+5.550985998 container died 7195222f906e4388066c21efd13455a5ec34f4ceae71d1474e201bf08717e033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T07:53:48.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:47 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:47.552+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:48.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:47 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:47.552+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:48.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:47 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:47.552+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:48.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:47 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:47.552+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:48.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:47 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:47.552+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:48.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:47 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:47.552+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:48.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:47 smithi152 podman[72460]: 2024-03-04 07:53:47.847994119 +0000 UTC m=+5.916485236 container cleanup 7195222f906e4388066c21efd13455a5ec34f4ceae71d1474e201bf08717e033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-04T07:53:48.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:47 smithi152 bash[72460]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8 2024-03-04T07:53:48.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:47 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:47.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:48.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:47 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:47.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:48.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:47 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:47.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:48.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:47 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:47.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:48.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:47 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:47.819+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:48.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:47 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:47.899+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:48.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:47 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:47.899+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:48.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:47 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:47.899+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:48.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:47 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:47.899+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:48.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:47 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:47.899+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:48.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:47 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:47.899+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:48.513 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:48 smithi152 podman[72477]: 2024-03-04 07:53:48.157669317 +0000 UTC m=+0.666895310 container remove 7195222f906e4388066c21efd13455a5ec34f4ceae71d1474e201bf08717e033 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:53:48.764 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:48 smithi152 podman[72571]: 2024-03-04 07:53:48.416098727 +0000 UTC m=+0.018011790 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:53:48.765 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:48 smithi152 podman[72571]: 2024-03-04 07:53:48.533568298 +0000 UTC m=+0.135481364 container create b1535d98770eb4f483c610f6e09e6543e212e4bbfead9fd14ceaa19e6d4a1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-04T07:53:48.765 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:48 smithi152 podman[72571]: 2024-03-04 07:53:48.761585386 +0000 UTC m=+0.363498442 container init b1535d98770eb4f483c610f6e09e6543e212e4bbfead9fd14ceaa19e6d4a1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T07:53:48.765 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:48 smithi152 podman[72571]: 2024-03-04 07:53:48.764930211 +0000 UTC m=+0.366843262 container start b1535d98770eb4f483c610f6e09e6543e212e4bbfead9fd14ceaa19e6d4a1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:53:49.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:48.559+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:49.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:48.559+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:49.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:48.559+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:49.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:48.559+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:49.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:48.559+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:49.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:48 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:48.559+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:49.021 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:48 smithi152 podman[72571]: 2024-03-04 07:53:48.853094232 +0000 UTC m=+0.455007283 container attach b1535d98770eb4f483c610f6e09e6543e212e4bbfead9fd14ceaa19e6d4a1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8-deactivate, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-04T07:53:49.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:48 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:48.827+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:49.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:48 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:48.827+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:49.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:48 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:48.827+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:49.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:48 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:48.827+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:49.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:48 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:48.827+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:49.326 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 04 07:53:49 smithi152 podman[72571]: 2024-03-04 07:53:49.056343856 +0000 UTC m=+0.658256895 container died b1535d98770eb4f483c610f6e09e6543e212e4bbfead9fd14ceaa19e6d4a1b8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, ceph=True) 2024-03-04T07:53:49.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:48 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:48.929+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:49.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:48 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:48.929+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:49.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:48 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:48.929+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:49.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:48 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:48.929+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:49.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:48 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:48.929+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:49.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:48 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:48.929+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:49.645 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.8.service' 2024-03-04T07:53:49.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:53:49.727 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-04T07:53:49.728 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-04T07:53:49.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.9 2024-03-04T07:53:49.816 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:49.528+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:49.816 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:49.528+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:49.816 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:49.528+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:49.816 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:49.528+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:49.816 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:49.528+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:49.816 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:49 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:49.528+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:50.258 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:49 smithi116 systemd[1]: Stopping Ceph osd.9 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:50.270 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:49 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:49.874+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:50.270 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:49 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:49.874+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:50.270 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:49 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:49.874+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:50.270 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:49 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:49.874+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:50.270 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:49 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:49.874+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:50.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:49 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:49.951+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:50.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:49 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:49.951+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:50.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:49 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:49.951+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:50.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:49 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:49.951+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:50.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:49 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:49.951+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:50.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:49 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:49.951+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:50.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.267+0000 7f6e84400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:50.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.267+0000 7f6e84400700 -1 osd.9 90 *** Got signal Terminated *** 2024-03-04T07:53:50.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.267+0000 7f6e84400700 -1 osd.9 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:53:50.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.481+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:50.760 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.481+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:50.760 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.481+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:50.760 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.481+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:50.760 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.481+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:50.760 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:50 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:50.481+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:51.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:50 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:50.911+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:51.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:50 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:50.911+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:51.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:50 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:50.911+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:51.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:50 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:50.911+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:51.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:50 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:50.911+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:51.298 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:50 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:50.972+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:51.298 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:50 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:50.972+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:51.298 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:50 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:50.972+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:51.298 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:50 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:50.972+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:51.298 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:50 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:50.972+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:51.298 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:50 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:50.972+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:51.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:51.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:51.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:51.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:51.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:51.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:51.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:51.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:51.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:51.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:51.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:51 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:51.502+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:52.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:51.939+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:52.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:51.939+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:52.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:51.939+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:52.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:51.939+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:52.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:51 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:51.939+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:52.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:52 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:52.002+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:52.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:52 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:52.002+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:52.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:52 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:52.002+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:52.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:52 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:52.002+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:52.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:52 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:52.002+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:52.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:52 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:52.002+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:52.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:52 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:52.002+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:53.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:52 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:52.520+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:53.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:52 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:52.520+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:53.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:52 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:52.520+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:53.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:52 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:52.520+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:53.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:52 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:52.520+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:53.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:52 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:52.520+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:53.009 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:52 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:52.520+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.854068+0000 front 2024-03-04T07:53:26.853953+0000 (oldest deadline 2024-03-04T07:53:52.153765+0000) 2024-03-04T07:53:53.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:52 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:52.938+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:53.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:52 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:52.938+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:53.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:52 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:52.938+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:53.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:52 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:52.938+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:53.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:52 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:52.938+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:53.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:52 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:52.938+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:53:53.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:53 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:53.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:53.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:53 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:53.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:53.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:53 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:53.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:53.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:53 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:53.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:53.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:53 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:53.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:53.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:53 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:53.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:53.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:53 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:53.021+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:53.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:53.495+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:53.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:53.495+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:53.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:53.495+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:53.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:53.495+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:53.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:53.495+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:53.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:53.495+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:53.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:53 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:53.495+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.854068+0000 front 2024-03-04T07:53:26.853953+0000 (oldest deadline 2024-03-04T07:53:52.153765+0000) 2024-03-04T07:53:54.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:53 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:53.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:54.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:53 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:53.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:54.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:53 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:53.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:54.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:53 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:53.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:54.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:53 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:53.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:54.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:53 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:53.964+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:53:54.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:54 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:54.062+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:54.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:54 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:54.062+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:54.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:54 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:54.062+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:54.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:54 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:54.062+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:54.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:54 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:54.062+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:54.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:54 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:54.062+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:54.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:54 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:54.062+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:54.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:54.450+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.442282+0000 front 2024-03-04T07:52:36.442644+0000 (oldest deadline 2024-03-04T07:53:00.541737+0000) 2024-03-04T07:53:54.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:54.450+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.843315+0000 front 2024-03-04T07:52:45.843248+0000 (oldest deadline 2024-03-04T07:53:11.743524+0000) 2024-03-04T07:53:54.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:54.450+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:51.744592+0000 front 2024-03-04T07:52:51.744437+0000 (oldest deadline 2024-03-04T07:53:21.745778+0000) 2024-03-04T07:53:54.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:54.450+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:53:01.746776+0000 front 2024-03-04T07:53:01.746837+0000 (oldest deadline 2024-03-04T07:53:26.446896+0000) 2024-03-04T07:53:54.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:54.450+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:09.949014+0000 front 2024-03-04T07:53:09.949161+0000 (oldest deadline 2024-03-04T07:53:34.649064+0000) 2024-03-04T07:53:54.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:54.450+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:19.252297+0000 front 2024-03-04T07:53:19.252364+0000 (oldest deadline 2024-03-04T07:53:42.152223+0000) 2024-03-04T07:53:54.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:54 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9[65070]: 2024-03-04T07:53:54.450+0000 7f6e7fc00700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.854068+0000 front 2024-03-04T07:53:26.853953+0000 (oldest deadline 2024-03-04T07:53:52.153765+0000) 2024-03-04T07:53:55.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:54 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:54.914+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:55.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:54 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:54.914+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:55.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:54 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:54.914+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:55.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:54 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:54.914+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:55.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:54 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:54.914+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:55.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:54 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:54.914+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:53:55.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:55 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:55.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:55.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:55 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:55.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:55.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:55 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:55.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:55.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:55 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:55.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:55.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:55 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:55.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:55.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:55 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:55.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:55.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:55 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:55.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:56.075 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:55 smithi116 podman[97789]: 2024-03-04 07:53:55.811902958 +0000 UTC m=+5.661116729 container died 9f24c844307ab3b3a7c1fea0365b75aa14f74abad70d66d8fc26e9ed7db6afc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-04T07:53:56.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:55.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:56.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:55.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:56.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:55.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:56.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:55.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:56.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:55.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:56.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:55.945+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:53:56.387 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:56 smithi116 podman[97789]: 2024-03-04 07:53:56.202312351 +0000 UTC m=+6.051526146 container cleanup 9f24c844307ab3b3a7c1fea0365b75aa14f74abad70d66d8fc26e9ed7db6afc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2) 2024-03-04T07:53:56.388 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:56 smithi116 bash[97789]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9 2024-03-04T07:53:56.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:56 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:56.117+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:56.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:56 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:56.117+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:56.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:56 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:56.117+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:56.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:56 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:56.117+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:56.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:56 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:56.117+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:56.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:56 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:56.117+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:56.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:56 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:56.117+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:56.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:56 smithi116 podman[97807]: 2024-03-04 07:53:56.386861617 +0000 UTC m=+0.560365922 container remove 9f24c844307ab3b3a7c1fea0365b75aa14f74abad70d66d8fc26e9ed7db6afc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-04T07:53:57.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:56 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:56.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:57.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:56 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:56.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:57.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:56 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:56.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:57.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:56 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:56.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:57.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:56 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:56.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:57.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:56 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:56.933+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:53:57.439 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:57 smithi116 podman[97901]: 2024-03-04 07:53:57.042576277 +0000 UTC m=+0.032121864 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:53:57.440 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:57 smithi116 podman[97901]: 2024-03-04 07:53:57.176936532 +0000 UTC m=+0.166482143 container create f09eda7415db5f4b82c7653e183fa0c936c2739a6f8cfdc04ae7ade3e8ffacc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T07:53:57.440 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:57 smithi116 podman[97901]: 2024-03-04 07:53:57.439159019 +0000 UTC m=+0.428704677 container init f09eda7415db5f4b82c7653e183fa0c936c2739a6f8cfdc04ae7ade3e8ffacc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True) 2024-03-04T07:53:57.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:57 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:57.109+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:57.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:57 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:57.109+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:57.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:57 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:57.109+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:57.588 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:57 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:57.109+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:57.588 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:57 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:57.109+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:57.588 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:57 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:57.109+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:57.588 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:57 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:57.109+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:57.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:57 smithi116 podman[97901]: 2024-03-04 07:53:57.4493758 +0000 UTC m=+0.438921380 container start f09eda7415db5f4b82c7653e183fa0c936c2739a6f8cfdc04ae7ade3e8ffacc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9-deactivate, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:53:57.759 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:57 smithi116 podman[97901]: 2024-03-04 07:53:57.546295031 +0000 UTC m=+0.535840675 container attach f09eda7415db5f4b82c7653e183fa0c936c2739a6f8cfdc04ae7ade3e8ffacc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-03-04T07:53:58.255 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 04 07:53:57 smithi116 podman[97901]: 2024-03-04 07:53:57.88224783 +0000 UTC m=+0.871793405 container died f09eda7415db5f4b82c7653e183fa0c936c2739a6f8cfdc04ae7ade3e8ffacc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-04T07:53:58.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:57 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:57.941+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:58.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:57 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:57.941+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:58.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:57 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:57.941+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:58.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:57 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:57.941+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:58.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:57 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:57.941+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:58.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:57 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:57.941+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:53:58.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:58.072+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:58.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:58.072+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:58.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:58.072+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:58.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:58.072+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:58.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:58.072+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:58.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:58.072+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:58.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:58.072+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:58.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.9.service' 2024-03-04T07:53:58.806 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:53:58.806 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-04T07:53:58.806 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-04T07:53:58.806 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.10 2024-03-04T07:53:59.087 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:58 smithi133 systemd[1]: Stopping Ceph osd.10 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:53:59.088 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.071+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:53:59.088 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.071+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:53:59.088 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.071+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:53:59.088 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.071+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:53:59.088 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.071+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:53:59.088 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.071+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:53:59.088 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.071+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:53:59.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:58 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:58.932+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:53:59.297 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:58 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:58.932+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:53:59.297 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:58 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:58.932+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:53:59.297 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:58 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:58.932+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:53:59.297 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:58 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:58.932+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:53:59.297 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:58 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:58.932+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:53:59.587 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.168+0000 7f8b3fbc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:53:59.588 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.168+0000 7f8b3fbc7700 -1 osd.10 90 *** Got signal Terminated *** 2024-03-04T07:53:59.588 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:53:59 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:53:59.168+0000 7f8b3fbc7700 -1 osd.10 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:54:00.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:59 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:59.955+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:00.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:59 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:59.955+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:00.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:59 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:59.955+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:00.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:59 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:59.955+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:00.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:59 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:59.955+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:00.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:53:59 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:53:59.955+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:00.489 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:00.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:54:00.489 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:00.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:54:00.489 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:00.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:54:00.489 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:00.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:54:00.489 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:00.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:54:00.490 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:00.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:54:00.490 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:00 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:00.086+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:54:01.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:00.956+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:01.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:00.956+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:01.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:00.956+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:01.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:00.956+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:01.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:00.956+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:01.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:00.956+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:01.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:00.956+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:01.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:01.038+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:54:01.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:01.038+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:54:01.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:01.038+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:54:01.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:01.038+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:54:01.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:01.038+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:54:01.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:01.038+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:54:01.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:01 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:01.038+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:54:02.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:01 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:01.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:02.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:01 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:01.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:02.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:01 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:01.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:02.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:01 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:01.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:02.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:01 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:01.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:02.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:01 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:01.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:02.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:01 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:01.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:02.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:02.069+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:54:02.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:02.069+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:54:02.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:02.069+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:54:02.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:02.069+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:54:02.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:02.069+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:54:02.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:02.069+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:54:02.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:02 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:02.069+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:54:03.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.029+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:54:03.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.029+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:54:03.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.029+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:54:03.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.029+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:54:03.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.029+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:54:03.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.029+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:54:03.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.029+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:54:03.521 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:03.035+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:03.521 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:03.035+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:03.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:03.035+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:03.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:03.035+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:03.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:03.035+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:03.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:03.035+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:03.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:03 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:03.035+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:04.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.987+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:36.107635+0000 front 2024-03-04T07:52:36.107864+0000 (oldest deadline 2024-03-04T07:53:00.108232+0000) 2024-03-04T07:54:04.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.987+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:45.210577+0000 front 2024-03-04T07:52:45.210599+0000 (oldest deadline 2024-03-04T07:53:09.910711+0000) 2024-03-04T07:54:04.337 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.987+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:56.113204+0000 front 2024-03-04T07:52:56.113294+0000 (oldest deadline 2024-03-04T07:53:18.413287+0000) 2024-03-04T07:54:04.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.987+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-04T07:52:58.914543+0000 front 2024-03-04T07:52:58.914640+0000 (oldest deadline 2024-03-04T07:53:29.514755+0000) 2024-03-04T07:54:04.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.987+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:13.015647+0000 front 2024-03-04T07:53:13.015621+0000 (oldest deadline 2024-03-04T07:53:36.515776+0000) 2024-03-04T07:54:04.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.987+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:17.017089+0000 front 2024-03-04T07:53:17.017034+0000 (oldest deadline 2024-03-04T07:53:42.316946+0000) 2024-03-04T07:54:04.338 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:03 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10[61403]: 2024-03-04T07:54:03.987+0000 7f8b3c1bb700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:26.418175+0000 front 2024-03-04T07:53:26.418204+0000 (oldest deadline 2024-03-04T07:53:51.117891+0000) 2024-03-04T07:54:04.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.034+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:04.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.034+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:04.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.034+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:04.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.034+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:04.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.034+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:04.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.034+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:04.523 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.034+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:04.786 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:04 smithi133 podman[71881]: 2024-03-04 07:54:04.52294908 +0000 UTC m=+5.518503888 container died 0613f3f15cb9b635bd552dad0a7901619ffc0bc4b196afbe2664b7bbf037e82a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-04T07:54:05.087 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:04 smithi133 podman[71881]: 2024-03-04 07:54:04.885096666 +0000 UTC m=+5.880651490 container cleanup 0613f3f15cb9b635bd552dad0a7901619ffc0bc4b196afbe2664b7bbf037e82a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-04T07:54:05.087 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:04 smithi133 bash[71881]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10 2024-03-04T07:54:05.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:05.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:05.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:05.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:05.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:05.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:05.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:04 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:04.986+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:05.502 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:05 smithi133 podman[71896]: 2024-03-04 07:54:05.14483709 +0000 UTC m=+0.623728420 container remove 0613f3f15cb9b635bd552dad0a7901619ffc0bc4b196afbe2664b7bbf037e82a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-04T07:54:05.814 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:05 smithi133 podman[71986]: 2024-03-04 07:54:05.405112454 +0000 UTC m=+0.017778826 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:54:05.814 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:05 smithi133 podman[71986]: 2024-03-04 07:54:05.512287396 +0000 UTC m=+0.124953766 container create effc1a97e3d263f5810ee678bd98655fe3b54a41373ca92e14da3270bc2ab8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:54:05.814 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:05 smithi133 podman[71986]: 2024-03-04 07:54:05.731066372 +0000 UTC m=+0.343732758 container init effc1a97e3d263f5810ee678bd98655fe3b54a41373ca92e14da3270bc2ab8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T07:54:05.814 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:05 smithi133 podman[71986]: 2024-03-04 07:54:05.734138101 +0000 UTC m=+0.346804488 container start effc1a97e3d263f5810ee678bd98655fe3b54a41373ca92e14da3270bc2ab8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-04T07:54:05.814 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:05 smithi133 podman[71986]: 2024-03-04 07:54:05.814307502 +0000 UTC m=+0.426973899 container attach effc1a97e3d263f5810ee678bd98655fe3b54a41373ca92e14da3270bc2ab8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-04T07:54:06.087 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:06 smithi133 podman[71986]: 2024-03-04 07:54:06.019986607 +0000 UTC m=+0.632652998 container died effc1a97e3d263f5810ee678bd98655fe3b54a41373ca92e14da3270bc2ab8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T07:54:06.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:05.965+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:06.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:05.965+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:06.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:05.965+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:06.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:05.965+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:06.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:05.965+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:06.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:05.965+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:06.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:05.965+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:06.556 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.10.service' 2024-03-04T07:54:06.581 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:06 smithi133 podman[72011]: 2024-03-04 07:54:06.534114787 +0000 UTC m=+0.509105333 container remove effc1a97e3d263f5810ee678bd98655fe3b54a41373ca92e14da3270bc2ab8e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-10-deactivate, RELEASE=wip-lusov-qdb-1bc250c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T07:54:06.581 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:06 smithi133 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.10.service: Deactivated successfully. 2024-03-04T07:54:06.581 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:06 smithi133 systemd[1]: Stopped Ceph osd.10 for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:54:06.581 INFO:journalctl@ceph.osd.10.smithi133.stdout:Mar 04 07:54:06 smithi133 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.10.service: Consumed 10.270s CPU time. 2024-03-04T07:54:06.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:06.595 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-04T07:54:06.596 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-04T07:54:06.596 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.11 2024-03-04T07:54:06.900 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 systemd[1]: Stopping Ceph osd.11 for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:07.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.900+0000 7f326161d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.900+0000 7f326161d700 -1 osd.11 90 *** Got signal Terminated *** 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.900+0000 7f326161d700 -1 osd.11 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.963+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.963+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.963+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.963+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.963+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.963+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:07.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:06 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:06.963+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:08.271 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:08.001+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:08.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:08.001+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:08.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:08.001+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:08.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:08.001+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:08.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:08.001+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:08.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:08.001+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:08.272 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:08 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:08.001+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:09.521 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:09.042+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:09.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:09.042+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:09.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:09.042+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:09.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:09.042+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:09.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:09.042+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:09.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:09.042+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:09.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:09 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:09.042+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:10.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:10.032+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:10.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:10.032+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:10.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:10.032+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:10.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:10.032+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:10.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:10.032+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:10.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:10.032+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:10.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:10.032+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:11.521 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-04T07:52:39.040261+0000 front 2024-03-04T07:52:39.040339+0000 (oldest deadline 2024-03-04T07:53:04.340388+0000) 2024-03-04T07:54:11.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6804 osd.1 since back 2024-03-04T07:52:47.142117+0000 front 2024-03-04T07:52:47.142071+0000 (oldest deadline 2024-03-04T07:53:12.442083+0000) 2024-03-04T07:54:11.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-04T07:52:54.742913+0000 front 2024-03-04T07:52:54.742958+0000 (oldest deadline 2024-03-04T07:53:18.243022+0000) 2024-03-04T07:54:11.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6812 osd.4 since back 2024-03-04T07:53:12.245704+0000 front 2024-03-04T07:53:12.245594+0000 (oldest deadline 2024-03-04T07:53:36.345545+0000) 2024-03-04T07:54:11.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-04T07:53:16.846546+0000 front 2024-03-04T07:53:16.846528+0000 (oldest deadline 2024-03-04T07:53:45.047085+0000) 2024-03-04T07:54:11.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-04T07:53:29.648989+0000 front 2024-03-04T07:53:29.648942+0000 (oldest deadline 2024-03-04T07:53:51.948660+0000) 2024-03-04T07:54:11.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.133:6820 osd.7 since back 2024-03-04T07:53:38.350075+0000 front 2024-03-04T07:53:38.350061+0000 (oldest deadline 2024-03-04T07:54:00.650001+0000) 2024-03-04T07:54:11.522 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:11 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11[62311]: 2024-03-04T07:54:11.060+0000 7f3259c09700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.152:6820 osd.8 since back 2024-03-04T07:53:46.351762+0000 front 2024-03-04T07:53:46.351795+0000 (oldest deadline 2024-03-04T07:54:11.051806+0000) 2024-03-04T07:54:12.717 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:12 smithi152 podman[72688]: 2024-03-04 07:54:12.440733957 +0000 UTC m=+5.640141062 container died 82fef648b6ec1b768528649c4d123e42affb1594a51e91e69d265ff1e07dad07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.license=GPLv2) 2024-03-04T07:54:13.021 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:12 smithi152 podman[72688]: 2024-03-04 07:54:12.716674456 +0000 UTC m=+5.916081611 container cleanup 82fef648b6ec1b768528649c4d123e42affb1594a51e91e69d265ff1e07dad07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-04T07:54:13.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:12 smithi152 bash[72688]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11 2024-03-04T07:54:13.022 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:12 smithi152 podman[72706]: 2024-03-04 07:54:12.934518667 +0000 UTC m=+0.496827041 container remove 82fef648b6ec1b768528649c4d123e42affb1594a51e91e69d265ff1e07dad07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T07:54:13.479 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:13 smithi152 podman[72799]: 2024-03-04 07:54:13.218480043 +0000 UTC m=+0.092448837 container create 574d0ffe35f8b9e81fff5558c7bdca826f1b699c9cc8bba9bd0a7da6adb0b9e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:54:13.479 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:13 smithi152 podman[72799]: 2024-03-04 07:54:13.144326057 +0000 UTC m=+0.018294861 image pull f88ffb32aa6a7d5dfdc954d9f096f0c9cc47981d05ec8d117ba859a22d8185ba quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5 2024-03-04T07:54:13.479 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:13 smithi152 podman[72799]: 2024-03-04 07:54:13.412687113 +0000 UTC m=+0.286655924 container init 574d0ffe35f8b9e81fff5558c7bdca826f1b699c9cc8bba9bd0a7da6adb0b9e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-qdb-1bc250c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T07:54:13.479 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:13 smithi152 podman[72799]: 2024-03-04 07:54:13.415827734 +0000 UTC m=+0.289796524 container start 574d0ffe35f8b9e81fff5558c7bdca826f1b699c9cc8bba9bd0a7da6adb0b9e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T07:54:13.479 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:13 smithi152 podman[72799]: 2024-03-04 07:54:13.479086114 +0000 UTC m=+0.353054919 container attach 574d0ffe35f8b9e81fff5558c7bdca826f1b699c9cc8bba9bd0a7da6adb0b9e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T07:54:13.771 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 04 07:54:13 smithi152 podman[72799]: 2024-03-04 07:54:13.704661642 +0000 UTC m=+0.578630444 container died 574d0ffe35f8b9e81fff5558c7bdca826f1b699c9cc8bba9bd0a7da6adb0b9e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-03-04T07:54:14.262 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@osd.11.service' 2024-03-04T07:54:14.304 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:14.304 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-04T07:54:14.304 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-04T07:54:14.304 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.a 2024-03-04T07:54:14.758 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:54:14 smithi116 systemd[1]: Stopping Ceph mds.a for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:15.258 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:54:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a[71571]: 2024-03-04T07:54:14.873+0000 7fcd52600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:15.259 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:54:14 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a[71571]: 2024-03-04T07:54:14.873+0000 7fcd52600700 -1 mds.a *** got signal Terminated *** 2024-03-04T07:54:17.357 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:54:17 smithi116 podman[98005]: 2024-03-04 07:54:17.054636558 +0000 UTC m=+2.323054961 container died e387a3fde33126a95b50b72e22860bacb4107e97fe03f5522ff38a14a6b895c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T07:54:17.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:54:17 smithi116 podman[98005]: 2024-03-04 07:54:17.500289566 +0000 UTC m=+2.768708011 container cleanup e387a3fde33126a95b50b72e22860bacb4107e97fe03f5522ff38a14a6b895c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:54:17.759 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:54:17 smithi116 bash[98005]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a 2024-03-04T07:54:18.215 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 04 07:54:17 smithi116 podman[98030]: 2024-03-04 07:54:17.759549192 +0000 UTC m=+0.690174569 container remove e387a3fde33126a95b50b72e22860bacb4107e97fe03f5522ff38a14a6b895c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-04T07:54:18.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.a.service' 2024-03-04T07:54:18.360 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:18.360 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-04T07:54:18.360 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-04T07:54:18.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.d 2024-03-04T07:54:18.758 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:54:18 smithi116 systemd[1]: Stopping Ceph mds.d for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:19.258 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:54:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d[70586]: 2024-03-04T07:54:18.936+0000 7ff233800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:19.259 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:54:18 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d[70586]: 2024-03-04T07:54:18.936+0000 7ff233800700 -1 mds.d *** got signal Terminated *** 2024-03-04T07:54:22.659 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:54:22 smithi116 podman[98137]: 2024-03-04 07:54:22.403698839 +0000 UTC m=+3.610510392 container died dcb27b9d2112add20d58a298f88ef1e1a21b6077a1d8324a9fdcecfdfd00f5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-04T07:54:22.972 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:54:22 smithi116 podman[98137]: 2024-03-04 07:54:22.770793407 +0000 UTC m=+3.977605000 container cleanup dcb27b9d2112add20d58a298f88ef1e1a21b6077a1d8324a9fdcecfdfd00f5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T07:54:22.972 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:54:22 smithi116 bash[98137]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d 2024-03-04T07:54:23.258 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 04 07:54:22 smithi116 podman[98151]: 2024-03-04 07:54:22.971780111 +0000 UTC m=+0.557882873 container remove dcb27b9d2112add20d58a298f88ef1e1a21b6077a1d8324a9fdcecfdfd00f5ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T07:54:23.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.d.service' 2024-03-04T07:54:23.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:23.598 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-04T07:54:23.598 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-04T07:54:23.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.g 2024-03-04T07:54:24.008 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:54:23 smithi116 systemd[1]: Stopping Ceph mds.g for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:24.508 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:54:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g[71233]: 2024-03-04T07:54:24.215+0000 7f5211a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:24.509 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:54:24 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g[71233]: 2024-03-04T07:54:24.215+0000 7f5211a00700 -1 mds.g *** got signal Terminated *** 2024-03-04T07:54:28.508 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:54:28 smithi116 podman[98256]: 2024-03-04 07:54:28.232705239 +0000 UTC m=+4.143412269 container died 6f4a49fbf911f726872920823c5a75a33dbbbe82cfaadd9a65f3028f86475704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-04T07:54:28.866 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:54:28 smithi116 podman[98256]: 2024-03-04 07:54:28.665371199 +0000 UTC m=+4.576078259 container cleanup 6f4a49fbf911f726872920823c5a75a33dbbbe82cfaadd9a65f3028f86475704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-04T07:54:28.867 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:54:28 smithi116 bash[98256]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g 2024-03-04T07:54:29.258 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 04 07:54:28 smithi116 podman[98271]: 2024-03-04 07:54:28.866232693 +0000 UTC m=+0.616181829 container remove 6f4a49fbf911f726872920823c5a75a33dbbbe82cfaadd9a65f3028f86475704 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:54:29.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.g.service' 2024-03-04T07:54:29.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:29.403 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-04T07:54:29.403 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-04T07:54:29.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.j 2024-03-04T07:54:30.009 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:54:29 smithi116 systemd[1]: Stopping Ceph mds.j for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:30.508 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:54:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j[70244]: 2024-03-04T07:54:30.041+0000 7f6097e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:30.508 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:54:30 smithi116 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j[70244]: 2024-03-04T07:54:30.041+0000 7f6097e00700 -1 mds.j *** got signal Terminated *** 2024-03-04T07:54:34.333 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:54:34 smithi116 podman[98375]: 2024-03-04 07:54:34.05835971 +0000 UTC m=+4.180474557 container died 61a6923b3de48b55eef09dfbe9edf2b778d9a1b39e6b7667fc8d12d6f6677243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:54:34.680 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:54:34 smithi116 podman[98375]: 2024-03-04 07:54:34.444046255 +0000 UTC m=+4.566161095 container cleanup 61a6923b3de48b55eef09dfbe9edf2b778d9a1b39e6b7667fc8d12d6f6677243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:54:34.680 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:54:34 smithi116 bash[98375]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j 2024-03-04T07:54:35.008 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 04 07:54:34 smithi116 podman[98401]: 2024-03-04 07:54:34.678195323 +0000 UTC m=+0.600871717 container remove 61a6923b3de48b55eef09dfbe9edf2b778d9a1b39e6b7667fc8d12d6f6677243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-j, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-04T07:54:35.193 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.j.service' 2024-03-04T07:54:35.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:35.304 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-04T07:54:35.304 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-04T07:54:35.304 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.b 2024-03-04T07:54:35.587 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:54:35 smithi133 systemd[1]: Stopping Ceph mds.b for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:35.844 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:54:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b[67089]: 2024-03-04T07:54:35.825+0000 7fa5b5a4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:35.845 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:54:35 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b[67089]: 2024-03-04T07:54:35.825+0000 7fa5b5a4e700 -1 mds.b *** got signal Terminated *** 2024-03-04T07:54:40.087 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:54:39 smithi133 podman[72089]: 2024-03-04 07:54:39.834763675 +0000 UTC m=+4.341279564 container died 104f6c82bd6c3afd1904912c8eb96f04db43be68ad382116bd338a8c80c00c0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T07:54:40.559 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:54:40 smithi133 podman[72089]: 2024-03-04 07:54:40.157532558 +0000 UTC m=+4.664048439 container cleanup 104f6c82bd6c3afd1904912c8eb96f04db43be68ad382116bd338a8c80c00c0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-04T07:54:40.559 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:54:40 smithi133 bash[72089]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b 2024-03-04T07:54:40.560 INFO:journalctl@ceph.mds.b.smithi133.stdout:Mar 04 07:54:40 smithi133 podman[72115]: 2024-03-04 07:54:40.367106177 +0000 UTC m=+0.529707323 container remove 104f6c82bd6c3afd1904912c8eb96f04db43be68ad382116bd338a8c80c00c0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T07:54:40.578 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.b.service' 2024-03-04T07:54:40.616 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:40.617 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-04T07:54:40.617 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-04T07:54:40.617 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.e 2024-03-04T07:54:40.837 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:54:40 smithi133 systemd[1]: Stopping Ceph mds.e for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:41.337 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:54:40 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e[66472]: 2024-03-04T07:54:40.928+0000 7f767be82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:41.337 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:54:40 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e[66472]: 2024-03-04T07:54:40.928+0000 7f767be82700 -1 mds.e *** got signal Terminated *** 2024-03-04T07:54:45.236 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:54:44 smithi133 podman[72223]: 2024-03-04 07:54:44.934413088 +0000 UTC m=+4.100248118 container died e8b7fc0b65d68f17d5be640f336610b4f37e0db5389ac2dedfb21316d44f0fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-04T07:54:45.587 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:54:45 smithi133 podman[72223]: 2024-03-04 07:54:45.318995397 +0000 UTC m=+4.484830427 container cleanup e8b7fc0b65d68f17d5be640f336610b4f37e0db5389ac2dedfb21316d44f0fb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:54:45.587 INFO:journalctl@ceph.mds.e.smithi133.stdout:Mar 04 07:54:45 smithi133 bash[72223]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-e 2024-03-04T07:54:45.745 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.e.service' 2024-03-04T07:54:45.780 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:45.780 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-04T07:54:45.780 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-04T07:54:45.781 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.h 2024-03-04T07:54:46.264 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:54:45 smithi133 systemd[1]: Stopping Ceph mds.h for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:46.265 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:54:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h[66781]: 2024-03-04T07:54:46.096+0000 7fc105bcb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:46.265 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:54:46 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h[66781]: 2024-03-04T07:54:46.096+0000 7fc105bcb700 -1 mds.h *** got signal Terminated *** 2024-03-04T07:54:50.165 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:54:49 smithi133 podman[72345]: 2024-03-04 07:54:49.905351861 +0000 UTC m=+3.911283607 container died fca8568dd783e833847f42bc164e928abc46314f02ed44f5db148e778c0ca501 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:54:50.573 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:54:50 smithi133 podman[72345]: 2024-03-04 07:54:50.280598168 +0000 UTC m=+4.286529915 container cleanup fca8568dd783e833847f42bc164e928abc46314f02ed44f5db148e778c0ca501 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:54:50.573 INFO:journalctl@ceph.mds.h.smithi133.stdout:Mar 04 07:54:50 smithi133 bash[72345]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-h 2024-03-04T07:54:50.705 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.h.service' 2024-03-04T07:54:50.740 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:50.740 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-04T07:54:50.740 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-04T07:54:50.740 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.k 2024-03-04T07:54:51.087 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:54:50 smithi133 systemd[1]: Stopping Ceph mds.k for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:51.087 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:54:51 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k[66144]: 2024-03-04T07:54:51.058+0000 7f17f2ae1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:51.087 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:54:51 smithi133 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k[66144]: 2024-03-04T07:54:51.058+0000 7f17f2ae1700 -1 mds.k *** got signal Terminated *** 2024-03-04T07:54:54.650 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:54:54 smithi133 podman[72468]: 2024-03-04 07:54:54.368832648 +0000 UTC m=+3.407381975 container died da4094da1eadb24e7b240bc2ccb19a71a1b19f6f497d46bf63e358a15ac27657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T07:54:55.000 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:54:54 smithi133 podman[72468]: 2024-03-04 07:54:54.750347901 +0000 UTC m=+3.788897284 container cleanup da4094da1eadb24e7b240bc2ccb19a71a1b19f6f497d46bf63e358a15ac27657 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-04T07:54:55.001 INFO:journalctl@ceph.mds.k.smithi133.stdout:Mar 04 07:54:54 smithi133 bash[72468]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-k 2024-03-04T07:54:55.171 DEBUG:teuthology.orchestra.run.smithi133:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.k.service' 2024-03-04T07:54:55.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:55.205 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-04T07:54:55.205 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-04T07:54:55.205 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.c 2024-03-04T07:54:55.506 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:54:55 smithi152 systemd[1]: Stopping Ceph mds.c for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:54:55.771 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:54:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c[67644]: 2024-03-04T07:54:55.505+0000 7f8676115700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:54:55.772 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:54:55 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c[67644]: 2024-03-04T07:54:55.505+0000 7f8676115700 -1 mds.c *** got signal Terminated *** 2024-03-04T07:54:59.415 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:54:59 smithi152 podman[72907]: 2024-03-04 07:54:59.133302531 +0000 UTC m=+3.735254700 container died 44d4152152c3db31c851b8931aa768cb22b988d2a477e5c7abd497852bba0662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T07:54:59.765 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:54:59 smithi152 podman[72907]: 2024-03-04 07:54:59.489072046 +0000 UTC m=+4.091024213 container cleanup 44d4152152c3db31c851b8931aa768cb22b988d2a477e5c7abd497852bba0662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-04T07:54:59.765 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 04 07:54:59 smithi152 bash[72907]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-c 2024-03-04T07:54:59.922 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.c.service' 2024-03-04T07:54:59.960 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:54:59.961 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-04T07:54:59.961 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-04T07:54:59.961 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.f 2024-03-04T07:55:00.521 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:00 smithi152 systemd[1]: Stopping Ceph mds.f for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:55:00.521 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f[67337]: 2024-03-04T07:55:00.259+0000 7f421625f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:55:00.521 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:00 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f[67337]: 2024-03-04T07:55:00.259+0000 7f421625f700 -1 mds.f *** got signal Terminated *** 2024-03-04T07:55:04.522 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:04 smithi152 podman[73040]: 2024-03-04 07:55:04.246165619 +0000 UTC m=+4.072537412 container died 15259a754df582f202c0e65a9647504263ce8df5075ab5927f6ef2e17c265a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T07:55:04.877 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:04 smithi152 podman[73040]: 2024-03-04 07:55:04.67596374 +0000 UTC m=+4.502335535 container cleanup 15259a754df582f202c0e65a9647504263ce8df5075ab5927f6ef2e17c265a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T07:55:04.877 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:04 smithi152 bash[73040]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f 2024-03-04T07:55:05.101 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.f.service' 2024-03-04T07:55:05.127 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:04 smithi152 podman[73054]: 2024-03-04 07:55:04.876994731 +0000 UTC m=+0.623649628 container remove 15259a754df582f202c0e65a9647504263ce8df5075ab5927f6ef2e17c265a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-f, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-qdb-1bc250c) 2024-03-04T07:55:05.127 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:05 smithi152 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.f.service: Deactivated successfully. 2024-03-04T07:55:05.127 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:05 smithi152 systemd[1]: Stopped Ceph mds.f for af2ccd48-d9f8-11ee-95c6-87774f69a715. 2024-03-04T07:55:05.127 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 04 07:55:05 smithi152 systemd[1]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.f.service: Consumed 1.113s CPU time. 2024-03-04T07:55:05.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:55:05.137 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-04T07:55:05.138 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-04T07:55:05.138 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.i 2024-03-04T07:55:05.463 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 04 07:55:05 smithi152 systemd[1]: Stopping Ceph mds.i for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:55:05.771 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 04 07:55:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-i[67953]: 2024-03-04T07:55:05.462+0000 7f7644a85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:55:05.771 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 04 07:55:05 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-i[67953]: 2024-03-04T07:55:05.462+0000 7f7644a85700 -1 mds.i *** got signal Terminated *** 2024-03-04T07:55:09.163 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 04 07:55:08 smithi152 podman[73161]: 2024-03-04 07:55:08.779226315 +0000 UTC m=+3.417057091 container died f793c284997a3e602e6e85f27833aa3e72fbe7c66ceae3388be97cf1a5cc9728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-lusov-qdb-1bc250c, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T07:55:09.496 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 04 07:55:09 smithi152 podman[73161]: 2024-03-04 07:55:09.237459196 +0000 UTC m=+3.875289962 container cleanup f793c284997a3e602e6e85f27833aa3e72fbe7c66ceae3388be97cf1a5cc9728 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-i, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-qdb-1bc250c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-04T07:55:09.497 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 04 07:55:09 smithi152 bash[73161]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-i 2024-03-04T07:55:09.652 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.i.service' 2024-03-04T07:55:09.689 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:55:09.689 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-04T07:55:09.690 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-04T07:55:09.690 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.l 2024-03-04T07:55:10.008 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:55:09 smithi152 systemd[1]: Stopping Ceph mds.l for af2ccd48-d9f8-11ee-95c6-87774f69a715... 2024-03-04T07:55:10.271 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:55:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l[67026]: 2024-03-04T07:55:10.007+0000 7f78c067c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T07:55:10.271 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:55:10 smithi152 ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l[67026]: 2024-03-04T07:55:10.007+0000 7f78c067c700 -1 mds.l *** got signal Terminated *** 2024-03-04T07:55:11.800 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:55:11 smithi152 podman[73283]: 2024-03-04 07:55:11.545175329 +0000 UTC m=+1.643341374 container died 48ac649e0e10ebbaeb0bd90cd120f0a2567e16dc7745cf60871c4cdd1ce41225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-04T07:55:12.117 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:55:11 smithi152 podman[73283]: 2024-03-04 07:55:11.899899569 +0000 UTC m=+1.998065604 container cleanup 48ac649e0e10ebbaeb0bd90cd120f0a2567e16dc7745cf60871c4cdd1ce41225 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3011ef4e883c5a20dd92da8f81055baaa8abf2682670e25e08f983b156727fe5, name=ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-qdb-1bc250c, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T07:55:12.117 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 04 07:55:11 smithi152 bash[73283]: ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715-mds-l 2024-03-04T07:55:12.254 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-af2ccd48-d9f8-11ee-95c6-87774f69a715@mds.l.service' 2024-03-04T07:55:12.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T07:55:12.288 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-04T07:55:12.288 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T07:55:12.609 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:55:17.215 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T07:55:17.406 INFO:teuthology.orchestra.run.smithi133.stdout:Deleting cluster with fsid: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:55:19.956 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T07:55:20.149 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:55:22.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:55:22.705 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:55:22.739 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T07:55:22.767 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-04T07:55:22.770 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi116/crash 2024-03-04T07:55:22.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash -- . 2024-03-04T07:55:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T07:55:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-04T07:55:22.886 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi133/crash 2024-03-04T07:55:22.887 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash -- . 2024-03-04T07:55:22.928 INFO:teuthology.orchestra.run.smithi133.stderr:tar: /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T07:55:22.928 INFO:teuthology.orchestra.run.smithi133.stderr:tar: Error is not recoverable: exiting now 2024-03-04T07:55:22.932 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi152/crash 2024-03-04T07:55:22.933 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash -- . 2024-03-04T07:55:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T07:55:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2024-03-04T07:55:22.966 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-04T07:55:22.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-04T07:55:23.072 INFO:teuthology.orchestra.run.smithi116.stdout:2024-03-04T07:31:05.924106+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T07:55:23.073 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-04T07:55:23.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[SEC\]' /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-04T07:55:23.160 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]' /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-04T07:55:23.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[WRN\]' /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-04T07:55:23.324 INFO:teuthology.orchestra.run.smithi116.stdout:2024-03-04T07:31:05.924106+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T07:55:23.325 INFO:tasks.cephadm:Compressing logs... 2024-03-04T07:55:23.326 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T07:55:23.329 DEBUG:teuthology.orchestra.run.smithi133:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T07:55:23.332 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T07:55:23.359 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T07:55:23.359 INFO:teuthology.orchestra.run.smithi152.stderr:: No such file or directory 2024-03-04T07:55:23.359 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.l.log 2024-03-04T07:55:23.360 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T07:55:23.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.i.log 2024-03-04T07:55:23.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.11.log 2024-03-04T07:55:23.378 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.z.log 2024-03-04T07:55:23.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.i.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T07:55:23.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.8.log 2024-03-04T07:55:23.378 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.5.log 2024-03-04T07:55:23.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.f.log 2024-03-04T07:55:23.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T07:55:23.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T07:55:23.379 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log 2024-03-04T07:55:23.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log: 82.3% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T07:55:23.379 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-04T07:55:23.379 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log: 94.6% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.l.log.gz 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr: 94.7%gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.c.log 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.i.log.gz 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.c.log: 87.6% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log: 92.9% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.z.log.gz 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.f.log.gz 2024-03-04T07:55:23.380 INFO:teuthology.orchestra.run.smithi152.stderr: 94.7% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.c.log.gz 2024-03-04T07:55:23.381 INFO:teuthology.orchestra.run.smithi133.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T07:55:23.381 INFO:teuthology.orchestra.run.smithi133.stderr:: No such file or directory 2024-03-04T07:55:23.381 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.10.log 2024-03-04T07:55:23.382 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T07:55:23.382 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.b.log 2024-03-04T07:55:23.382 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.4.log 2024-03-04T07:55:23.382 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-04T07:55:23.382 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.b.log: /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.4.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T07:55:23.382 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.k.log 2024-03-04T07:55:23.383 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.h.log 2024-03-04T07:55:23.383 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.7.log 2024-03-04T07:55:23.383 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.e.log 2024-03-04T07:55:23.383 INFO:teuthology.orchestra.run.smithi133.stderr: 90.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T07:55:23.383 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-04T07:55:23.383 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.e.log: 94.5% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.k.log.gz 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr: 94.6%gzip -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.h.log.gz -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log: 94.5% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.e.log.gz 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log: 82.3% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log: 87.6% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T07:55:23.384 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-04T07:55:23.385 INFO:teuthology.orchestra.run.smithi133.stderr: 90.9% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T07:55:23.385 INFO:teuthology.orchestra.run.smithi152.stderr: 90.9% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T07:55:23.392 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-04T07:55:23.393 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T07:55:23.394 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.6.log 2024-03-04T07:55:23.395 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-04T07:55:23.396 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T07:55:23.396 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.g.log 2024-03-04T07:55:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.j.log 2024-03-04T07:55:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.3.log 2024-03-04T07:55:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.j.log: 90.1% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T07:55:23.401 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-04T07:55:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-04T07:55:23.402 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T07:55:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.3.log: /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T07:55:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.0.log: /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log: 82.5% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T07:55:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 94.6% -5 --verbose -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.j.log.gz -- 2024-03-04T07:55:23.406 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log 2024-03-04T07:55:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T07:55:23.410 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log: /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.9.log 2024-03-04T07:55:23.413 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T07:55:23.415 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.d.log 2024-03-04T07:55:23.416 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.a.log 2024-03-04T07:55:23.429 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.d.log: /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.a.log: 90.8% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T07:55:23.433 INFO:teuthology.orchestra.run.smithi116.stderr: 94.6% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.a.log.gz 2024-03-04T07:55:23.435 INFO:teuthology.orchestra.run.smithi116.stderr: 94.3% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.d.log.gz 2024-03-04T07:55:23.703 INFO:teuthology.orchestra.run.smithi116.stderr: 90.0% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-04T07:55:23.736 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.g.log.gz 2024-03-04T07:55:23.890 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-04T07:55:23.931 INFO:teuthology.orchestra.run.smithi133.stderr: 91.7% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-04T07:55:24.062 INFO:teuthology.orchestra.run.smithi116.stderr: 92.5% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.3.log.gz 2024-03-04T07:55:24.124 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.8.log.gz 2024-03-04T07:55:24.264 INFO:teuthology.orchestra.run.smithi152.stderr: 93.4% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.11.log.gz 2024-03-04T07:55:24.402 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-04T07:55:24.467 INFO:teuthology.orchestra.run.smithi133.stderr: 92.6% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.4.log.gz 2024-03-04T07:55:24.489 INFO:teuthology.orchestra.run.smithi133.stderr: 92.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.7.log.gz 2024-03-04T07:55:24.554 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.9.log.gz 2024-03-04T07:55:24.559 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-04T07:55:24.637 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.6.log.gz 2024-03-04T07:55:24.641 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-04T07:55:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m1.279s 2024-03-04T07:55:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m5.457s 2024-03-04T07:55:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m0.520s 2024-03-04T07:55:24.671 INFO:teuthology.orchestra.run.smithi152.stderr: 91.4% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.5.log.gz 2024-03-04T07:55:25.030 INFO:teuthology.orchestra.run.smithi133.stderr: 92.9% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-mds.b.log.gz 2024-03-04T07:55:25.039 INFO:teuthology.orchestra.run.smithi152.stderr: 92.4% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-04T07:55:25.041 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-04T07:55:25.041 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m1.694s 2024-03-04T07:55:25.041 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m5.040s 2024-03-04T07:55:25.041 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m0.204s 2024-03-04T07:55:25.177 INFO:teuthology.orchestra.run.smithi133.stderr: 91.4% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.10.log.gz 2024-03-04T07:55:25.291 INFO:teuthology.orchestra.run.smithi133.stderr: 92.4% -- replaced with /var/log/ceph/af2ccd48-d9f8-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-04T07:55:25.293 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-03-04T07:55:25.293 INFO:teuthology.orchestra.run.smithi133.stderr:real 0m1.947s 2024-03-04T07:55:25.293 INFO:teuthology.orchestra.run.smithi133.stderr:user 0m7.985s 2024-03-04T07:55:25.293 INFO:teuthology.orchestra.run.smithi133.stderr:sys 0m0.286s 2024-03-04T07:55:25.293 INFO:tasks.cephadm:Archiving logs... 2024-03-04T07:55:25.294 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi116/log 2024-03-04T07:55:25.295 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T07:55:26.124 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi133/log 2024-03-04T07:55:26.125 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T07:55:27.314 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi152/log 2024-03-04T07:55:27.315 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T07:55:28.154 INFO:tasks.cephadm:Removing cluster... 2024-03-04T07:55:28.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 --force 2024-03-04T07:55:28.541 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:55:29.128 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 --force 2024-03-04T07:55:29.313 INFO:teuthology.orchestra.run.smithi133.stdout:Deleting cluster with fsid: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:55:29.579 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid af2ccd48-d9f8-11ee-95c6-87774f69a715 --force 2024-03-04T07:55:29.771 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: af2ccd48-d9f8-11ee-95c6-87774f69a715 2024-03-04T07:55:30.084 INFO:tasks.cephadm:Removing cephadm ... 2024-03-04T07:55:30.085 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T07:55:30.142 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T07:55:30.159 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T07:55:30.174 INFO:tasks.cephadm:Teardown complete 2024-03-04T07:55:30.174 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T07:55:30.185 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T07:55:30.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T07:55:30.193 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T07:55:30.202 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T07:55:30.357 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T07:55:30.367 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T07:55:30.367 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T07:55:30.375 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T07:55:30.378 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T07:55:30.390 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-03-04T07:55:30.392 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-04T07:55:30.394 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T07:55:30.395 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-03-04T07:55:30.395 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 0 +1032us[+1032us] +/- 83ms 2024-03-04T07:55:30.395 INFO:teuthology.orchestra.run.smithi133.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -1122us[-1116us] +/- 81ms 2024-03-04T07:55:30.395 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +301us[ +301us] +/- 92ms 2024-03-04T07:55:30.395 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:55:30.396 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T07:55:30.396 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-04T07:55:30.396 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +955us[ +955us] +/- 83ms 2024-03-04T07:55:30.396 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -1204us[-1189us] +/- 82ms 2024-03-04T07:55:30.396 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 1 +243us[ +243us] +/- 92ms 2024-03-04T07:55:30.396 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:55:30.423 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-04T07:55:30.432 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T07:55:30.432 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-04T07:55:30.432 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 196 +993us[+1026us] +/- 80ms 2024-03-04T07:55:30.432 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 -1148us[-1142us] +/- 81ms 2024-03-04T07:55:30.432 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +293us[ +293us] +/- 92ms 2024-03-04T07:55:30.432 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T07:55:30.434 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T07:55:30.444 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T07:55:30.445 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T07:55:30.454 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T07:55:30.696 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T07:55:30.774 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T07:55:30.849 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T07:55:30.952 INFO:teuthology.orchestra.run.smithi116.stdout:type=AVC msg=audit(1709536733.073:197): avc: denied { checkpoint_restore } for pid=1170 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1 2024-03-04T07:55:30.954 DEBUG:teuthology.task.selinux:ubuntu@smithi116.front.sepia.ceph.com has 1 denials 2024-03-04T07:55:30.954 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T07:55:30.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:55:30.987 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T07:55:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:55:31.018 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 166, in teardown self.get_new_denials() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 215, in get_new_denials raise SELinuxError(node=remote, teuthology.exceptions.SELinuxError: SELinux denials found on ubuntu@smithi116.front.sepia.ceph.com: ['type=AVC msg=audit(1709536733.073:197): avc: denied { checkpoint_restore } for pid=1170 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-04T07:55:31.019 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T07:55:31.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T07:55:31.040 INFO:teuthology.task.internal:Duration was 2199.026731 seconds 2024-03-04T07:55:31.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T07:55:31.051 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T07:55:31.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T07:55:31.059 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T07:55:31.062 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T07:55:31.105 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T07:55:31.106 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T07:55:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T07:55:31.592 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T07:55:31.592 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-04T07:55:31.592 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T07:55:31.661 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2024-03-04T07:55:31.661 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T07:55:31.691 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-03-04T07:55:31.691 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T07:55:31.713 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T07:55:31.713 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T07:55:31.721 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T07:55:31.733 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T07:55:31.812 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T07:55:31.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T07:55:31.818 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T07:55:31.820 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T07:55:32.292 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T07:55:32.304 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T07:55:32.304 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T07:55:32.388 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T07:55:32.415 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T07:55:32.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T07:55:32.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T07:55:32.462 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T07:55:32.464 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T07:55:32.487 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2024-03-04T07:55:32.511 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-03-04T07:55:32.563 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-04T07:55:32.616 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T07:55:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:55:32.663 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T07:55:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:55:32.677 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T07:55:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T07:55:32.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T07:55:32.704 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T07:55:32.705 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi116 2024-03-04T07:55:32.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T07:55:32.820 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi133 2024-03-04T07:55:32.820 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T07:55:32.858 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706/remote/smithi152 2024-03-04T07:55:32.859 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T07:55:32.898 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T07:55:32.898 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T07:55:32.906 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T07:55:32.908 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T07:55:32.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T07:55:32.970 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T07:55:32.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T07:55:32.981 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T07:55:32.981 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T07:55:32.988 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T07:55:32.990 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T07:55:33.002 INFO:teuthology.orchestra.run.smithi133.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 07:55 /home/ubuntu/cephtest 2024-03-04T07:55:33.005 INFO:teuthology.orchestra.run.smithi152.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 07:55 /home/ubuntu/cephtest 2024-03-04T07:55:33.038 INFO:teuthology.orchestra.run.smithi116.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 07:55 /home/ubuntu/cephtest 2024-03-04T07:55:33.041 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T07:55:33.052 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T07:55:33.064 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi116.front.sepia.ceph.com: ['type=AVC msg=audit(1709536733.073:197): avc: denied { checkpoint_restore } for pid=1170 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-04T07:55:33.064 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T07:55:33.091 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-04T07:55:33.091 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T07:55:33.115 DEBUG:teuthology.nuke:shortname: smithi133 2024-03-04T07:55:33.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T07:55:33.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 07:06:36.225350', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNx5nWQWa/4dHq4IL0y2koYBMRp0lbmm2KW06oL0OEMIW+lsJZwomGs45GMJZNJsl6WhrLQ7TTNS1zY+EJ29BgQ='} 2024-03-04T07:55:33.139 DEBUG:teuthology.nuke:shortname: smithi152 2024-03-04T07:55:33.139 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T07:55:33.142 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 07:06:36.224414', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE77/V2gQpv2FdAfkeHRIYQ4VPJrWTNJeQhWMjdMgn9BVhBXz95IwO4lHKJ1/cuOrVGYMHfs+ho308kHSddU8rc='} 2024-03-04T07:55:33.162 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-04_07:05:03-fs-wip-lusov-qdb-distro-default-smithi/7580706', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 07:06:36.222827', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbrYJDc/cEMuOlrsDnRJysw6nf7F0Bpy3Zyjj/vUuVB7ASJ5+tIamGzJfZECXZECZXMKiucdSyGRbFB70Emig4='} 2024-03-04T07:55:33.174 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-04T07:55:33.174 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T07:55:33.192 INFO:teuthology.orchestra.console.smithi133:Power off 2024-03-04T07:55:33.192 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T07:55:33.201 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-04T07:55:33.201 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:33.213 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2024-03-04T07:55:33.214 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:33.227 INFO:teuthology.orchestra.console.smithi152:Power off 2024-03-04T07:55:33.227 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T07:55:33.236 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-04T07:55:33.236 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-03-04T07:55:33.251 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-03-04T07:55:33.251 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:33.273 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-04T07:55:37.237 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:37.245 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:37.274 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:37.347 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-04T07:55:37.354 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-03-04T07:55:37.383 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-04T07:55:41.352 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:41.360 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:41.384 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:41.462 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-04T07:55:41.470 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-03-04T07:55:41.494 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-03-04T07:55:41.594 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-03-04T07:55:45.464 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:45.472 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T07:55:45.575 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-04T07:55:45.581 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2024-03-04T07:55:45.675 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-04T07:55:45.682 INFO:teuthology.orchestra.console.smithi133:Power off completed 2024-03-04T07:55:45.787 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 786939134d7eee4dc7a805245d3f595929af37d7 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} duration: 2199.0267312526703 failure_reason: '"2024-03-04T07:31:05.924106+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-03-04T07:55:45.788 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T07:55:45.878 INFO:teuthology.run:FAIL